unit_key_opens is false for other formats, units that are short, clear, or carry
fewer than KEY_PROOF_PACKETS content packets (before, an empty slice or an
all-zero unit opened under any key). The fast path returns packet counts, and a
seeded differential pins them to decrypt_unit's over padding, zero-sync and
corrupted packets; it fails if either skip branch is removed. The compile_fail
door names E0425.
Add `aacs::content::unit_key_opens(unit, key, format) -> bool` behind a new
non-default `keyproof` feature — one bit (key right/wrong), never plaintext —
for a private offline key-resolver (kdb). BD-TS per-packet fast reject, with a
decrypt_unit + is_clean fallback for PS/odd units. A not(keyproof)-gated
compile_fail door in keys/public_api_cannot_decrypt_aacs pins its absence by
default, so the default public API still cannot decrypt AACS (KU §2.2). Also
expose test_util::encrypt_unit alongside decrypt_unit for consumer fixtures.
Equivalence test pins the fast path to decrypt_unit + is_clean. Version left to
the release owner.
- Key evidence reads the filesystem in batches with the metadata partition
prefetched, not one command per sector (~1 s on a USB UHD).
- The live Blu-ray scan prefetches only the files it parses (~22 MB and
three long seeks less on a UHD). The MKB is read from a 4 MiB prefix with
its record headers walked; its bytes are unchanged.
- macOS: no fixed 500 ms sleep after unmounting before claiming the drive.
- FMTS discs use the trusted CPS path; unit-key samples skip forensic
segment units. The parity_image_fmts_alternate_phase golden changes only
in request count, on a fixture title under two keys (not a real disc).
1.8.0 read up to 32 units from every stream file before asking any key
source: 1,472 random reads (~95 s over USB) on a 46-file UHD that declares
one CPS unit, answered by one keyserver request.
One declared unit: the main title's samples ask once and the key that
opens them keys every file. Several: one encrypted unit per file, then
each unopened unit asks with 8 of its own units; one answer may open many
files. A source answering no key is not asked again, and a disc no key
opens is refused at once (blanked in a whole-disc copy) rather than
probed. Discs with an FMTS layout in scope keep the probed path.
Key acquisition also logs its phase timings.
DriveInfo gains display_name: the drive letter on Windows, /dev/srN on
Linux and diskN on macOS while a disc is mounted, or the path when the OS
has no name. path stays the selector to open. Windows lists lettered drives
in letter order, then unlettered drives by CdRomN.
Name STORAGE_DEVICE_NUMBER's fields in snake_case (repr(C) layout is
unchanged) and compile the 36-byte INQUIRY retry only on Windows, which keeps
lint_stop's #[allow] baseline.
0b146ef7 moved decrypt.rs and sector/decrypting.rs tests into
decrypt_tests.rs, decrypt_spec_guards_tests.rs and sector/decrypting_tests.rs;
they are the same modules' tests, so they share the allow-path.
Linux also lists optical sr block devices that have no sg node (sg module
not loaded), and without sysfs reads /dev for sg nodes instead of probing
sg0..15. macOS grows its listing buffer instead of stopping at 8 drives.
Changelog covers the Windows enumeration fix as well.
Report the selected IOKit service, plug-in metadata, each interface
initialization stage, exclusive-access attempts and cleanup results through
tracing, so a failed drive open shows which step failed.
Enumeration probed only \\.\CdRom0..15 and fell back to drive letters only
when that found nothing, so drives numbered above 15 (common after USB
replugging) were silently missing. List every CdRomN from QueryDosDeviceW,
add optical drive letters (GetDriveTypeW) not already covered by their
device number, retry INQUIRY with the standard 36 bytes for strict USB
bridges, and log each skipped device with its reason. Also syncs the stale
Cargo.lock with the 1.8.0 manifest.
DVD titles play the way their navigation packs say: interleaved cells
as their own units, chapter names from the VMG, region mask, IFO stream
labels, a whole-title forced-subtitle probe, DvdPs-only stages. UHD
keeps mastering metadata and hdr_plus_flag and fixes dvcC. HD DVD
decrypts AACS per pack by each EVOBU's CPI and joins PEVOB features;
Class II MKBs are reported.
MPEG-2, H.264, HEVC and VC-1 drop pictures whose references are missing
at a title start, clip join, gap resync or a trimmed clip mark; the
M2TS sink rides the clip-join timeline. An image extent's sub-unit tail
is judged by the decrypt stage instead of refused (E9030).
Wall-clock-sensitive tests hardened; monthly grouped dependabot.
Fixes and tests from the second audit across bdnav, labels, ifo, codec
parameter sets, mkv/mp4/stdio, css, aacs, keysource, keys, decrypt and
sector code. An HD DVD copy already in the clear needs no key, with one
encryption decision for every AACS format; image and folder titles are
named by the meta title, else the playlist, as in 1.7.7.
One run context (halt, events, loss counters) serves every stage; keys
come from KeyEvidence into a KeyRing. open_source opens every input,
disc:// included, replacing MuxSource; PesSource/PesSink replace
pes::Stream; one Read stage replaces DiscStream and the byte
prefetcher. iso://, null:// and dir:// are sinks. Dead error variants
(E9009, E9063, E9082) and scaffolding removed.
Every input decrypts through one stage that detects its content, with
strict stale-CPI handling and partial units only at EOF. Damage no
longer ends a rip where usable content remains (E7031, UDF FID,
MPLS/CLPI truncation, whole-disc E7032). M2TS output declares tracks in
an HDMV PMT and paces PCR; network:// allows LAN and loopback and
is_blocked_ip is public; parity goldens pin today's outputs.
MKV read-back carries display shape, colour, frame rate, content
encodings and BCP47 language, and drops malformed blocks with a count.
The FMKV header carries display aspect, CICP, labels, chapters and
format. Live reads refuse a zero read batch (E9085) and treat a stopped
read as a stop. freemkv-unlock becomes a version dep with a git-tag
override; dependency bumps (#27, #28); LPCM keeps its source bit depth; 1.8.0
changelog.
Skips duplicate packets, gathers PES headers that span packets, resyncs
after byte slips and across zero-filled gaps, prefers a current
CRC-valid PAT/PMT, and drops the open PES on transport_error_indicator.
M2TS output remaps PIDs into the BD ranges.
E9077-E9084 for engine remux and preflight refusals; MuxOptions loses
its send deadline. The MKV writer drops unwrap/expect and overflow
paths, tags SDR BT.2020 correctly, leaves out streams with no Matroska
CodecID and derives the Dolby Vision level. On-arrival damage with no
vouching partner is blanked, and a garbled head is damage, not a stop.
Lands the code-audit branches for aacs, css, keys, labels, udf, ifo and
dvd, bdnav/mpls/clpi, disc/bluray, extract, io, scsi, the mkv, mp4,
network and mpg muxers, the audio and video codecs, and CI. Fixes
include MKB and FMTS validation, UDF bounds, IFO fallback to the BUP,
DTS Express framing, MPEG-2 field pairing and mpg re-basing; the rest
pins existing behaviour with tests.
Damaged AACS units are holed, blanked and counted on every path instead
of stopping with E7013; failed batch partners fall back to side reads
and a wrong FMTS key is judged per key. ResolvedKeySet is the only AACS
decrypt door; a raw whole-disc reader never decrypts and refuses titles
with no stream folder (E6003) or a piece no held key opens (E7032).
Key-source contexts carry the op token and progress; DiscSession gains
open_with, scan_with and finish on the held handle. A consumer that
closes after a cancel keeps *.partial, and an artifact lock sidecar
guards each final output. Stop rules are enforced as lint tests.
A container-neutral fit plan, then mpg:// as an H.222.0 program-stream
sink on the PES IR and as a source through the DVD sector pipeline. PES
and AU placement, start-up hold caps and 20/24-bit LPCM timing follow
the spec; a clear MPEG-1/2 PS is never read as scrambled and CSS is
judged per pack. E9074 for an output with no video track.
ResolvedKeySet resolves keys up front, forensic keys included, and every
read surface decrypts through it. A keyless session never muxes
ciphertext; a single declared key keys only the pieces it opens; held
keys are tried before side reads. Resolve reports whether the disc's
VID would help, returns its trace on refusal, and never codes an
unreachable key service as Missing.
Waits are halt-aware on a stall timer; every CDB carries the op token;
prefetchers and pipeline joins honour Stop; flush is durable with
measured progress; the macOS shim open is cancellable. Also: CPI is
cleared on every decrypted unit, FMTS phase verification, the HD DVD
declared CPS count, AES work hoisted out of AACS hot paths, and the
last production unsafe removed.
A bus-encrypted stream file whose File Entry cannot be located is
re-read, then recorded rather than failing the scan; MKV staging ranges
and E6022 for a partial image. Audio framing chains headers while
resyncing, reports every lost slot, keeps good frames beside corrupt
ones and places locks by PTS, with ADTS and MPEG property fuzzing.
libfreemkv::spec holds the verbatim specification quotes the code
cites, with guards tying each governed site to its quote. A test-util
feature exposes shared fixtures to downstream tests. M2TS output
derives DTS for reordered video and rounds to whole 90 kHz ticks.
DVD subtitles and audio route by PGC SPST_CTL and AST_CTL instead of
stream ordinal, and MPEG audio is fed from its PES id. The DVD MPEG-2
extension stream rides the IR with its channel count verified by CRC.
Audio framing stops false syncs poisoning a track; PGS and DVD subtitle
parser fixes; E9073 and E7034 declared.
Scan reads UDF and the AACS files before the drive handshake and records
a missing DVD /AACS key file instead of failing. Decryption needs an
explicit AACS unit base and refuses off-grid units; an all-broken
host-cert list is its own error (E7033). Every whole-disc and image to
ISO copy goes through one shared reader that refuses an unkeyable file
up front (E7032). HD DVD XPL tick and timecode fixes; CI action bumps.
Fixes from the mkv, mp4, network and ts sink audit; track timing and
DiscardPadding ride the FMKV wire. MP4 writes DTS per ETSI TS 102 114
(ddts rates and timescales). The MKV PCM depth probe is bounded and
evidence-based. Network receives use keepalive and wait on readiness,
so a Stop or a dead peer ends them.
Disc presence is tri-state per MMC-6, an empty drive fails fast, a
cleaning cartridge reads as no disc, and the tray is unlocked only when
we locked it. BD and DVD LPCM decode to plain PCM, with whole-frame DVD
blocks and the BD layout carried through M2TS.
DVD and HDMV navigation commands follow the reference semantics
(SetGPRMMD, set-links, saturating ADD/MUL, unmasked shifts). Label
detection bounds the BD-J menu walk, opens large jars in place and
tightens Paramount, Deluxe and BD-J feature evidence. The CLPI audit
keys streams by clip and PID; channel layouts keep the LFE split and
TrueHD counts for unnamed layouts.
Read-time key fetches are validated and source, halt and drive faults
surface. Every BDMV/STREAM file is bus-decrypted per the BEE flag and
unit copy-permission bits; a missing or unparseable cert fails safe.
Extract treats a drive Stop as a halt, not a hole. io latches writeback
errors, fixes the read eviction window and never waits on writeback
during seeks.
UDF validates allocation-descriptor chains, maps fragmented metadata
files and tolerates broken subdirectory chains. The muxer holds headers
until every AAC track has its AudioSpecificConfig and backpatches a late
CodecPrivate. HEVC PPS tables are bounded, DTS EXSS aligned and MPEG
free-format frames capped. A no-media drive is released before the next
candidate opens; CI builds freemkv's app as a consumer.
Empty-tray optical drives couldn't be opened because the macOS backend
resolved drives by media-child `BSD Name`, which doesn't exist without a
disc loaded. Result: `list_drives()` returned nothing for a drive with no
media, and any attempt to `open("disc://")` on that state failed.
Fix: enumerate `IOBDServices` directly. With media present, the returned
selector is the familiar `/dev/diskN` path (unchanged). With no media,
the selector is an opaque `ioreg:<registry-id>` string that resolves to
the same IOBDServices entry — Disk Arbitration and unmount are skipped
because there's no media node.
- `src/drive/macos.rs`, `src/drive/mod.rs`, `src/scsi/mod.rs`: threads
the selector through the transport so empty-tray drives are openable
without punning it as a filesystem path.
- `src/scsi/macos.rs`, `src/scsi/macos_shim.c`: the C shim now accepts
both `/dev/diskN` and `ioreg:<id>` in `shim_open_exclusive`, and
`shim_list_drives` reports the correct selector for each drive.
- `tests/macos_drive_tests.rs` (new): covers empty-tray discovery.
- `docs/drive-access.md`, `docs/scsi-macos.md`: document the two-path
selector model.
[skip-media]
The 1.7.6 unified release ships no rip-path change (Linux desktop shell,
notifications, menu_layout refactor, XDG paths). Real-media QA on EC2
adds ~2h + real money to the release cycle to prove code that never
enters the rip pipeline. This empty commit puts `[skip-media]` in qa's
HEAD so the media matrix in qa.yml auto-skips per the documented bypass;
the full-platform builds/tests still run.
Same fix as the other 7 repos: qa held two release-orchestrate commits
(v1.7.5 bump + post-release freemkv-unlock path restore) that never
propagated back to dev, so promote's fast-forward gate refused. Ours-merge
records qa as an ancestor without applying its content; dev has already
moved past 1.7.5 to 1.7.6.
The freemkv 1.7.6 Linux target compiles a GTK4 + libadwaita desktop shell,
so this job's downstream `cargo test`/`cargo check` on freemkv hits
`gtk4-sys`/`glib-sys` pkg-config probes that fail on a stock ubuntu
runner. Adds the apt-get step alongside the existing sibling checkout,
mirroring the fix in freemkv's own ci.yml `cli-linux` job.
Kept local (not in the shared reusable checkout action) — only `freemkv`
links against GTK; adding the deps to every downstream consumer would tax
the six that don't.
Bumps Cargo.toml + Cargo.lock to 1.7.6. Also fixes a CI-shape defect the
previous release left behind: dev's CHANGELOG still said
`## [1.7.5] — UNRELEASED` after 1.7.5 shipped (release-orchestrate dated
main but did not push a fresh UNRELEASED heading back to dev). Corrects
that here so the changelog job accepts 1.7.6 on the way to qa:
## [1.7.6] — UNRELEASED
- Version aligned; unified release driven by freemkv 1.7.6.
## [1.7.5] — 2026-09-23
- Version aligned (or actual content, per the shipped main entry).
The driver for 1.7.6 is the freemkv Linux desktop shell (GTK4 +
libadwaita) + rip-finished desktop notification (issue #56).
Version aligned to 1.7.5. No functional change to this crate — the release
is driven by freemkv-unlock mirroring the freemkv-firmware 0.9.0 ABI, where
the drive's separate Ake (0x06) and Bus (0x07) levers are retired into a
single Encryption (0x06) lever.
Convert qa.yml to the central pattern already applied to ci.yml (86d66d3):
- add top-level concurrency (cancel-in-progress) group
- replace per-job freemkv-unlock checkout (ref qa) with the
freemkv/.github checkout-siblings@v1 composite (checkout: freemkv-unlock)
in release-tests, cross-lint and windows-build
- standardize actions/checkout to @v5
No heredoc patching existed in any job, so no patch inputs. The composite
resolves siblings 3-way, so qa runs now pin siblings at qa automatically.
Replaces the 8x-repeated sibling-checkout block with freemkv/.github checkout-siblings@v1.
Adds concurrency cancel-in-progress. All jobs (incl. consumers running dependent suites) preserved.
Thin callers to freemkv/.github@v1; delete the now-duplicated local ci/ scripts.
Adds concurrency cancel-in-progress and drops the push+PR double-fire (fork PRs
still run). doc-links now resolves toolchain from rust-version (was hardcoded 1.98).
Test gaps (guarded code was untested):
- disc: cover is_complete_feature_presentation's CHAPTER_SPAN_MIN_FRAC (0.5)
boundary — clustered marks (span < 0.5*dur) are not a feature; marks at the
0.5 threshold are. Adds bd_title_chaptered_span helper (issue #45).
- hevc: exercise the length-prefix self-check WIRING in HevcParser::parse
end-to-end — a desynced access unit is dropped, a well-formed one emitted.
Adds a test-only FORCE_FRAMING_DESYNC seam (frame_data is length-prefixed by
construction, so no real input desyncs it) (issue #52).
- labels: cover resolve_feature_hint's winner_hint early-return — a non-empty
winner_hint wins over the Tier-2 bdj_feature menu-walk result (issue #46).
Doc drift:
- keysource.md: resolve_unit_keys + UnitKeyResolution (matched/miss_path/
matched_entry/store_entries de-conflation).
- aacs-trace.md: MatchedEntry, KeyStep.matched_entry/store_entries, and the
KeyNode::NoDerivableKey node.
- hevc.md: add the "length-prefix self-check (issue #52)" section the hevc.rs
comment points to.
Craft:
- labels: hoist the duplicated MIN_FEATURE_SECS (60) from fox.rs and
paramount.rs to a single shared const in labels/mod.rs.
Adds a generic BD-J feature-playlist resolver that runs when no framework
parser surfaced a manifest hint, plus a refinement to Sony/Fox feature
selection. The hint now rides an independent pass so it survives even when
the winning parser (or no parser) produced any labels.
- labels/mod.rs: resolve_feature_hint() independent hint pass wired into
apply(); register mod bdj_feature.
- labels/paramount.rs (+ fox.rs mirror): refine feature selection —
exact name="Feature" → /feature/i by longest duration → most-audio by
duration; hard-reject sub-minute playlists (kills the _Start_Angle 2s
decoy). Fixes Sony SM3 UHD. feature_hint_from_xml() exposed for Tier-1.
- labels/bdj_feature.rs (NEW): Tier-1 jar-space resource sweep
(dcx.xml/playlists.xml/*.properties) reusing fox/paramount hints; Tier-2
autostart-Xlet locator scan (BDJO AMT → autostart jar → NNNNN.mpls /
PLAYLIST:N String constants → intersect real PLAYLIST set →
longest-dominant multi-audio wins; integer-only never emits).
- bdnav/bdjo.rs (NEW): minimal bounds-checked /BDMV/BDJO/*.bdjo
Application Management Table reader (autostart initial_class + jar ids),
panic-free like bdnav/index.rs.
- labels/jar.rs: try_each_resource() helper for the Tier-1 sweep.
No changes to disc/mod.rs selection or bdnav/vm.rs.
Fix (B) — the reproducible bug behind ffmpeg "Timestamps are unset in a
packet for stream N" on PGS subtitle tracks (worst on the sparse forced
track). Subtitle tracks carry no DefaultDuration, so a frame emitted with
duration_ns: None became a bare SimpleBlock with nothing to bound the cue.
- codec/pgs.rs: synthesize a fallback dwell (DEFAULT_PGS_DURATION_NS, 5 s)
for every emission that was duration_ns: None — the EOS trailing set and
the no-PTS clear/display, truncated, and lone-pass-through arms. emit_pending
keeps its real computed span but clamps a pathologically large one
(missing intermediate PCS) past MAX_PGS_DURATION_NS back to the fallback.
- mkv.rs: track_is_subtitle (track_type == 17), mirroring track_is_video. In
write_frame_at a subtitle frame with no duration now substitutes a minimum
fallback (MIN_SUBTITLE_BLOCK_DURATION_NS, 1 s) so it is emitted as a
BlockGroup WITH BlockDuration, never a bare SimpleBlock (defense in depth).
Fix (A) — defensive HEVC guard (the "Invalid NAL unit size" error is not
reproducible in our clean disc->MKV path; this makes our output structurally
incapable of originating it):
- codec/hevc.rs: length_prefix_tiles() self-consistency check — the
[u32-BE len][body] records must EXACTLY tile the frame; on mismatch the
desynced access unit is dropped (logging src position) rather than emitted.
- LENGTH_PREFIX_SIZE constant + regression test that hvcC byte-21
lengthSizeMinusOne+1 equals the prefix width push_length_prefixed writes.
Tests: pgs.rs None-arm duration assertions updated to the fallback + a
clamp/no-clamp pair; mkv.rs subtitle-BlockGroup and two-track (dense+sparse)
"every block has a BlockDuration" cases; hevc.rs tiling accept/reject,
byte-21 regression, and a positive parsed-AU-tiles-exactly test.
A BD-J disc (Alita) has no authoritative static playlist chain, so main-
feature selection is a structural failsafe. Its chaptered feature 00800 is
a SUPERSET of an un-chaptered body subset 00703 that is 0.97 of its size,
so the size-only seamless bumper limb fired and demoted the real feature,
ripping the body instead.
Gate ONLY the bumper limb on is_complete_feature_presentation(q): the
wrapped/subset title must carry a real chapter table (>= MIN_FEATURE_CHAPTERS
marks spanning >= CHAPTER_SPAN_MIN_FRAC of its runtime). The concat limb
(much-shorter cut) is untouched. This is the ABSOLUTE-threshold predicate;
a one-line swap to the RELATIVE form (p.chapters.len() <= q.chapters.len())
is documented at the call site and in the predicate doc, pending a hoard
sweep of SM3's real chapter counts.
Also add lowest playlist_id as the final canonical_title_order tie-break so
equal-size seamless siblings (00800-00808) pick 00800 deterministically.
Fixtures re-based on the real chapter discriminator (not a size coincidence)
and new tests cover both topologies (Alita superset-feature, SM3 subset-
feature) plus the un-chaptered-subset keep-superset case.
The keydb lookup returned an empty Vec for two very different outcomes:
a true hash miss, and a MATCHED disc that derived no key (e.g. an MK-only
entry with no VID). Both rendered as `keydb > no entry`, misleading
reporters whose disc IS present in the keydb.
Add KeySource::resolve_unit_keys -> UnitKeyResolution { keys, matched,
miss_path, matched_entry, store_entries }; the trait default delegates to
get_unit_keys (matched=false) so every existing source is unchanged.
resolve_and_apply_traced now renders MatchedDisc + NoVid/NoDerivableKey
for a matched-but-no-key disc and reserves NoEntry for a genuine miss.
New KeyNode::NoDerivableKey, a MatchedEntry shape (booleans + lengths, no
key material), and KeyStep.{matched_entry, store_entries} carry the
diagnostics to the app-layer renderer. Tests cover both verdict paths.
Bus content-range de-bus:
- F1 disc/mod.rs: always install content ranges (extracted wire_bus_removal);
empty ranges => Some([]) => de-bus NOTHING (fail-safe), not None => de-bus
everything — closes silent clear-sector corruption on the AACS cert route.
- F4 aacs/content.rs, F5 decrypt.rs: content gate is now a partition_point
binary search (O(log n)/sector) over the sorted, merged-disjoint ranges,
replacing the per-sector O(ranges) linear scan on the de-bus hot path.
- F2 scsi/linux.rs: RECOVERY_THREADS cap uses atomic reserve-then-check
(fetch_add first, fall back on overshoot) — removes the TOCTOU that let
concurrent multi-drive rips exceed MAX_RECOVERY_THREADS.
- F3 pes.rs: PesFrame::deserialize reads each chunk straight into data's tail;
drops the scratch Vec + double-copy, keeps the incremental-growth DoS guard.
- F7 disc/mod.rs: six inline rank_titles thresholds -> named consts.
Tests: bus fail-safe + scan wiring (both routes), per-sector gate straddle,
read_directory type-3 continuation/AED-skip/cycle-guard, strengthened
without_bus_removal no-op guard. F6 (pub decrypt_bus) left by-design.
Relocate AACS bus decryption to one spot at the bottom of the sector chain:
the Drive's SectorSource impl applies decrypt_bus_in_content (gated to
encrypted-content ranges), with the stage decided once from the unlock/
handshake result (BusStage::from_read_data_key). Remove read_data_key from
DecryptKeys::Aacs and AacsState and the two downstream decrypt_bus call sites;
every reader (key sampler, mux, unit-key validation) now sees already-de-bussed
content and never reasons about bus encryption. Add BusRemovalSectorSource
stream + tests; keep the AacsBusKeyUnavailable fail-loud gate.
The PlayItem STN (stream-number) table offset was hard-coded to 32, which
is only correct for a single-angle PlayItem. A multi-angle PlayItem inserts
number_of_angles(1) + flags(1) + (number_of_angles-1) angle references (10
bytes each) right after still_time, so its STN begins at 32 + 2 +
(number_of_angles-1)*10. The fixed offset lands inside that angle block and
misreads the stream counts as garbage (e.g. n_video=51 on Spider-Man 3 UHD
00245/00246.mpls, number_of_angles=2 => real offset 44), dropping every
stream including the primary video. That makes DiscTitle::has_video() return
false, which drops a multi-angle/seamless-branch feature from the disc's nav
candidate set and loses it in main-title selection.
Compute the STN offset from is_multi_angle (bit 4 of item[10]) + number_of
_angles. Add a regression test building a 2-angle PlayItem and asserting the
video stream is found past the angle block.
Validated on a 92-disc real UHD/BD corpus (offline scan): 0 main-feature
pick changes vs baseline, 3 multi-angle titles (Spider-Man 3 244/245/246)
recover has_video. Adds examples/corpus_titledump.rs as the regression tool.
The fallback composite gate in rank_titles flagged title P as a "play-all
wrapper" whenever some other title Q's clip set was a proper subset of P's,
Q had video, and Q.size >= 0.5 * P.size. main_feature_order then ranks
standalone-before-composite, demoting P below the decoy Q.
On a seamless-branch feature this inverts the pick: the real 00800 (2h03,
4 clips = body + intro/credits/seamless segments) is a SUPERSET of the
single-clip body decoy 00700 (1h50), and 700 >= 50% of 800's size, so 800
was mis-flagged composite and lost to the shorter 700 — the wrong title
gets ripped.
Add a duration-first wrapper-shape gate: P is only a composite when the
subset Q is EITHER a much shorter cut (Q.duration < 0.85 * P.duration, a
concat/play-all padding Q's runtime) OR nearly size-equal (Q.size >= 0.90 *
P.size, a bumper/outro wrapper). A near-full-length but markedly smaller
body subset (Q runs ~0.89 of P's duration yet holds far fewer bytes,
because P's branch segments add bytes not runtime) is a real seamless
branch, so P is kept. Uses per-title duration_secs/size_bytes, stays
capacity-independent and feature-gate-safe.
Preserves the real play-all avoidance: the 00245 bumper-wrapper decoy
(Q ~0.96 of both duration and size) stays demoted via the size limb, and
a concat play-all (subset < half the runtime) via the duration limb.
Add regression tests: seamless_branch_feature_not_demoted_issue_45 and
true_play_all_concat_wrapper_still_demoted. All existing selection tests
pass.
Bring every flagged comment block within its cap (inline // <=3 lines,
/// item-doc prose <=8): tighten wording, or convert an over-long inline
block that documents the item below it into a /// doc, preserving all
technical facts. No code, string literals, or logic changed.
Probe dir_is_case_insensitive against the real target volume by creating
the dir first. Previously the test probed a not-yet-created dir, which
returns a conservative true; that disagreed with extract_tree's own
post-mkdir probe on a case-sensitive host (Linux CI), panicking the test.
Both branches (collision on insensitive, clean coexist on sensitive) still
verified.
- decrypt.rs: three refs to the #[cfg(test)]-only decrypt_sectors_mapped
wrapper can't resolve in a non-test doc build; de-link to inline code
(keeps the name, no broken link).
- Fix DecryptingSectorSource::with_content_ranges link to its full
crate::sector path so it resolves.
READ CAPACITY is the only authoritative whole-disc size for an image
(the UDF partition is a subset — measured 288 sectors / ~576 KB short on
a real BD, which would truncate the backup anchor). Its field failures
are sporadic (5/5 reliable on the target drive), so retry it a few times
rather than substituting a short value. On persistent failure the shared
scan/identify path stays lenient (capacity 0; MKV reads by extent) and
image_read_sectors() returns EmptyImage so an ISO fails cleanly instead
of writing a truncated or empty file.
shim_list_drives: reorder the drain loop condition so IOIteratorNext is
not called on the exit iteration, releasing every obtained io_service_t.
shim_open_exclusive: the fallback matcher ignored the requested bsd_name
and could open a different drive; iterate and accept only the candidate
whose IOMedia BSD name matches, else fail. Correct the misleading da_hold
TOCTOU comment.
class_reader: checked pc + size add (guards 32-bit overflow) and remove
the dead BadInstruction variant.
criterion: dedup duplicate stream numbers instead of emitting a number
twice. deluxe: continue past a bad studio attribute instead of aborting
the whole parse. bdmt: treat mixed-content XML (embedded markup) as XML
so it is dropped, without over-rejecting a bare '<' in prose.
pes: grow the frame buffer incrementally instead of an eager
MAX_FRAME_SIZE allocation. extract: give the case-probe a unique random
suffix so concurrent probes cannot collide (TOCTOU-safe).
network: add the NAT64 (64:ff9b::/96) arm to the IP classifier -
extract the embedded IPv4 from the last 32 bits and re-check it, matching
the keysources canonical copy, so a NAT64-wrapped private/special
address is blocked like the bare IPv4.
error: add Display arms for DirInsufficientSpace (E9027), DirNameCollision
(E9028), DirWriteFailed (E9029) so their fields render (codes-only, like
the sibling Dir variants).
hex: reject a leading + in the hex int parser.
inf: read_mkb_from_drive used .is_ok() after the first pack, silently
truncating the MKB on a mid-walk transport error; propagate the error
instead. Add checked offset guards (94/117) and reject a truncated title
table (161) like the key list already does.
segment: the FMTS key-range carry assumed 2048-aligned segment starts;
use the floor-difference of the actual LBAs so a non-2048-aligned
segment carries correctly.
Annotate css::keyless::recover_title_key as a test-only helper and
correct the clpi_audit dedup-by-PID doc.
lpcm: carry last PTS + DTS fallback (add last_pts_ns) and propagate
pes.discontinuity instead of hardcoding false, matching adts/flac/
mpegaudio.
pesbuf: add an internal MAX_BUFFERED_BYTES cap (defense-in-depth like
reorder.rs's MAX_GOP) so a misbehaving caller can't grow it unbounded.
dts: drop a truncated trailing core at end-of-stream instead of emitting
it, while keeping genuine trailing extensions.
hevc: resolve num_extra_slice_header_bits from the PPS the slice
references (slice_pic_parameter_set_id), not the last-active PPS.
crc: remove the dead, misnamed crc8_atm (impl was CRC-8/SMBUS, no
callers). Add the missing drop-gate mismatch test; fix the startcode
read_bits contract doc and the h264 SEI comment.
mkvstream: add the consumed > remaining => MkvSourceInvalid guard (as
BLOCK_GROUP already had) to the INFO/TRACKS/TrackEntry/Video/Audio child
loops, and route the DURATION/SAMPLING_FREQUENCY float reads through the
checked_size cap instead of truncating cs as usize.
mp4/read: audio sample_rate was taken from the mdhd timescale, ignoring
the AudioSampleEntry 16.16 samplerate at the stsd body; read the real
field and fall back to timescale only when it is 0. Drop the unreachable
MAX_TRACKS break (the box scan is already capped) for a debug_assert.
tsmux + m2ts_mux: PTS origin was seeded on video only, so audio
preceding the first video (or an audio-only stream) rebased onto its own
PTS and lost A/V offset; seed the origin from the first frame of any
kind.
demux_sink: dropped_total() folded drops across ALL tracks while
frames_mapped now counts only persisted frames, so a filtered
(sub://audio://) multi-clip export could false-trip SeamPlanDroppedMost /
SinkWroteNothing. Compare drops and mapped frames over the same
persisted-track set via TimelineContinuity::dropped_for; update the
frames_mapped doc to the persisted-only semantics.
resolve: resolve_mux_key_map(...).ok() swallowed the error silently
(TrueHD channel correction skipped with no trace); log it like the
sibling reopen() fix before skipping.
meta_sink: TitleProfile::from_title dropped json:// stream fields
(pid, color/aspect/dimensions/interlaced/mvc, audio secondary/
sample_rate/purpose, subtitle pid/descriptive_service); restore them at
the json document level.
disc/encrypt: on the disc-keyed cert route, drive_unlocked was derived
from is_drive_unlocker(matched), which is provably always false -
run_bus only ever dispatches the disc unlockers (AACS/DVD), never a
firmware/drive unlocker. Hardcode false with an honest comment; the
genuine drive-unlock case (drive_unlocked, no read_data_key) is the
OEM-VID short-circuit and the bus-key gate already credits it. Gate the
now test-only is_drive_unlocker behind cfg(test) and add a regression
test through resolve_vid_only.
aacs/derive: resolve_dk_node looped 0..u_mask_shift shifting 1u32 << b,
which panics/wraps when the disc/keydb-controlled u_mask_shift is >= 32.
Cap the search at the 32 real bit positions.
drive: unlock_tray dropped its SCSI result with let _; warn! on failure
so a stuck medium-removal PREVENT lock is visible. Fix the stale
probe_disc doc (now a no-op; calibration moved into the unlocker).
DecryptingSectorSource::with_content_ranges was a silent no-op: the
proactive AACS key_map path early-returned before consulting the content
extents, and decrypt_span ignored its _content argument. A whole-disc
reader (sweep/patch) that installed both a key map and content ranges
therefore decrypted/verified clear UDF/BDMV-nav units, and its
misaligned filesystem reads tripped the AACS unit-alignment gate.
Thread the content extents through decrypt_sectors_mapped_in_content ->
decrypt_span -> apply_aacs_map, skipping any unit whose LBA falls outside
every content range (never decrypted, verified, or counted as loss), and
gate the alignment check on span_touches_content so out-of-content
filesystem reads pass through untouched.
Also drop the redundant is_multiple_of clause in the prefetched
unit-tail guard (remaining > 0 is already guaranteed).
Reverts the cross-repo E7031 addition: it forced enumerating the code across
freemkv test_support + freemkv-i18n's 29 locales for a LOW config-fault nicety.
keysources config faults keep borrowing the transient key-service code until a
coordinated i18n release; decrypt_bus-pub and the SSRF-parity additions stay.
- aacs/content: widen decrypt_bus to pub so freemkv-drivetest can invert
AACS 2.0 bus encryption out-of-tree (was pub(crate), broke that build).
- mux/network: port the 6to4 (2002::/16) and Teredo (2001:0000::/32)
tunnel-decode arms and the 198.18.0.0/15 + 192.0.0.0/24 blocks from the
canonical keysources SSRF classifier so both cover the same set.
- error: add distinct KeyServiceConfig variant (E_KEY_SERVICE_CONFIG 7031)
for the standing key-service misconfiguration fault, separate from the
transient KeyServiceUnavailable; keysources' online.rs adopts it later.
- dirimage: add a MAX_DEPTH nesting-cap test that walks a real
deeper-than-cap directory chain and expects DirImageTooLarge.
- udf: add a long_ad (ICB flags = 1) directory read test — the LBA sits
at AD offset 4, not 12 like extended_ad.
- labels/paramount: derive the feature hint's id and filename from one
parsed number (canonical 5-digit NNNNN.mpls) so they can't disagree,
the same fix already applied to fox, and add a hint-parse test.
Add unit tests for to_c_path (single trailing NUL, path bytes preserved
for libc::open) and resolve_to_sg (sg passthrough plus the no-filename
and non-sr/sg fallbacks that need no sysfs) — the helpers the open /
drive_has_disc / recovery reopen path relies on. Verified compiling for
x86_64-unknown-linux-gnu (this module is Linux-only).
- drive: SET CD SPEED and PREVENT/ALLOW MEDIUM REMOVAL are best-effort; a
drive rejection must warn, not fail the rip. New test drives a failing
transport through set_speed/lock_tray/unlock_tray.
- bdnav: CallObject/CallTitle follow the same play path as JumpObject/
JumpTitle for feature resolution; add tests for both.
- mkv: FlagInterlaced is corrected at finish() to the whole-stream scan
majority; add a test that a progressive leader on an interlaced feature
is patched from progressive to interlaced.
Add a direct Vm::set test exercising mov/add/sub/mul/div/mod/and/or/xor
and the divide- and modulo-by-zero guards (register left intact, no
panic) plus an unmodelled-op no-op, and a First-Play test that a
SetSystem instruction keeps resolution running so a following JumpTT
still resolves.
- bdnav/mobj: drop the unreachable MAX_CMDS cap (num_cmds is a u16 and the
span is already bounded against the input) and its flagged NOTE.
- drive/wait_ready: break out immediately on a dead bus (transport
failure) instead of polling a phantom for the full ~30 s; add a test.
- css/is_scrambled_pack: document that the 0x11 stream_id offset assumes
zero pack stuffing (true for DVD-Video VOB packs) and why a stuffed pack
degrades safely.
- scsi/macos_shim: document why releasing g_handle_lock before da_hold is
TOCTOU-safe (da_hold re-validates the handle under the lock).
passive_offset tagged a provisional offset with epoch_offsets.len() as
the epoch identity, but that length is capped at MAX_EPOCHS. Past the cap
the ring's remove+push leaves len() constant, so a provisional captured
then compared equal to the current identity forever and never retired,
drifting that track off every other one for the rest of the title. Add a
monotonic epoch_seq bumped once per open_epoch and tag provisionals with
it. Add a past-the-cap retirement test.
mpls: the STN table gate required item.len() > STN_OFFSET + 16, one byte
more than the 16-byte header actually spans, skipping a first play item
whose STN header ends exactly at the item boundary. Use >=, with a
boundary test.
ifo: the subtitle palette layout [padding, Y, Cr, Cb] (Cr BEFORE Cb) was
correct but under-documented; harden the comment against a future
byte-swap and add a test that cross-checks the parse against the DvdSub
ycbcr_to_rgb consumer so the two read Cr/Cb the same way.
fox: playlist_id and filename were derived independently from the raw id,
so a 5-digit id above u16::MAX left playlist_id None while the filename
stayed Some — a half-hint naming different things. Derive both from one
parsed number via a feature_hint() helper with the canonical NNNNN.mpls
filename, and test that the fields agree.
deluxe: parse_studio_attr anchored on a bare find("studio"), which also
matched substudio=/studioId= and the word 'studio' inside another
attribute's value, then read a later attribute's quotes as the studio.
Require a word boundary before it and '=' after, and add anchoring tests.
read_sectors fabricated zeros and reported success for any LBA, including
one at or beyond the image's own capacity — an oversized/crafted offset
that a real file/ISO source rejects. Refuse a read whose START is past
capacity (as read_exact does past EOF) while still zero-padding a batch
that starts in range and overshoots the end, which the mux's final batch
and the FS scan's fixed-LBA probes rely on. Split the read-range test to
pin both behaviours.
The TrueHD channel-correction probe's reopen closure returned Option<S>
via .ok(), collapsing a real io::Error into an indistinguishable None.
Make reopen return io::Result<S> and log the actual error on the failure
arm so a failed re-open is diagnosable, not silent.
unlocker_names() was a hand-maintained list that could drift from the
unlockers that actually dispatch, and the cert route hardcoded
drive_unlocked:false. Factor the firmware and disc unlocker sets into
single constructors, derive unlocker_names() and a new is_drive_unlocker()
from their real .name(), and set HandshakeResult.drive_unlocked from
whichever unlocker claimed the drive so the bus-key gate credits a drive
unlock even on the cert dispatch path. Add derivation and classification
tests.
The extract collision check always folded names to lowercase, so two
disc files differing only by case (Movie/movie) aborted the extract even
on a case-sensitive volume where they are distinct, coexisting files.
Probe the real target directory's case-sensitivity and fold only when the
host would (APFS/NTFS), defaulting to case-insensitive when the probe
can't run. Update the case-collision test to track the real volume, and
drop a dead duplicate pack-start write in the CSS cross-contamination test.
When a PTS-less PES arrived with pending_pts invalidated (e.g. after a
forced flush), the fallback fell straight to 0, snapping the timeline
backwards. Add continuation_base_ns() to reach for the projected next-AU
start, then the last PES front, before conceding 0, and stop stashing a
drained buffer's stale front mark as pending_pts. Add a fallback test.
read_directory read only the FIRST allocation descriptor of an out-of-
line directory, so a FID list spanning multiple extents was truncated at
the first boundary, and a type-3 continuation descriptor was mis-read as
FID data. Walk the whole descriptor list (types 0/1/2 as allocated FID
extents with the 2-bit type masked out of the length; type 3 followed as
a continuation block, bounded by MAX_AD_BLOCKS), gathering every extent
into one buffer while tracking the declared FID-list length separately
from the sector-rounded buffer. Bound the FID walk and the name read by
that declared length, and read a FID whose 38-byte header ends exactly at
the boundary (<=, not <) while still rejecting one whose name spills past
it. Add a multi-extent directory test.
The open-GOP keyframe promotion trusted only the first slice's coding
type and wasn't gated on the base view. A picture whose first slice is I
but whose later slices are P/B is not a clean random-access point, and an
intra MVC dependent view is never an independent anchor. Track every VCL
slice's coding type (all_vcl_intra) and require a real VCL slice, then
promote only when all slices are intra AND !mvc. Add mixed-slice and MVC
non-promotion tests.
The MAX_META_BYTES guard multiplied (dir_count + file_count) by the
sector size, counting one File Entry per node but omitting each
directory's FID-list data sectors. A wide, shallow tree's directory data
can dwarf the per-node entries, so an oversized folder could slip past
the guard and be materialized in full. Gate on metadata_block_count()
instead, which already sums File Entries and FID-list sectors, and add a
test proving the FID-list sectors are counted.
frames_mapped incremented for every mapped frame, including kind-filtered
ones that write to no track file (e.g. video during an audio:// export).
That inflated the denominator so a seam plan dropping every persisted
frame could ship zero-byte files at exit 0. Move the increment inside the
write block so it counts persisted frames only, and add a test where more
frames are seam-dropped than persisted -> SeamPlanDroppedMost fires.
ImageTruncated fell through to the bare-code catch-all, dropping the
have/want byte lengths a truncated-resume report needs. Add an explicit
arm emitting them in the identifier-free E{code} have/want form.
read_first propagated every non-UdfNotFound error immediately, so a
DiscRead on the primary Unit_Key_RO.inf aborted instead of trying the
/AACS/DUPLICATE/ backup that AACS ships for exactly this case. Treat a
DiscRead as retriable like an absent candidate: continue to the next
path, remembering the error, and only propagate it once every candidate
has been tried (still more informative than a bare AacsNoKeys). Add
fall-through and propagate-after-exhaustion tests.
find_boxes_capped used plain `pos + box_size` on an attacker-controlled
64-bit largesize cast to usize. A value near usize::MAX at a nonzero pos
wraps past the length guard, producing a release panic on the payload
slice or an infinite loop on the advance. Mirror read_moov: compute the
end once with checked_add, reject on overflow or overrun, and reuse it
for the guard, slice, and advance. Add the hostile-largesize test.
scsi::windows::list_drives delegated to crate::drive::windows::find_drives(),
but the drive and identity modules are #[cfg(feature = "rip")]. A scsi-only
build (default-features = false, features = ["scsi"]) — which is exactly how
freemkv-firmware consumes libfreemkv — therefore failed to compile: 'could not
find drive in the crate root'. The --all-features CI never saw it (rip is on).
Enumerate directly at the SCSI layer via scsi::open + scsi::inquiry, mirroring
the Linux and macOS backends, which already build their DriveInfo without the
rip-gated modules. The rip-side drive::windows::find_drives() is untouched.
Verified: cargo check --target x86_64-pc-windows-msvc passes both scsi-only and
with default features.
A drive claimed by a firmware unlocker (freemkv/LD/Renesas) at init() is that
unlocker's sole VID source. When it stashed no VID, do_handshake_cert now returns
(None, None) instead of running the AACS cert route (run_bus's AacsUnlocker) whose
REPORT KEY AGID cycle poisons the drive's AGID selector so every later bare 0xAD
VID read aborts until the medium is reloaded. A key source can still supply the key.
Adds a red-before-green test proving the guard issues zero SCSI on that path.
The dev branch tip must path-dep ../freemkv-unlock so CI's sibling checkout
(and local dev) resolves the branch tip; release.sh swaps it to a git tag
only in the tagged commit. dev had been left in the released git-tag
(v1.6.14) form after the last release, so the new 1.7.0 unlock API couldn't
resolve. Restore the path form the comment already describes.
Adopt freemkv-unlock's new Unlocker::unlock(). The bridge assembles the
firmware set (freemkv / LD / Renesas) for drive-prep and the disc set
(AACS cert + DVD) for bus removal, injecting host certs into the AACS
unlocker at construction — the loop no longer knows any unlocker's guts.
- run_features / run_bus now loop unlock() and return Ok(Option<Unlocked>);
drive init and do_handshake_cert adapt to the Option.
- unlocker_matrix reports the LD name and credits Renesas as a drive-level
(raw-read) route, so AACS shows "no" when any firmware route took the bus.
- Cert route sets HandshakeResult.drive_unlocked=false: the bus is removed
via read_data_key (AKE), not at the drive; the gate credits it that way.
Fixed
- mux/mp4: honour 64-bit largesize (size==1) and size==0 (run-to-end) in the
capped child-box scan; a largesize sibling previously ended the walk early and
later boxes vanished from a MOV/MP4.
Version 1.6.15 (unreleased) -> 1.7.0; changelog folded under 1.7.0.
Earlier 1.7.0 work remains as its own commits below this one: the
GHSA-j8ww-f5fg-9pmh sector-recycle soundness fix, CI/lint hardening (#15/#17),
docs, and Rob Ekl's scsi raw_command / fd-handoff change (#16).
raw_command clamped CDBs with cdb.len().min(16) instead of routing through
the shared checked_cdb_len guard, so an over-length or empty CDB became a
different command rather than a rejected one; it now returns the crate Result
so a rejected CDB is distinguishable from an I/O failure.
The recovery/Drop fd hand-off lacked a release edge: the teardown swap was not
a release op and the recovery CAS was not an acquire load, so a recovery thread
publishing its fd after Drop drained the slot could miss dead==true and leak the
descriptor. Both sides are now AcqRel, with the execute() drain left at Acquire
(sound via the release-sequence rule). A loom model (cfg(loom) dev-dep, its own
CI job) provides the regression coverage cargo test cannot.
Module docs for fd_handoff and timeline now resolve in their own scope, and item
docs stay within comment-guard's caps.
Fixes#16.
Declared rust-version and the ci.yml/qa.yml toolchain pins both sit at 1.88 (the
edition-2024 floor once zip 8 and let-chains are accounted for), so CI builds on
exactly the MSRV: an accidental bump to a newer toolchain fails CI by design and
moving the floor takes a deliberate pin edit. The uninlined_format_args allow
reconciles 1.88's clippy with the allow-by-default it became in 1.97+.
Correctness/robustness:
- scsi/macos_shim: guard g_handle / g_da_bsd read-modify-write with a
process-global mutex (scoped OUT of the 5 s async-claim wait) — closes the
DiskArbitration global-state TOCTOU on concurrent/re-entrant open+close.
- disc/profile: the public DiscProfile now surfaces `encrypted` and a numeric
`key_error` (Error::code of aacs/css failure) so a disc with failed key
resolution no longer serializes identically to a rippable one.
Performance:
- disc/dvd: read VIDEO_TS.IFO once and reuse it for both parse_vmg and
resolve_main_title (parse_vmg_with / resolve_main_title gain an optional
vmg_bytes) — eliminates a duplicate VMGI read on every DVD scan.
Tests:
- bdnav: MAX_OBJECTS count-cap rejection, MAX_STEPS non-convergence guard, and
an end-to-end index.bdmv + MovieObject.bdmv resolver test (all red-green).
- drive/capture: cover the Renesas "SAT"-signature capture gate (match + miss).
Craft/docs:
- css/keyless: factor the duplicated LFSR0 output-tap into one helper.
- drive/capture: name the Renesas vendor CDB constants.
- labels/deluxe: use the named ANEWARRAY/NEWARRAY opcodes, not bare hex.
- dvdnav/vmcmd: tighten pub -> pub(crate). dvdnav/mod: mark the parked stub.
- disc/bluray: document that whole-clip extents are intentional (trimming is
PTS-level in SeamPlan). disc/mod + error: doc/comment drift corrected.
- scsi/macos_shim: the DA claim used a stack DAClaimResult passed to an async
callback and never released its semaphore — on the 5 s timeout that is both a
use-after-free (the callback outlives da_hold) and a semaphore leak per drive
open. Heap-allocate it, reference-count it (rc=2), and let whichever side
finishes last release the semaphore and free. (Compiles; needs on-hardware
DA validation.)
- bdnav/index: add a regression test that a title count over MAX_TITLES is
rejected before any record read (untrusted index.bdmv count).
- dvdnav/nav: share ifo::MAX_TT_SRPT_TITLES instead of a second private copy so
the two 99-title caps can never diverge.
Correctness:
- dvd: join the nav main-feature target on the REAL vts_title_num (carried on
DvdTitle) instead of the title's position in the vec, which desyncs when a
sibling PGC is dropped as unparseable or TT_SRPT is non-monotonic.
- diag: the "demoted: no-video" dump now gates on has_probable_video (the gate
main_feature_order actually uses), not the stricter has_video.
Robustness:
- labels/deluxe: clinit_enum_field_names now caps retained bytes
(MAX_CLINIT_LDC_BYTES), like its sibling, so a malicious BD-J .class with a
huge CONSTANT_Utf8 field name can't grow the output unbounded.
Mux (FlagInterlaced correction cleanup):
- factor the PGS/FlagInterlaced deferred byte patches through one patch_bytes
helper; name the InterlacedRewrite record; add ebml::void_element (validated)
for the demotion Void; bound the muxed_* test scans to the pre-Cluster region.
Tests:
- add the FlagInterlaced majority tie-break test (tie stays progressive).
- de-vacuum the profile subtitle-count and meta_sink channel-count assertions.
Docs:
- clpi module doc described an EP-map/sector-range parser that was never
implemented; corrected to what the code actually reads.
FlagInterlaced was written from only the FIRST coded picture, but MPEG-2
progressive_frame is a per-picture flag: a progressive leader on an
interlaced feature (or an interlaced first picture on progressive film)
mislabeled the whole track, so players either deinterlaced progressive
frames or left interlaced frames combed.
Tally every picture's measured scan and correct the reserved FlagInterlaced
byte to the dominant scan at finish() -- the same reserve-then-patch idiom as
PgsForcedFixup and the AC-3/duration fixups. On a demotion to progressive the
now-stale FieldOrder element is Void'd. Also pins the interlaced direction at
the muxed-byte level and adds promote/demote regression tests.
Split the crate into two additive features so a transport-only consumer
(freemkv-flash) can depend on the SCSI passthrough layer without pulling the
ripping dependency tree (serde, zip, rayon, aes, freemkv-unlock, …):
- scsi = the SCSI passthrough transport (src/scsi/ + error): ScsiTransport,
CDB builders, per-OS backends + the macOS cc shim, sense parsing. Pulls only
tracing + the always-on libc target dep.
- rip = the full ripping tree (drive/disc/UDF/MPLS/CLPI/AACS/CSS/mux/keydb +
the freemkv-unlock seam) and all its heavy deps. rip implies scsi.
- default = ["rip", "scsi"], so the full build stays byte-for-byte identical for
the existing consumers; transport-only users set default-features = false,
features = ["scsi"].
All ripping modules are gated #[cfg(feature = "rip")]; error + scsi stay always
compiled. Two read-path timeout consts are rip-gated so a transport-only build
does not warn on unused.
MkvTrack::video sets `interlaced` from the source scan's DECLARED
resolution. An NTSC DVD's IFO declares 480i whether or not its pictures
are coded progressive -- film and animation routinely are -- so
progressive content ships as FlagInterlaced=1, and every player that
honours the flag runs a deinterlacer over progressive frames, softening
every frame of an otherwise bit-exact remux.
The measurement was already there and already trusted for the sibling
property. `apply_coding_to_track` receives the first coded picture's
PictureInfo and derives FieldOrder::Progressive from it
(progressive_sequence / progressive_frame, ISO/IEC 13818-2 6.3.5 /
6.3.10) -- it used that verdict only to clear FieldOrder and then
discarded it. It now also corrects the scan type, exactly as a measured
TFF/BFF already overrides the declaration.
Measured on a real NTSC DVD: ffmpeg's `idet` reports 100% progressive
(multi-frame detection: 0 TFF, 0 BFF, 0 repeated fields) on titles that
shipped FlagInterlaced=1. progressive_frame is set on every picture
coding extension in the stream while progressive_sequence stays 0, which
is the ordinary shape for a film or animation DVD.
Video payload is unaffected -- byte-identical before and after; only the
track header changes.
- Cap every inline `//` comment block at 3 lines; distil the rationale
rather than delete it. Enforced by a new comment-guard CI check.
- Fix stale and orphaned rustdoc: broken intra-doc links (renamed or
removed items), doc blocks that had merged onto the wrong item (leaving
the intended item undocumented), and doc-contradicts-code drift.
- Add a doc-links CI check (cargo doc, deny broken intra-doc links) so
both classes of comment rot are caught going forward.
The macOS SCSI transport took exclusive SCSI access plus a one-shot
`diskutil unmountDisk`, but never claimed the disc from DiskArbitration.
diskarbitrationd was therefore free to remount the volume mid-rip (a
Spotlight re-probe is the usual trigger), yanking the device out from
under an in-progress read; the failure surfaced to the user as E1000.
Hold a DADiskClaim plus a mount-approval dissenter scoped to our disc for
the transport lifetime, so the OS cannot remount it until shim_close().
Link the DiskArbitration framework in build.rs.
Refine the First-Play SPRM-taint abstain to fix two defects:
- Over-abstention: a tainted compare that guards a SetGPRM store (not a
control transfer) no longer forces abstention. The predicate only
decides whether the store runs; a following unconditional dispatch is
still resolvable. Taint now gates only JumpTT/Goto (the control
transfers the resolver follows); a store under a tainted guard taints
its destination GPRM (unknown post-state) instead of abstaining.
- Under-abstention: an SPRM laundered through a GPRM (SetGPRM g = SPRM,
then a later GPRM-only compare on g) previously evaded taint and let
the resolver commit to the cold-machine arm. Add per-GPRM taint bits:
a store propagates its source's taint (SPRM or tainted GPRM) and a
concrete/immediate store clears it; eval marks a compare tainted when
it reads a tainted GPRM.
Adds red-before-green tests for both plus taint-clear guards.
1. dvdnav/vmcmd: the register operand of the if_v1/if_v3 compare layouts is the
LOW byte of the reg-or-immediate field (b[5] / b[7]), not the high byte
(b[4] / b[6]). Fix both layouts and their comment; add register-mode KATs
with distinct high/low bytes.
2. mpls: connection_condition is the low nibble of PlayItem byte[10]
(byte[9] is fully reserved), corroborated by the byte-11 stc_id and
byte-12 in_time reads. Relocate every fixture's encoded value and update the
regression test to assert the reserved byte does not leak.
3. labels/deluxe: escape the two disc-derived class-name tracing fields via the
Debug path, matching the other log sites.
4. disc/dvd: add an integration test that drives scan_dvd_titles with a resolved
First-Play nav target so the 1-based promotion join is verified end-to-end.
5. bdnav/vm: add unit tests for the SET/SWAP immediate gate — full register
exchange, and that an immediate-flagged operand is never a write target.
6. dvdnav/nav: honor the documented contract — a First-Play compare that reads a
system parameter is undecidable at scan time, so eval now taints on SPRM reads
and the executor abstains rather than committing to the cold-power-on arm.
GPRM reads never taint.
7. labels/deluxe: narrow four now-internal helpers from pub(crate) to private.
Add an additive coverage job to CI that generates lcov via cargo llvm-cov
and uploads to Codecov (tokenless). Mirrors the test job's pinned 1.94
toolchain and dual freemkv-unlock checkout for the path dep. Informational
only: no gate depends on it. Adds codecov.yml and a README badge.
Audit round-2 (fix-adversary): the round-1 deluxe sweep fed the zip local-file-
header magic, but ZipArchive::new validates the end-of-central-directory at the
buffer's end — which the byte generators (front magic + noise) can never
synthesize — so the archive open always failed and decode_binding was never
reached: false coverage for the very bytecode-decoder panic path it guarded.
Drive decode_binding_class directly on a parsed .class with an empty master-enum
table instead, so the symbolic stack machine is actually exercised.
Continuing the /codeaudit HEAD..v1.6.10 round:
- disc/mod.rs rank_titles: bound the O(n^2 * clips) composite scan to a
sub-second work budget (both title count and per-title clip count are
untrusted disc metadata); above it, skip composite classification and degrade
to the has-video + canonical order. Reorder the per-pair checks so the O(len)
is_subset runs only after the cheap video/size gates. (regression test)
- bdnav/vm.rs SWAP: honour the imm_op1/imm_op2 store gates so a SWAP to an
immediate operand does not corrupt a register.
- harness.rs: fuzz the previously-uncovered new parsers — the Deluxe bytecode
decoder, the Fox dcx.xml scanner, and the combined bdnav index+mobj+VM resolve
path (labels::{deluxe,fox} and bdnav::vm made pub(crate) for in-crate fuzzing).
- dvdnav: pub(crate) the module + submodules (was pub) to match bdnav and stop
leaking the DVD IFO/VM internals as public API; drop an unused re-export.
- labels: escape disc-derived strings in diagnostics (Debug not Display) so a
hostile disc cannot inject terminal control sequences into logs.
From the /codeaudit HEAD..v1.6.10 round (each verified against the libbluray
HDMV VM reference, and design-refuted SOUND on two models):
- bdnav/vm.rs title_obj: resolve First Play via the spec sentinel 0xFFFF (was
num_titles+1). A JumpTitle 0xFFFF now reaches First Play, and an out-of-range
JumpTitle(N+1) correctly abstains instead of mis-jumping there.
- bdnav/vm.rs wr: refuse stores to PSR-tagged registers (matches _store_reg),
so an illegal SET/SWAP to a PSR is a no-op and later compares agree with a
real player instead of diverging.
- disc/profile.rs main_title(): return Option<&TitleProfile> instead of indexing
titles[0], which panicked on a disc that scanned to zero titles.
Regression tests added, each shown red-before-green.
The MSRV split raised the dependents above libfreemkv's own floor — the
bins (freemkv, autorip, bdemu) are 1.98 and freemkv-keysources is 1.97 —
so the consumers job could no longer build them at libfreemkv's 1.94
toolchain (`rustc 1.94.0 is not supported by ... freemkv-keysources
requires rustc 1.97`), leaving dev red since the MSRV change. Pin that one
job to 1.98, which satisfies every dependent's floor. libfreemkv's own
lib/lint/test jobs stay at its 1.94 MSRV.
A nav- or authoring-resolved playlist could promote a video-bearing but
negligible-payload branch above the substantial feature it sits beside:
Spotlight (2015) UHD navigates to a Dolby-Vision seamless-branch playlist
(00001, 0.4 GB across 102 tiny clips) instead of the 91.5 GB single-clip
feature (00002). A player plays the branch; a rip wants the feature.
Require a nav/authoring-promoted title to also carry at least 10% of the
largest video-bearing title's bytes; otherwise selection falls through to
the composite/size ranking, which recovers the real feature. Adds the
Spotlight-shaped regression test.
Verified against the 88-disc hoard: title selection is identical to the
prior release on every disc except the one it fixes (Spider-Man 3 UHD,
00245 wrapper decoy -> 00001 feature).
Also clears three 1.97 clippy lints in the HDMV VM and drops two
reference-implementation filename citations from the bdnav module docs.
Implement the DVD navigation feature resolver (#40): read the disc's own
First-Play program out of VMGI (VIDEO_TS.IFO) and execute it on a bounded
register machine to identify which title a player would land on, then prefer
that title in DVD main-feature selection.
- src/dvdnav/nav.rs: new module.
* Parses the VMGI First-Play PGC command table (FP_PGC pointer at VMGI+0x84,
PGC command-table pointer at PGC+0xE4, pre-command list) and TT_SRPT
(title -> VTS/VTS_TTN, pointer at VMGI+0xC4).
* Executes the pre-command list with the existing vmcmd decoder over 16 GPRMs
+ SPRMs from a cold machine: SetGPRM (mov/add/sub/mul/div/mod/and/or/xor),
compare predicates, Goto, Break/Exit, and JumpTT (title dispatch). Menu /
sub-domain jumps and intra-domain links are treated as undecidable.
* resolve_from_vmg(&[u8]) is pure and total; resolve_main_title(reader, udf)
reads the IFO and returns the resolved (title, vtsn, vts_ttn) or None.
* Every offset/count is bounds-checked; a step budget bounds non-convergence.
- src/disc/dvd.rs: scan_dvd_titles now resolves the nav main title and returns
its playlist_id alongside the titles; test call sites updated for the tuple.
- src/disc/mod.rs: the DVD nav pick feeds sort_titles_by_main_feature as
nav_feature, the same seam the BD path uses, guarded to a video-bearing
title. None -> today's leading-cell heuristic stands (strict improvement).
- src/harness.rs: add resolve_from_vmg to the never-panic sweep.
Tests: 11 nav unit tests (unconditional/conditional/set-then-dispatch convergence,
menu/empty/out-of-range/self-loop/truncation/overflow abstention) built to the
IFO spec using the real First-Play command bytes from the existing decoder KATs
(unconditional JumpTT 1; menu entry). Full suite green on 1.94.
Add a thin, format-agnostic typed VIEW over the scanned Disc model so
downstream reads `profile.titles[i].subtitles[j].forced` (and analogous
audio/video flags) for any disc with no per-format conditionals.
New src/disc/profile.rs:
- DiscProfile / TitleProfile / VideoTrack / AudioTrack / SubtitleTrack
(serde Serialize + Deserialize), re-exported from the disc module.
- DiscProfile::from_disc / From<&Disc>, Disc::profile(), and accessors
(DiscProfile::main_title, TitleProfile::video/audio/subtitles).
- Splits each title's Stream enum into three typed vectors; flattens
normalized fields (language default "und", forced/sdh from the subtitle
qualifier, commentary/descriptive from the audio purpose, label -> name).
- Hoists derived flags so downstream never recomputes: main_title (the
pre-sorted main feature at index 0), is_main, and the default
audio/video track (first non-secondary, keep only the first), subtitle
default always false.
Make the profile types the single serialization source of truth: the
json:// sink now serializes TitleProfile instead of hand-building JSON,
so json:// and any downstream share one schema. Stream arrays are split
by kind and qualifier/purpose decomposed into booleans; json://-only
extras (clips, full chapter_marks list) are spliced back on. meta_sink
tests reconciled to the new schema.
Tests: profile normalization across BdTs + MpegPs titles, typed-vector
split, populated fields, hoisted main/is_main/default flags,
qualifier/purpose mapping, and a serde round-trip.
The Deluxe parser detected com/bydeluxe discs but emitted nothing, so
mpls_universal won at Low. Grounding the decoder on real Universal discs
(Fast Five / F9 / Bourne, studio="uni") surfaced three defects that
kept the full Phase A-D pipeline from ever resolving a stream on any
real disc, all fixed here:
1. Master enums were keyed by the ZIP ENTRY name ("pd.class") but the
binding class references enum constants by JVM INTERNAL name
(getstatic pd.<field>), so Phase C's getstatic count never matched
the identified enums and found zero binding classes. Key master enums
by this_class_name instead.
2. MasterEnumTable resolved getstatic operands by enum VALUE string
("English") but the operand is the obfuscated static-FIELD name
("a"), so every language/purpose ordinal came back unresolved.
Capture the ordered putstatic field names (clinit_enum_field_names)
and key the resolver on them.
3. The Language-enum fingerprint demanded 70+-4 values; Universal's
enum has 65, so Phase A rejected it outright. Give each fingerprint
its own count window (Language widened to span 65 and 70); the
ordered 4+-string prefix stays the real discriminator.
Also: model anewarray/newarray so array construction keeps the symbolic
stack aligned, and skip constructors with an array parameter so the
Universal title-wrapper object (oq.<init>(...,[Lnp;,[Lwb;,...)) is not
mistaken for a subtitle binding. Recover SDH/RNIB from the language
display name (Universal encodes it as a distinct Language value, not a
Purpose). Add config.xml <TitleConfig studio="..."> detection. Raise
confidence to High when the binding table decodes >=1 stream; the
mpls_universal floor still back-fills un-named STN slots.
Fixtures: verbatim pd/lp/tl .class files from FastFive.iso decode to
English/French/Spanish + English Commentary (Dolby Digital) audio and
per-title eng/spa/fra subtitles. Validated end to end: labels-analyze on
FastFive/F9/Bourne now reports parser=deluxe High with real per-stream
languages, names, purposes and codecs instead of mpls_universal Low.
Treat Java bytecode as a documented format we READ (never execute), a
peer of mpls/clpi. The class reader's bounds checking was already sound
(typed Reader with checked u8/u16/u32/u64 + slice, saturating switch
arithmetic, iterator halt-on-truncation); this adds the untrusted-input
entry points the never-panic harness was missing and the malformed-input
regression tests the format warranted.
- harness: add class_reader (magic CAFEBABE) and jar (zip PK header)
sweep targets. The class_reader target also drives the accessors a
label parser exercises (this_class_name, member Code walk, cp_index,
member_ref, load_constant_display) so their indexing is fuzzed, not
just the structural parse. The jar target opens the sweep bytes as a
zip and walks every .class through for_each_class exactly as deluxe/dbp
do.
- class_reader tests: oversized constant_pool_count and interfaces_count
reject with UnexpectedEof (u16 counts keep with_capacity bounded);
out-of-range cp indices return None from every accessor; a Code
attribute whose declared code_length runs past its body is rejected.
scan_with runs the bdnav VM for BDMV discs and feeds the video-gated result
as the new top MAIN_FEATURE_ORDER key (nav-feature > authoring-feature >
standalone > has-video). A nav result is only honoured for a video-bearing
title, so a mis-resolve can never reopen the decoy hole; when the VM abstains
(BD-J, non-convergence) the structural/heuristic order decides exactly as
before. Adds nav selection tests and updates the diag decision reason.
Add src/bdnav: a libbluray-faithful, bounded, never-panic HDMV navigation VM
that runs First-Play with correct power-on PSR/GPR state to resolve the
playlist the disc itself plays as the feature. One generic VM handles the
obfuscated Sony dispatcher and every other HDMV disc with no vendor
special-casing; it abstains (None) on the BD-J boundary, malformed data, or
non-convergence. index/mobj parsers added to the never-panic harness.
Older Fox discs ship a loose plain-XML manifest at
/BDMV/JAR/<id>/dcx.xml (root <dcx>) that names every feature stream's
language, editorial purpose and forced/SDH state in attributes — fully
parseable and previously unowned (2 Fox discs matched no parser, 11 were
mis-owned by the deluxe stub).
New src/labels/fox.rs:
- detect(): loose dcx.xml under /BDMV/JAR/*/ (cheap dir walk), falling
back to a com/foxbd/ jar central-directory prefix so newer Fox discs
are attributed to Fox even though their bytecode form isn't decoded
yet (Phase 2, sketched in-file).
- parse(): scopes extraction to one <playlist name="feature"> element
(the richest, to avoid merging regional 00800/00801 STN tables),
reusing labels/xml.rs. audio type=rnib -> Descriptive, comment* ->
Commentary; subtitle form=sdh -> Sdh, type=embed -> Forced; nested
id="NN" -> 1-based per-type STN slot. High confidence.
- Feature playlist id recorded via tracing::debug for now; dev's
ParseResult has no feature_playlist field (that FeaturePlaylistHint
slot lives on selection-hardening). An in-code note documents wiring
it at reconciliation, matching paramount::parse.
Registered ahead of mpls_universal, alongside the High-confidence vendor
parsers; registry-order lock test updated.
Tests embed the real Life of Pi dcx.xml (/BDMV/JAR/05001), asserting the
00800 feature selection + id, all 11 audio labels (incl. the eng rnib
descriptive track) and 11 subtitle labels (sdh/forced/none), the
no-cross-playlist-merge scope rule, plus negative-detect cases.
The has-video gate alone fixed Spider-Man 3 UHD only by accident: the
00245 decoy = [bumper][00001 feature clip][outro], and streams come from
PlayItem 0 (the bumper) only, so it reported v=0. That is fragile — a
video-bearing composite [feature][outro] still beat the feature on size
(audit R2), and a real feature with a non-video first PlayItem would be
wrongly disqualified (R6).
Add structural clip-set composite detection: a title whose clip set
properly contains another substantial video title's whole clip set is a
play-all/wrapper composite and is demoted below the standalone title it
wraps — independent of stream counts, physical size, and disc capacity
(which is 0/unknown when READ CAPACITY fails, disabling the oversize
gate). Replace the has-video disqualification with the more-permissive
has_probable_video (real video, or size+duration+clips consistent with
video) so a bumper-lead feature is never disqualified while streamless
menus still are. Corroborate the authoring hint by duration (a hint at a
short branch/trailer can't override the feature) and require real video
streams for it. Precedence: authoring -> standalone-over-composite ->
has-video -> canonical (with a reserved top slot for the future BD-nav
key). Unify the session.rs key-sampling/verdict pickers and
image_crack_extents on has_probable_video.
Tests: video-bearing composite (T2), oversize composite with unknown
capacity (T3), short-branch hint not honoured (T4), bumper-lead feature
not disqualified (T6), all-streamless deterministic + single-source pick
(T7), equal-clip twins not composite, video-only title. T2/T3/T6 fail
under the prior has-video-only logic.
Main-feature selection ranked purely on physical size -> duration ->
audio (canonical_title_order), with no requirement that the chosen title
carry a decodable video stream. A playlist-obfuscation decoy can inflate
a streamless playlist past the real feature on size AND duration (Spider-
Man 3 UHD: 00245.mpls, 78.4GB/8350s/v=0 beat 00001.mpls, 76.5GB/8038s/
v=1), so the mux ran a title with zero streams -> zero frames -> E6008.
- Add Disc::main_feature_order layering two gates atop canonical_title_order:
authoring-feature (a title the disc's menu authoring designates AND that
has video) wins outright; then has-video demotes any streamless title
below every title that has video. scan_with re-sorts with it after labels
run so titles[0] is the selected feature. New MAIN_FEATURE_ORDER_KEYS +
DiscTitle::has_video().
- Surface the authoring feature-playlist id: labels::FeaturePlaylistHint,
captured from Paramount playlists.xml <playlist name="Feature" id=..>,
threaded through ParseResult/extract; labels::apply now returns it.
- Unify pickers so key-sampling and the structural verdict (session.rs) and
image_crack_extents prefer the has-video feature, not a streamless decoy.
- diag: decision row logs video= count + main_feature_order reason and a
tag=decision.demoted line per no-video title (closes the field-diagnosis
gap this incident exposed).
Tests: main_feature_order_demotes_streamless_decoy (pre-gate vs post-gate
contrast), _honours_authoring_hint, authoring_hint_never_promotes_a_
streamless_title, feature_playlist_hint_matches_on_id_or_filename; diag
reason test updated. QA green on 1.94 (fmt/clippy -D warnings/test).
Lower the crate MSRV from 1.97 to 1.94 -- the lowest toolchain on which
`cargo build`, `cargo clippy --all-targets -D warnings`, and
`cargo test --tests` all pass clean. Dependencies floor the build at 1.90
(freemkv-unlock 1.90, zip 1.88), but clippy is only warning-clean from
1.94 (older clippy has a `duplicated_attributes` false positive on
`#![cfg(test)]` and flags a now-documented `let_and_return`). Update
`rust-version` and every `dtolnay/rust-toolchain` CI pin to 1.94.0, and
add a CHANGELOG Unreleased entry.
Extend the byte-provenance coverage added in f7d38ca (currently one
TS-path test) with two PS-path tests driving the real `Mpeg2Parser`:
- `ps_stream_stamps_source_provenance`: the PS path forwards `ps.source`
onto `PesPacket`, so demuxed frames carry `Some(SourcePos)`.
- `ps_stream_carries_feed_base_across_reads`: the cumulative `fed_bytes`
base carries ACROSS read buffers -- a frame decoded from the second
sector is stamped `source.byte >= 2048`, not reset per read.
Also record the f7d38ca provenance fix in the CHANGELOG, inline a
`let_and_return` test binding, and trim a redundant sentence from
`PsDemuxer::feed_at`'s doc comment.
On direct disc reads (`disc://...`), `DiscStream` called `demuxer.feed(&self.read_buf[..bytes])`
instead of `feed_at(buf_base, ...)`. Without a running feed base offset, demuxed `PesPacket`s
had `source = None`, stripping byte provenance from downstream `PesFrame`s.
Under a `SeamPlan`, when `src_byte` is `None`, `SeamPlan::place` falls back to a timestamp
heuristic across clips. On multi-clip playlists where a subsequent clip starts at the same
base timestamp as the initial clip (`in_time = 600s`), B-frame PTS reorder dips (where
`raw_ns < last` and `|raw_ns - next_in| <= 250ms`) falsely triggered a `stepped_back` clip
transition on the first reordered frame. Because the subsequent clip's span was short (e.g. 1s),
all subsequent video frames across the rest of the title were rejected and dropped as outside
clip marks. Audio tracks (which have monotonic PTS without reordering) were unaffected.
Track `fed_bytes` in `DiscStream` and route through `feed_at` for both TS and PS demuxers,
forwarding `ps.source` to `PesPacket` on the PS path.
Test: `mux::disc::tests::stream_surface_tests::disc_stream_stamps_source_provenance`
MLP/TrueHD carries predictor + restart state across access units, so after a source transport-stream discontinuity (continuity-counter gap) the parser was splicing post-gap audio mid-stream, decoding against stale state and producing a decoder-choking seam (a burst of "restart header sync incorrect" / "Invalid blocksize" errors that could flag the whole track corrupt).
Honor the source's own gap signal: drop forward to the next CRC-valid major-sync AU (a clean re-init point) before resuming, guarded so it never arms before a validated baseline exists (no stream-head whole-track drop). Mirrors the video keyframe resync and the ResyncGate design. Corrects the "audio has no inter-frame refs" comments in truehd/dts/resync. Adds a regression test.
Adds a 'changelog' job that fails when Cargo.toml's version has no matching
'## [X.Y.Z]' heading in CHANGELOG.md, hard on main and PRs into main, warning
on dev/qa — so the unified release can't ship a crate with a blank changelog.
The B1 resync gate (mux/resync.rs) arms on a concealed-gap / clip-seam
discontinuity and drops video frames until it sees a keyframe. The H.264
parser flagged a keyframe only on an IDR slice (NAL type 5), so on a BD
open-GOP tail whose only random-access point is a non-IDR I-frame (the
recovery point) the gate never disarmed and dropped every remaining frame
to EOF — truncating the video by up to a GOP-plus tail and failing the
cli-acceptance timeline_sound strict span check for BD.
Promote an intra-coded access unit (measured coding_type == I) to a
keyframe, alongside the existing IDR-by-NAL-type flag. This is the H.264
equivalent of HEVC's open-GOP IRAP (CRA) handling, which the HEVC parser
already flags by NAL type — H.264 has no such NAL tag, so the measured
intra coding type is the signal. Keying off coding type rather than a
recovery_point SEI covers every signalling: an SEI-tagged recovery point
(which is intra by construction) and a disc that marks the recovery point
by GOP structure alone. Consistent with videomap.rs's model that a
keyframe IS the intra decode-restart point; the open-GOP anchor now also
gets the self-contained SPS/PPS re-assert, mirroring HEVC at CRA. IDR
stays flagged by NAL type, so a slice whose header fails to parse
(coding_type == None) is unaffected. HEVC, DVD/MPEG-2 unchanged.
Tests: open_gop_intra_access_unit_is_a_keyframe (red before this change),
inter_coded_access_unit_is_not_a_keyframe (precision guard).
find_drives and resolve_device each carried a three-arm cfg(target_os) block in
their bodies. Alias the current platform's module once (`use linux as platform`
under a single cfg) and let both dispatch through `platform::…`, so the function
bodies are cfg-free and a new entry point cannot forget an arm.
Two audit fixes plus two consistency cleanups, all in the changed 1.6.5 surface:
- hddvd: the VTI clip-order read used `.ok()`, flattening an unreadable
authored order (a scratched sector under the .vti — the name came from the
directory, so it is never "absent") into "no order" with no diagnostic. The
sibling clip-extent arms log every read failure with its code; this one now
does too, then falls back to the per-clip heuristic exactly as an unauthored
disc would. Behaviour is otherwise unchanged; the loud line is the point.
- extract: the progress sink hardcoded bytes_unreadable_total: 0 and counted
every zero-filled hole as good, so a progress-only consumer saw a holed
extraction climb to a clean 100%. Thread the running unreadable total through
and report the real good/unreadable split. The authoritative ExtractResult was
already truthful; only the live channel lied.
- labels: correct a stale comment that described testlog's old lock-based
capture; it now installs one global subscriber and routes to a thread-local
sink.
- ps: import SYSTEM_HEADER from consts instead of re-declaring 0xBB, matching
its sibling stream-id constants and the file's single-source rule.
testlog: capture through a single global subscriber that routes each
event to a thread-local sink and returns Interest::sometimes, so
tracing's global per-callsite interest cache can never short-circuit a
live capture when another thread rebuilds it. The old scoped
with_default lost events at random in the full suite (an empty capture
failed the log-accounting assertions ~1 run in 15).
labels: capture the escaping test through testlog instead of installing
its own process-wide subscriber, which hard-cached every foreign
callsite "off" and poisoned the captures above.
dirimage: name scratch directories from a process-monotonic counter
rather than SystemTime, which resolves to only a microsecond — two of
the parallel tests sharing a tag drew the same value, collided on one
directory, and the first to finish removed it out from under the
other's reads.
Commit b02c402 ("key the DVD crack on the disc, not on the container") fixed a
real bug — a decrypted HD-DVD hit E7023 (CssKeyMissing) because the per-title
CSS crack keyed on the MPEG-PS container, which DVD and HD-DVD share — but did
it by adding a required `disc_format: DiscFormat` parameter to the PUBLIC
`DiscStream::new` and `build_iso_pipeline`, a `disc_format` field to
`MuxInput::Iso`/`Live`, and `DiscFormat::may_have_css`, threading the axis down
through mux/driver and mux/resolve. That changed the public API and broke every
downstream caller's compilation (freemkv-engine's integration test now needed 8
args, autorip's MuxInput arms a new field). 1.6.4 shipped and worked with these
exact signatures; a bug fix must not reshape them, and needing a whole
disc-format plumb for HD-DVD was a code smell.
Revert all of that plumbing (public signatures restored to their pre-b02c402
form; no `disc_format` parameter or field, no `may_have_css`, anywhere), and fix
the ACTUAL bug where it lives: the scramble-detection heuristic.
Root cause: `is_scrambled_pack` counted a sector as CSS scramble evidence on
pack-start (00 00 01 BA) + bits 4-5 of byte 0x14. Offset 0x14 is only the PES
scrambling-control field when the sector is a genuine elementary-stream pack. An
HD-DVD `.evo` RDI navigation pack is private_stream_2 (stream_id 0xBF), an
MPEG-PS pack exactly like a DVD VOB, whose byte 0x14 is raw nav payload that
routinely has bits 4-5 set. On a decrypted HD-DVD (None keys, MPEG-PS, so it
reaches the crack) those nav packs flipped the scan's `saw_scrambled` flag; the
crack then found no key — there is no CSS on an HD-DVD — and the scan returned
ScrambledUncracked, hard-failing a good disc with E7023.
Fix: exclude the MPEG-PS structural stream_ids CSS never scrambles — system
header (0xBB), padding (0xBE), private_stream_2 (0xBF) — by the stream_id at
offset 0x11. This is the refinement the DVD design notes already called for
("matches CrackTitleKey"). It needs no format plumbing because byte 0x11 lives
in the CSS-clear header (0x00-0x7F, untouched by scrambling), so it is the true
stream_id even on ciphertext. A decrypted HD-DVD now scans to Unencrypted and
muxes cleanly.
The DVD CSS crack is preserved and proven: a genuinely CSS-scrambled DVD sector
is always video (0xE0-0xEF) or private_stream_1 (0xBD), never an excluded id, so
its scrambled packs still set saw_scrambled and still hard-fail an uncrackable
disc — the "ciphertext muxed as plaintext at rc=0" catastrophe cannot slip
through. Red-before-green both directions: dropping the 0x11 exclusion turns the
decrypted-HD-DVD case back into E7023; inverting it (only nav ids count) turns a
real uncrackable DVD into Unencrypted and strands a crackable one. Both mutations
are caught by tests.
Gate (cargo +1.97): fmt, clippy --all-targets -D warnings, 3539 tests green;
freemkv-engine and autorip both compile against this tree again; precommit.sh
libfreemkv clean.
The same hole, two disc families, two answers — and the asymmetry is now a
decision written into both files instead of an oversight in one.
`file_extents` can return `Ok` and still yield no usable extent: an empty
allocation-descriptor list, or one every entry of which the `sectors > 0 &&
lba > 0` filter discards. An ordinary zero-byte file reaches it; no crafted
disc is needed.
On HD-DVD that was the flagship failure shape. The clip entered neither
`clip_extents` nor `unusable`, and nothing was logged, so the composer's
`any(|n| unusable.contains(..))` guard missed it while the
`filter(|n| clip_extents.contains_key(..))` beside it quietly deleted the
part: a `FEATURE_2.EVO` of size 0 next to a healthy `FEATURE_1.EVO` composed
a FEATURE title out of part one alone, still advertising the whole runtime,
at rc=0, in silence. Half a movie presented as a whole one. Round 1 accounted
for every `Err` from the resolver and left this route open. It now marks the
clip unusable and logs it under its own new code, E6019
(`E_UDF_NO_USABLE_EXTENT`) — deliberately not the neighbouring E6017, which
would file a zero-length file as an authoring hole and send whoever triages
it at the wrong population.
On Blu-ray the identical hole stays open, as previously decided, and the
reasons are now recorded on both sides. BD has no `unusable` set, so closing
it there means inventing a post-loop "every clip_id must appear in `spans`"
invariant that DROPS the title, and it is not settled that an empty-but-Ok
resolve is always a defect; dropping healthy titles is worse than the gap.
The consequence is milder too: on BD the clip is one PlayItem of an otherwise
whole title, on HD-DVD the feature is COMPOSED from parts. Same hole,
different price.
Also in this change:
* bluray: a non-absence SSIF failure that the `.m2ts` fallback papers over is
logged. `unresolved` had exactly one reader, `if let (None, Some(code))`, so
when `/BDMV/STREAM/SSIF/<clip>.ssif` failed with DiscRead /
UdfAdChainTooLong / UdfEmbeddedData and the base view then resolved, the
code was recorded and thrown away: the title shipped base-view 2D off a 3D
disc at rc=0 with no log at all. The site's own doctrine is "ABSENCE is the
only benign failure". Logged, not refused — the base view is a real rip.
* drive: `wait_ready` polled TEST UNIT READY through a bare `execute` and its
60 x 500 ms loop never read `self.halt`, so a Stop during spin-up did
nothing for ~30 s while every other drive path returns Halted at the next
command boundary. `spin_cycle` issued both START STOP UNIT commands outside
`checked_exec` and slept `SPIN_DOWN_IDLE_SECS` + `SPIN_UP_SETTLE_SECS`
blind — ~15 s deaf to Stop, from the recovery path, exactly when the
operator is most likely to press it. Both now use `checked_exec` and
`sleep_until_halted`, which already lived in this file with four tests and
was `#[cfg(test)]`, called from nowhere. It is production code again.
* drive: a READ(10) that returns GOOD status with a residual underrun was
correctly refused and logged NOWHERE, while the sibling `Err` arm warns with
lba/count/status. A residual-underrunning drive was indistinguishable from a
scratched disc — two populations with opposite remedies. It now warns with
transferred vs expected, which is the whole signal.
* error: `all_error_code_constants_are_unique` was a hand-maintained `vec![]`
naming 109 of the 127 declared codes while its doc claimed to pin them all,
and an earlier audit trusted that claim while assigning new ones. The list
is now derived from the declarations by parsing `include_str!("error.rs")`,
so a new constant is covered the moment it is written. A parser self-test
cross-checks the count and three known name/value pairs, so it cannot pass
vacuously.
* testlog: a test-only `tracing` capture (~120 lines, no new dependency) so
the logging contract is enforced rather than commented. Three sites carry
long comments insisting they log the error's OWN code; putting a literal
back broke nothing. They are pinned now, along with the two new log lines.
Captures are serialised process-wide: `tracing`'s interest cache is global
while `with_default` is thread-local, and the rebuild on the exiting
capture can land after the entering one's, leaving the cache at "never"
while a capture is live. That produced a real empty-event flake.
* disc: `scan_with`'s halt wiring for the BD and DVD enumerators had no test —
every BD/DVD cancellation test calls the scanners directly, so passing
`None` on either branch left the suite green while Stop did nothing.
* mux::network: `accept_from_rejects_stream_without_fmkv_header` half-closes
instead of `Shutdown::Both`, which raced an RST against the server's read
and returned ConnectionReset instead of InvalidInput under load. The port
was already ephemeral; that was never the cause.
Gate: fmt, clippy --all-targets -D warnings, and 3439 tests green on 1.97;
precommit.sh libfreemkv clean.
`resolve_dvd_title_key` decided whether to run the DVD CSS crack by asking
what CONTAINER it was looking at — `ContentFormat::MpegPs`. MPEG program
stream is what DVD and HD-DVD have in common, so every HD-DVD title was
run through a crack scan for a copy-protection scheme HD-DVD does not
use and cannot carry: AACS is its family, and CSS appears nowhere on the
disc.
Both outcomes of that scan were wrong. The cheap one wasted up to 50,000
sector reads per title. The expensive one returned `CssKeyMissing` —
E7023 — refusing a perfectly good HD-DVD with a CSS error, which is what
a real CI run produced on the HD-DVD fixture. Reading E7023 there sends
whoever triages it looking for a missing DVD key on a disc that never had
one, which is how a routing bug spends a day disguised as a key problem.
The right axis was already in the codebase and already used: `mux/resolve`
asks `disc.format == DiscFormat::Dvd`. This threads the disc format down
to the decision and adds `DiscFormat::may_have_css` to name the question.
The asymmetry decides the default, so it is worth stating. Running CSS on
an HD-DVD costs a wasted scan or a false refusal — visible, recoverable,
annoying. NOT running it on a real DVD muxes scrambled sectors as
plaintext and exits 0, which is the failure-that-looks-like-success class
this project has shipped once already. So `may_have_css` is false ONLY
for the families proven CSS-free, and `DiscFormat::Unknown` — a bare
reader with no scan behind it — still cracks. An `== Dvd` allow-list
would have read as tighter while silently stranding every caller that
cannot name its disc.
Both directions are pinned: an HD-DVD title must never enter the crack,
and an unknown-format title must still enter it. Removing the disc-format
clause fails the first and leaves the second and every DVD test green.
A cancelled scan was indistinguishable from a disc that simply holds
fewer titles, in all three tree enumerators. Once the operator presses
Stop, `Drive::checked_exec` fails EVERY subsequent SCSI command with
`Error::Halted` and `Drive::read` deliberately preserves the variant —
so a cancel is not one failed read, it is every remaining read failing.
* `scan_bluray_titles` returned a bare `Vec<DiscTitle>` and
`parse_playlist` an `Option`, so neither had a channel to report a
halt. During a cancel each remaining `.mpls` read merely failed and
was skipped by an `if let Ok(..)`, and the scan returned a TRUNCATED
title list at rc=0. Both now return `Result`; `Ok(None)` from
`parse_playlist` keeps its two benign meanings (unparseable MPLS,
playlist under 30 s) and `Err` means the scan is over.
* `scan_dvd_titles` was worse: `Err(_) => return Vec::new()` turned a
cancel into ZERO titles at rc=0 — a DVD reported as carrying no
video at all. `ifo::parse_vmg` compounded it, treating a halted
title set as one of the placeholder TT_SRPT entries it is designed
to tolerate and continuing to the next; both now propagate.
* HD-DVD was already correct and is the model this follows, including
its reasoning: a cancelled enumeration that returned `Ok` would be
cached, displayed and ripped from as if it were the disc.
The cancel is propagated at ALL FOUR Blu-ray read sites, not just the
obvious one — the playlist read, the `.clpi` read, and both
`file_extents` calls in the clip resolver — and the halt is polled
between playlists and again after the loop, so the final iteration
cannot slip a truncated list through. The resolver's `note` closure
previously EXEMPTED `Halted` from its disc-defect classification, which
was right (a cancel is not an authoring hole) but left it swallowed:
with no exemption and no propagation a cancel landing on `file_extents`
returned a title claiming its full runtime with that clip's bytes
silently missing. Measured with the fix reverted: `Ok(Some((768000,
[])))`.
Second defect, same function: the `.clpi` read and `clpi::parse` sat
inside `if let Ok(..) && let Ok(..)`, with extent resolution nested
inside. A scratched CLIPINF sector, a missing `.clpi` or a malformed one
skipped the whole block — no extents, nothing added to `total_size` —
but `duration_ticks` had ALREADY been summed from the play items, so the
title still advertised the movie's full runtime. The `if let` discarded
the error, so nothing was logged either: a failure that looks like
success, which is this crate's flagship defect class. It is now a
`match` that drops the title and warns with the error's OWN code, the
same classification the extent resolver below it already uses. Logging a
fixed code would account a scratched disc as an authoring hole.
Deliberately NOT added: a post-loop "every unique clip_id must appear in
`spans`" invariant. `file_extents` can also return `Ok(vec![])`, or a
vector whose every entry the `sectors > 0 && lba > 0` filter discards,
and it is not settled that an empty-but-Ok resolve is always a defect
rather than a legitimate healthy-disc state (zero-length placeholder
ADs). Dropping healthy titles is a worse failure than that residual gap,
so the route stays open, with a comment at the site recording the
decision so the next audit finds it instead of re-deriving it.
Tests: `parse_playlist_missing_clpi_yields_no_extent_no_size` asserted a
title WAS returned with `size_bytes == 0` and empty extents, and its
docstring quoted the buggy control flow as the specification — it
blessed the defect. Inverted and renamed. `parse_playlist_keeps_exactly_
30_seconds` encodes correct behaviour but used a fixture wiring no
STREAM/CLIPINF, so the `.clpi` fix would have dropped its title for a
reason unrelated to the boundary it pins; repointed at the fully wired
fixture rather than weakening the assertion. Four new halt-propagation
tests (BD playlist read, BD `.clpi` read, BD extent resolve, DVD title
set + VMG), each proven red before green.
Ten lenses over v1.6.4..HEAD, every claim read against the code before
it was believed. Seven confirmed; six are here, one is recorded for the
next round. All of these are the same family — a failure wearing the
shape of success — which is the family that once shipped 9 MB of
ciphertext inside a main-movie m2ts at rc=0.
A clip whose extents cannot be resolved is now accounted for, in both
disc readers. Only `UdfUnrecordedExtent` used to count: every other way
`file_extents` can fail — a scratched sector under the clip's ICB
(DiscRead), an allocation-descriptor chain that never terminated, a file
whose data is embedded rather than extent-mapped — fell through to the
ordinary "file absent" path. On Blu-ray that yielded a title advertising
its full runtime with a clip's bytes silently missing, because the size
and the play-item timing had already counted it. On HD-DVD it was worse:
the clip was never added to `unusable`, so a split feature still composed
from FEATURE_1 alone and offered half a movie as the whole thing. Neither
emitted a single log line. Absence is still benign — a 2D disc has no
.ssif and the extension fallback exists for exactly that.
`Halted` is excluded deliberately, and that exclusion is the whole reason
the first version of this fix was wrong. Cancellation makes EVERY drive
command return `Halted`; classifying it as a disc defect would have
dropped each remaining playlist in turn and handed back a truncated title
list at success — the same defect, wearing a cancel. `parse_playlist`
returns Option and has no channel to propagate a halt, so the existing
behaviour is preserved rather than made worse. Propagating it properly is
next round's work.
Both log sites now emit the error's OWN code instead of a hardcoded 6017.
Accounting a scratched disc (E6000) as an authoring hole would send
anyone triaging it looking for the wrong thing entirely.
AD type 3 is embedded data, not a descriptor list (ECMA-167 4/14.6.8).
`read_icb_extents` lumped it in with the reserved values and decoded the
file's own CONTENT as (length, LBA) pairs, manufacturing extents out of
arbitrary bytes and pointing the reader at unrelated sectors. This same
release already taught `read_directory` to honour type 3; this is the
file half of that decision. It is an error rather than an empty list,
because an empty list reaches the caller as a clip that contributed
nothing while its declared duration still counts it — the silent loss
pointed the other way. A legally zero-length embedded file still returns
an empty list. New code E6018: reusing DiscRead would have mislabelled a
deterministic structural property as transient I/O and fed the retry and
NonTrimmed machinery a byte that will never change.
`file_extents_addressing`, `extents_abs_at` and `AbsExtent` drop to
`pub(crate)`. The first hands back unrecorded extents UNFLAGGED, in a
shape identical to the safe call's return; its doc says callers must use
`file_extents` instead, but a doc comment is not a guard. No dependent
crate references any of the three.
Three tests close gaps the audit found, each proven red before green:
a held AC-3 access unit must not resume as a normal frame after its track
poisons; the PS resume cursor must survive a drain that rebases it (three
separate mutants caught); and AD type 3 must be refused rather than
decoded. The first attempt at the HD-DVD test passed with the fix
reverted, which made it worthless — it needed a VTI fixture before the
composition path ran at all.
Also: four error codes were missing from the uniqueness test that claims
to cover every published code, so a new variant reusing 6014, 6016 or
6017 would have passed it.
`dtolnay/rust-toolchain` is not a dependency in the sense Dependabot
means: it is versioned by the Rust release it installs, and the tag we
pin is the toolchain CI is pinned to on purpose — precommit.sh runs the
same one locally, so a lint that passes on a developer's newer default
cannot pass CI by accident.
Read as semver, those tags produced a 1.97.0 -> 1.100.0 proposal, a Rust
version that does not exist. Every such PR 404s on toolchain download
across all eight repos and regenerates weekly: eight permanently-red PRs
that promote.yml then has to special-case when it decides whether dev is
green.
freemkv already carries this ignore; this is the same block in the other
seven, so the fleet stays uniform. Bumping the toolchain stays a
deliberate, all-eight-repos change made by hand together with
precommit.sh.
Squashed from 12 commits. Every fix was proven red-before-green and killed by a
mutation; the reasoning for each is in the private audit record.
UDF and extents
Honour ICB types rather than assuming a Short AD, so an AD-type-3 directory
is no longer decoded from FID bytes into a silently empty listing. Carry the
ECMA-167 recorded flag through to the resolvers: an allocated-but-never-
written extent used to reach the read plan as ordinary content and splice
undefined sectors into the rip. file_extents now refuses such a file, and
only when the hole actually occupies byte space — a zero-length one displaces
nothing, and refusing on it dropped whole titles off discs that ripped
correctly. Type-2 sparse extents are kept alongside type-1; they were falling
into a catch-all that exited the descriptor loop and returned a truncated
list as complete. merge_ranges no longer claims a sector neither input
covered. A short skip or an over-long AD chain errors instead of truncating.
HD-DVD and Blu-ray scanning
Bound the XPL nesting depth, title count, clips and chapters per title, and
memoize the clip-name fallback probe — four separate amplification axes, each
of which alone left the worst case unbounded. The clip and title caps are 512,
~10x any retail disc, and a test pins the product of cap and probe budget.
The scan is cancellable: it returned Ok with titles carrying no streams when
halted, presenting a cancelled scan as a successful one. A clip dropped for an
unrecorded extent now says so.
Codecs and muxing
Resume a held E-AC-3 access unit rather than rescanning from its first frame,
and drop it on a discontinuity — a stale hold indexed past the end of the new
buffer. Map every ISO 639-1 code instead of collapsing fifteen languages to
und. Correct the DVD palette order. Detect a skip past EOF.
Drive and I/O
Classify dead-bus faults so the wedged-drive path can see them; a catch-all
arm had been flattening the variants before the classifier ran. A prefetch
producer that dies now reports SourceTerminated instead of Ok(0), which the
reader legitimately read as a short read and zero-filled — a whole title
could be fabricated and the pass reported complete.
Also: charge Ok(0) reads to the CSS crack budget, drop the unreachable soft
re-crack, and send disc-derived strings to logs through the debug formatter so
a crafted label cannot paint an operator's terminal.
`resolve_vts_key` called the `Option`-returning `css::crack_key`, which
collapses "no scrambled sector was seen" with "scrambled sectors were
seen and no key came out", and then fell back to the disc-wide key. A
multi-VTS CSS DVD whose second title set resists the Stevenson scan was
descrambled under the FIRST set's key: corrupt PES behind an intact
header, written out with `complete = true` at exit 0.
`CrackOutcome` exists precisely to keep those two apart, and its doc says
callers must surface the second as a hard error. Every sibling path in
the crate already does — `Disc::decrypt_keys_for_title` and the mux path
both map `ScrambledUncracked` to `CssKeyMissing`. This was the one path
that did not. The ordering comment 15 lines above describes this exact
outcome as the bug it exists to prevent; ordering makes the crack far
more likely to succeed, but it cannot make a failed crack safe.
Also, three things nothing could catch:
- The demux output filename took the stream's language raw while the
base beside it was sanitised. A language code is three raw STN bytes
through `from_utf8_lossy`, and `00 00 00` is the ordinary "undefined"
encoding on real discs — a NUL in a path fails `File::create` with
InvalidInput, taking the whole export down before one track file
opened. `sanitize` now maps control characters too; it did not.
- `css::crack_key_scan`'s short-read handling was dead code under test:
every source in the module returned the full request, so reverting
`advance` to the requested count, or dropping the `.max(1)`, left the
suite green. The `.max(1)` is load-bearing — without it a source
returning `Ok(0)` never moves the cursor and never increments the
budget, so the scan spins forever. That mutation now HANGS the test
rather than failing it, which is the honest demonstration.
- `MAX_SUBDIRS`'s const-assert carried `#[cfg(not(test))]` inside a
`#[cfg(test)] mod tests`, so it compiled in no configuration and could
never fire — the dead gate the test above it was written to replace.
Moved to module scope, and verified it now rejects a wrong constant at
compile time.
`forced_sub` was the last unbounded attacker-controlled list in
paramount.rs. `MAX_COM_INDICES` capped the two `*_com1_idx` attributes;
this one had nothing capping it at all, and unlike them it has no value
to filter — a cell is a classification of the position it sits at, so
its bound has to be positional. Extracted as `forced_subs` for the same
reason `com_indices` was extracted: through `labels_from_feature` the
bound is unobservable, because the subtitle loop cannot reach those
cells either, so the assertion could not fail.
The `MAX_COM_INDICES` doc claimed the entry-allocation argument for the
whole constant. It is the VALUE filter that caps the set (values below
the bound, so at most that many distinct entries, however long the
attribute); the `take` caps the WORK. Both are real and they are not
the same bound; the doc now says which is which.
`jar_inventory_dedup_is_not_quadratic` called itself proof by deadline.
It is a hang guard — a return to the linear scan runs for minutes and
would wedge CI rather than fail it — and no assertion in it can tell a
BTreeSet from any other sub-quadratic dedup. Renamed and documented for
what it does. Its margin was measured before keeping it: 0.14s debug /
0.07s release against 10s, ~70x, unlike the 6x that made paramount.rs's
wall-clock test flake.
Both are mine, and both pass with the bound they "cover" deleted.
`a_hostile_commentary_index_list_is_bounded_not_merely_fast` asserted
only label purposes. The out-of-range filler it feeds is unobservable at
the label level and a HashSet collapses the repeats, so removing
MAX_COM_INDICES entirely leaves it green. What it DOES catch is a bound
set too low — verified at 2, where the real indices stop resolving. Named
and documented for that, and it no longer implies it guards enforcement.
`an_index_that_cannot_address_any_cell_is_not_retained` asserted through
the labels, where retention is by definition unobservable: the loop never
queries a cell that high. It now reads the set through `com_indices`,
where the claim is checkable.
Enforcement was and remains proven by
`distinct_unaddressable_indices_are_refused_not_stored`, which was red at
50,000 retained entries and green at zero. Two tests, two properties;
neither pretends to the other's job now.
The gate caught `commentary_index_lookup_is_not_quadratic` failing, then
passing on a re-run. Measured both ways: 1.62s alone, OVER 10s against
its own 10s deadline while the suite's other 3,347 tests ran
concurrently. A 6x margin against a shared CPU is not a margin, and a
test that gets re-run until it passes is not a test.
It was also measuring the wrong thing. Replacing the linear scan with a
HashSet bounded the LOOKUP; the set was still built from every entry the
disc declared. `playlists.xml` is attacker-controlled and has no length
of its own, so a hostile disc could still force an unbounded allocation
before any lookup happened — the parse, not the query, was the exposure.
The code's own comment said "unbounded parsed input" and only fixed half
of it.
`MAX_COM_INDICES` bounds both halves, at the one value that cannot
change behaviour: an index at or beyond `u16::MAX` can never match a
cell, because the labelling loops break at `u16::try_from(i + 1)`. Real
authoring is nowhere near it — the BD STN table admits 32 streams.
The parse moved into a `com_indices` helper so the bound is OBSERVABLE.
Through `labels_from_feature` it is not: a HashSet collapses repeated
values, and an out-of-range index changes no label, so the obvious test
passes with or without the cap — an assertion that cannot fail, which is
what the first draft of this fix shipped. The test now hands in 50,000
DISTINCT unaddressable indices and asserts they are refused. Proven red
with the bound removed (50,000 kept), green with it.
Applies to both `sub_com1_idx` and `aud_com1_idx`.
NOT done here: `labels/mod.rs:1421`'s `jar_inventory_dedup_is_not_quadratic`
is the same wall-clock shape and has the same flakiness. Named so it is
not lost.
Three round-1 audit findings on v1.6.0..HEAD.
Tests: `mux::resolve::input("dir://…")` — the door the CLI rips a folder
through — was never driven with a real folder. The only dir:// input test
uses a missing path, which fails in DirImage::open long before
`image_input`, so dropping the `is_folder` argument (and with it
`session::apply_folder_encryption_verdict`) passed the whole suite. Two
tests now close it, stated as a DIFFERENTIAL between the two doors,
because that is the invariant the shared function exists to hold: a
clear folder that kept its AACS/ directory must scan the same and select
the same extents through `scan_dir` and through `dir://`, and a
scrambled one must be refused through both with E9063. Proven red
against a mutated guard (both fail with E7022, the tree-shape verdict
the probe overrides) and green with it restored.
Docs: css/mod.rs documented `is_scrambled`, a function this line renamed
to `has_scramble_flag_bits`. Eleven stale sites, two of them broken
rustdoc links. Two were not stale names but false statements — they said
the descramble loop keeps the looser raw-flag test, when
`descramble_sector` and `descramble_region` both gate on
`is_scrambled_pack`; that parenthetical is rewritten to say what the
code does and why (the measured VIDEO_TS.IFO case: 38 titles became 10,
silently, at exit 0). The one mention that must stay is the sentence
explaining why the name was rejected.
Constants: io::image_writer and dirimage::encode each re-declared the
2048-byte sector that consts::SECTOR_BYTES already exports; both now
alias it. Note `BATCH_SECTORS = 2048` in image_writer is a different
quantity (sectors per batch, not bytes) and is deliberately left alone.
Bare 2048 literals elsewhere in the crate are out of scope here.
These are eight public repos that ship one product on one version, and
they had drifted: freemkv-engine carried no LICENSE at all (GitHub
reported its licence as none), two crates had no code of conduct, four
had no contributing guide, and none had a security policy — so there was
no private route to report a vulnerability in a disc-decryption tool.
SECURITY.md names GitHub Security Advisories on each repo rather than an
address, so there is nothing to keep in sync and no inbox to go stale.
dev is where work lands and is meant to be pushed to often, so what runs
there should be the cheap answer to "did I break it": lint, tests and the
Linux build. The macOS and Windows jobs now run on qa and main instead of
on every push to dev.
Nothing is deleted and no platform goes unchecked before a release. qa.yml
already covers macOS and Windows independently, and the jobs that live only
here -- the Intel macOS build, the Windows release build -- still run, on
the branches where a cross-platform break is worth blocking on.
They are SKIPPED on dev via `if`, not left unscheduled. A queued job would
be worse than a slow one: release.sh's CI gate refuses while any run for
the commit is still in progress, so a job that never gets a runner blocks
releases silently, with no error anywhere. That is the failure the
real-media note in qa.yml describes, and it is why this is an `if` on the
job rather than a narrower set of trigger branches.
Version bump and changelog for the 1.6.3 sync. All eight crates ship the
same version, so the crates with no functional change this cycle say so
rather than carrying an empty section.
Two problems, both invisible until the whole graph is looked at together.
DEAD: num-bigint, sha2, num-traits, num-integer, cmac and cbc are
declared here and referenced nowhere -- not in src, tests or benches.
They were being compiled, audited and offered version bumps forever for
no reason. Removing beats bumping.
cbc nearly survived the sweep: a substring search for "cbc" matches 44
occurrences of ycbcr_to_rgb in the DVD subtitle decoder, so it looked
used. Only a word-boundary search exposed it.
SKEW: this crate was the outlier on every shared dependency -- aes 0.8,
rand 0.8, base64 0.22.1 and zip 2 against 0.9 / 0.10 / 0.23 / 8
elsewhere. Cargo cannot unify across a major version, so it compiled
BOTH: 32 duplicated crates in the freemkv binary's graph, including two
complete AES implementations (aes 0.8 + 0.9, cipher 0.4 + 0.5), two
digest stacks and two getrandom. Two crypto stacks in one product is
worth removing on its own.
The aes bump is an API rename -- BlockCipher-prefixed traits, Array for
GenericArray -- and the obvious translation uses Array::from_slice,
which the new version deprecates and clippy's -D warnings would reject.
These use the From<[T; N]> conversion the crate points at instead.
3441 tests pass in debug and release. The AACS crypto here is covered by
known-answer tests, so a byte-order or sizing mistake in that rename
could not have passed.
Version updates were never configured here -- only security alerts, which
report but never open a pull request. So the absence of Dependabot PRs was
not "nothing to update", it was "nothing was checking".
Updates target `dev`, never `main`. main is a release pointer that
release.sh moves to each tag, so a bot commit on it would put work there
that no tag contains -- the exact state that aborted the 1.6.2 cascade at
its last step, and pointing a scheduled bot at main would recreate it
weekly.
Minor and patch bumps are grouped into one PR rather than one per crate:
eight repos times a handful of dependencies is a volume nobody reads, and
an unread PR queue is indistinguishable from no updates at all.
The freemkv crates are excluded. They depend on each other by git tag,
re-pinned by release.sh inside the release commit, and Dependabot cannot
see that cascade -- a PR bumping one could pin a version whose tag does
not exist yet.
It fired on `release: published` and committed to main, the branch the
release points at its tag, so it could only ever leave main ahead of the
tag. release.sh now makes the same edit inside the release commit.
In this repo it was a no-op regardless: the pattern it rewrote does not
appear in this README, so it has been running and changing nothing.
A clip's span is measured over the title's full extents when the disc is
scanned, and a frame is placed by the offset it was read from. Those only
agree while the mux reads every byte the scan counted.
On a disc carrying a forensic segment it does not: the read plan omits the
units belonging to another device group, so fewer bytes are fed than the
spans describe, and the shortfall grows through the title. Every frame past
the first segment then looks earlier than it is — placed in a clip it did
not come from, or dropped at a join for failing marks it was never inside.
The spans still tile one another perfectly, so the check that asks whether
they can be trusted cannot see any of it.
The plan is now compared against the extents it was built from. When they
differ the spans are dropped and placement falls back to timestamps, which
is what that path is for and what the surrounding comment already promised
would happen when an offset stops meaning anything. Ordinary discs are
untouched: with no forensic segment the plan IS the extents.
The provenance guard knew one way to lose a frame's source offset —
writing it as absent. Omitting the field entirely does the same thing,
because the frame type fills it in by default, and the guard read straight
past that. A parser rewritten into the second spelling would have kept its
green light while its track silently went back to being placed by guesswork.
It now reports which spelling it found and where.
The title-count clamp was asserted over a fixture too small to hold more
titles than the cap allows, so the walk stopped when the buffer ran out and
the clamp was never what bounded it — the assertion held with the clamp
deleted. The fixture now carries more entries than the cap.
The subdirectory limit was checked by restating the constant's own
definition; the guard itself had never run, and deleting it changed
nothing. The limit is lowered under test so a real folder can exceed it,
and the test now walks one and requires the refusal.
One test also carried two unrelated grounding notes while the test they
described had none, so reading the note above a test told you about a
different one.
A playlist may point at the same .m2ts from several play items — a
seamless split, a looped segment, multiple angles. The file has one set of
bytes, so it has one span, and the byte a frame was read from therefore
identifies the FILE, not which play item's range it falls in. Placement
took the first of them and judged every frame against its marks, so
everything past that range's end was treated as material the playlist
excludes and dropped: half the clip missing from the rip, with the
timeline still charged for its full duration.
The offset narrows a frame to the run; only its timestamp can finish the
job, and each play item carries its own marks. A frame is now matched
against the marks of the entry it actually falls in, falling back to the
first when it falls in none, which leaves genuinely-excluded material
dropped as before.
The test for this construction asserted only that such a playlist is
trusted, and never placed a frame from the second range — so the loss it
described in prose was invisible to it. It now places one from every range
and requires them all to survive.
The deferred-mux replay of buffered frames wrote them without their source
offset, quietly sending the head of every such title down the timestamp
heuristic the rest of this work exists to retire. It passes the offset
through now, and the frame writer that omits it is compiled out of the
library entirely: no production path can discard provenance any more.
Also: the AC-3 buffer-reuse test could not see the regression it named.
Feeding equal-sized packets, a fresh allocation per call yields the same
capacity as a reused one. It now feeds a large packet then a small one,
where only a reused buffer keeps the larger capacity.
An ADTS header that declares a CRC follows must be at least nine bytes —
seven of header plus the two the CRC occupies — because the declared frame
length counts them. The structural gate compared against a flat seven and
never read the bit that says whether a CRC is there at all, so a frame
whose own header describes something impossible was accepted and handed to
the muxer as decodable.
The pipeline's spawn doc named Sweep, and the thread Sweep would have
created, as callers to look for. Neither has been in this crate since the
recovery passes moved out. The same paragraph already records fixing this
once, for a different departed caller — it simply drifted again a sentence
later, so it now says to name callers that live here or name none.
Whether a disc is structurally AACS-encrypted was spelled out by hand in
both the fast identify and the full scan. They agreed today; nothing made
them agree tomorrow, and disagreeing would mean the same disc reported
encrypted by one path and clear by the other. There is one definition now,
and the comment that pointed at it by line number points at its name.
The AC-3 parser built a fresh buffer on every packet — of the order of a
hundred thousand times per title — to work around a borrow it cannot
avoid. The copy stays; the allocation does not. The buffer is now lent out
and handed back, and a test pins that, because reverting it would be
invisible in behaviour.
Left alone deliberately: send/send_with_halt and finish/finish_with_halt
look like one action under two names, and are not. After the consumer
fails, one must still accept items and the other must refuse them; that
difference is what stops a producer reading an entire disc for a write
that died on its first frame. Collapsing them was tried here and the
existing test caught it. Both now say so where the choice is made.
A recovery resumed against an image that has been truncated since the
previous pass — a full disk, an interrupted transfer, a remount — cannot
repair it: the pass only revisits the ranges the recovery data calls bad,
so everything past the cut stays a hole while the counts still describe a
whole disc. There was no way to say that. Reusing the recovery-data error
would have been wrong, because the recovery data is intact; the image is
not.
Registered in the uniqueness list, the code table and the range check, so
a future variant cannot silently reuse 6015 or map to the wrong code.
The `consumers` job checked out all five dependents against this commit and
ran `cargo check --all-targets`. That proves they still COMPILE, which
catches a changed signature and nothing else. The failures worth catching
here keep every signature intact and change behaviour: the library still
builds, the dependent still builds, and the dependent's tests are what go
red. Those never ran.
Run their suites instead. Same checkouts, same patched paths — only the
verb changes.
The demuxer interleaves, so at a boundary the streams do not reset on the
same frame: audio for the next segment can reach the muxer before the video
frame that opens the epoch it belongs to. Only the primary video track may
open one, so those frames rode the just-ended offset and landed a whole
segment in the past. Downstream the strictly-monotonic block nudge then
crushed the entire run onto one instant a tick apart, which is audible.
Such a frame is recognised on its OWN raw PTS, which within an epoch only
advances, so a large backward step is unambiguous — a different signal from
the shared frontier, which is what the old false-positive ratchet keyed on.
It then rides a provisional offset private to its track, computed the same
way the video path computes a real one, and drops it the moment the video
retires an epoch, so the run rejoins with no seam.
The offset is deliberately private: it never writes offset_ns, never
advances the frontier and never retires an epoch, so it cannot move the
video timeline. Letting a passive track open a REAL epoch was tried first
and inflated a 476.776 s title to 656.216 s, because every track meets a
boundary at its own pace and the video path rebased again on top.
Measured on a real DVD title with 8 cell boundaries: audio frames stamped
inside one cadence 120 -> 0, outsized gaps 7 -> 0, subtitle 1 -> 0, and the
video span byte-identical at 476.484. On the HD-DVD title the remaining
boundary gap on its first audio track falls from 0.999 s to 0.275 s.
Acceptance: 76 pass, 0 fail, 0 skip.
A non-video frame can reach the muxer after the video that opened the next
epoch has already been processed. It carries an old-epoch PTS, so adding
the new offset flings it forward by a whole clip. The remap that exists to
catch this compared the frame against the CURRENT frontier and only
accepted it within one backstep, which is the wrong yardstick: by the time
a straggler is seen the frontier has moved on into a new epoch, and how far
below it the frame lands says nothing about where it belongs.
Retiring an epoch now records the frontier it closed at alongside its
offset, and a straggler is judged against the end of the epoch its PTS came
from. Keeping the whole history rather than one previous offset also fixes
the case a single `prev_offset_ns` cannot express at all: a frame from two
or more epochs back.
Measured on a real HD-DVD title whose second audio track put its final
packet at 12834.587 s in a 6434.100 s file. That frame sits 23 s below the
current frontier — outside the old window, so it was refused — but 0.15 s
below the end of the clip it actually came from. It now lands at 6417.216,
one 32 ms cadence step after its neighbour, and the track spans 6416.160
with no outsized gaps. The other two streams are unchanged.
A title that never rebases retires no epoch, so the lookup finds nothing
and the mapping is bit-identical to before.
Four parsers (dvdsub, flac, lpcm, mpegaudio) stamp a source offset on
every frame but had no test that read one back, so a regression to
`source: None` would have been caught only by the brace-balanced audit
in codec/mod.rs — a lint, not a behavioural check. Each now asserts the
emitted frame carries the offset of the packet that supplied its first
byte.
The Blu-ray feed spans had no direct test at all. Add one that walks a
multi-item playlist and requires the spans to tile the feed with no gap
or overlap; it catches a one-sector-per-clip drift, which is exactly the
error class that would misattribute frames near a seam.
`no_provenance_still_places_by_marks` asserted only that placement
returned something, which passes for a frame placed in the wrong clip.
Its probe timestamp lands in an overlap between two clips in the real
mark table, so pinning one clip would assert a coin-flip; instead
require the offset to be one that a clip actually containing that
timestamp would produce.
The TT_SRPT pointer at 0xC4 is a SECTOR offset from the start of
VIDEO_TS.IFO. Reading it as a byte offset lands 1/2048th of the way in and
does not error — it silently returns a different title list, which is how
a disc enumerating 38 titles and an image enumerating 10 looked like a
scan difference rather than a failure.
The table already had tests for its entry layout, dedup and the 99-title
clamp. What it did not have was anything pinning the offset unit itself,
or the grouping of titles by title set, or a truncated table keeping the
rows it can read rather than failing a whole disc.
release.sh dates the heading when it cuts the tag, and refuses a bare
'## [1.6.1]' because it cannot tell an undated heading from one it has
already stamped. Shipping a tag whose public changelog says UNRELEASED is
permanent and cannot be corrected inside that tag, so the check is right
to stop rather than guess.
Every emitted frame must carry the source byte offset of the packet it
came from. That held only for video for as long as it existed, and
nothing asserted it, so nothing caught it: ten parsers built frames with
`source: None` and a multi-clip title could not place audio or subtitles
by byte at all.
Finding them took a brace-balanced scan of the tree by hand — a regex
cannot do it, because a Frame literal contains nested braces and a
non-greedy match stops at the first `}`, which is how five sites survived
the first pass. This is that scan, as a test.
A second test fails if a codec module is added and not listed, since an
unchecked parser is exactly how the gap persists. Modules that emit no
Frame are named explicitly rather than skipped silently.
The guard tripped on itself twice while being written — first on its own
doc comment, then on its own string literals — so it strips line comments
and excludes its own module. A check that matches prose about the defect
rather than the defect is the same mistake in a different place.
The entries had grown into narrative paragraphs; each is now the outcome
first and the mechanism in a sentence or two. The website changelog page
mirrors libfreemkv's, so it inherits this directly.
Two things that shipped in this cycle were not in it at all:
dir:// as a source. It was pulled from 1.6.1 on 2026-08-05 because the
folder reader produced a wrong title list on a real DVD. That defect was
the CSS descramble bug below, fixed since — the acceptance suite now
shows a folder reporting the same 38 titles as its ISO, with matching
streams, languages and runtime, and the CLI has been wired for it all
along. It ships, so it is listed.
The CSS title-table corruption itself: a disc enumerating 38 titles
produced a decrypted image enumerating 10, silently, at exit 0. That is
a data-integrity fix a user needs to know about.
A real rip caught these. The warning added for a track placed from
timestamps under a seam plan fired once, on one subtitle track of a
23-clip title, eighteen minutes in — a display set that arrives as a lone
non-PCS segment is emitted straight through rather than accumulated, and
that path still built its frame with source: None.
Auditing the rest the same way found four more: flac, lpcm, mpegaudio and
the passthrough parser. All are one-PES-one-frame, so the packet's own
facts are the unit's facts.
Every Frame construction in every codec parser now carries a source,
checked by walking balanced braces rather than by eye — the earlier
count was taken with a regex that cannot see a block containing nested
braces, which is how these survived the first pass.
Marks that do not advance across a title are normal, not a defect: each
clip file carries its own STC, so one clip's IN has no ordering
relationship to the previous clip's. Refusing those tables dropped
exactly the branched titles this type exists for onto the inference path
that cannot read them, and that is the overrun.
Lifting it was tried once before every track carried a source offset, and
nine audio and subtitle tracks with nothing to place them by pinned
themselves to clip 0 and dropped most of the title. Every parser stamps
provenance now, so the clip comes from the byte offset and the marks keep
one job: whether a frame lies inside its own clip's [in, out].
Still gated on spans_trusted. Without usable spans there is no offset to
place by and inference is all that is left, so a table inference cannot
read is still refused rather than silently truncating a title.
A frame reaching the timestamp path under a plan now says so once per
track. It is not expected any more, and it is how a mostly-dropped track
reached a user without a single line in the log.
dts, ac3 and truehd assembled access units across PES packets with three
private implementations of the same bookkeeping, and none of them carried
the source byte offset. They now hold a PesBuf, so a unit takes the
timestamp AND the source of the packet covering its first byte, from the
same mark, and no codec can answer that question its own way again.
dts is the reference case: its pts_marks already implemented the rule
correctly, and all 61 of its existing tests -- including the PTS
attribution ones -- pass unchanged on the shared type. That is the
evidence the type preserves the behaviour dts had right.
ac3 kept a single carry-over timestamp and one anchor offset, so it could
only attribute the first unit in a call; it now resolves each unit at its
own offset. truehd had no attribution at all beyond a running clock.
New tests cover the case that motivated this: a unit whose first bytes
arrive in one packet and whose remainder arrives in the next keeps the
FIRST packet's offset. At a clip seam those two packets belong to
different clips, and taking the later one places the audio in the wrong
one.
Clippy on the pinned toolchain caught an empty `if` block left where dts
used to clear stale marks -- restored as an explicit clear, with why it
is still needed once drain keeps the covering mark.
A parser that assembles an access unit across PES packets has to answer
one question for every unit it emits: which packet carried this unit's
FIRST byte? Its timestamp comes from that packet, and so does the source
byte offset that says which clip of a multi-clip title it belongs to.
The packets that complete the unit carry later values that must not
override it.
That question was being answered three ways. DTS kept a deque of
(offset, pts) markers and took the one covering offset 0. AC-3 kept a
single carry-over timestamp. TrueHD kept its own. PGS and DVD subtitles
each held a pending unit with just a start time. None of them carried
the source offset at all, which is why provenance existed only for video
and why nine audio and subtitle tracks on a branched title had nothing
to place them by.
So it lives in one place now. PesBuf owns the bytes AND the marks;
PesFacts returns a packet's timestamp, source and discontinuity
together, so a parser cannot take one from one packet and another from
the next, because it does not assemble them itself.
The type answers WHICH packet. How a codec reads a timestamp out of that
packet stayed the codec's business at first, and that turned out to be
the same drift one level down: dvdsub read pts alone and returned 0 for
a packet carrying only dts, while everything else took pts.or(dts). Now
there is one derivation. It is not a choice between two fields — for
audio and subtitles there is no reordering, so dts IS the presentation
time and reading it is reading the same value from whichever field the
packet used. Reordering video never calls it; that path reconstructs
display order instead.
Migrated: adts, pgs, dvdsub. dts, ac3 and truehd follow.
Accepting a table whose marks do not advance was wrong, and a real
22-clip title showed exactly how. The rip failed the drop-volume gate at
84 percent with more frames dropped than kept.
The cause is a fact the code never stated: provenance is VIDEO-ONLY.
Every audio and subtitle parser -- dts, ac3, adts, truehd, pgs -- builds
its frames with `source: None`, while hevc, h264 and vc1 propagate it.
So on that title the video track placed correctly by byte offset and
nine audio and subtitle tracks arrived with nothing to place them by.
The branch added to keep an unprovenanced frame from stranding did the
stranding itself. It held such a frame on its track's current clip,
reasoning that the cursor only ever advances under provenance and so
could not be wrong -- which is true only for a track that eventually
receives some. A track that never receives any stays pinned to clip 0
for the whole title, and every frame past clip 0's OUT mark is dropped.
That is what the disc showed: nine tracks pinned to clip 0, the first
drop nine milliseconds past its OUT.
So the refusal goes back, now with the real reason recorded, and the
test asserts it rather than asserting the behaviour that failed.
Lifting it needs the audio and subtitle parsers to carry provenance
first, stamped from the PES that STARTED each access unit -- the same
rule au_assembly already applies to video. The diagnostics that found
this are kept.
The heuristic path has always logged its first drop per track with the
frame's timestamp and the clip marks it was judged against. The
provenance path did not log at all, so a title that dropped MOST of its
frames there failed the volume gate without a single line saying which
frame, which clip, or which marks — a full-length rip produced zero drop
events and an error, which is not a diagnosable failure.
Also pins the all-identical-spans hazard as a test. Every PlayItem
referencing one clip file gives every clip the same span, which passes
the tiling check and is therefore "trusted" while carrying no
information at all about which PlayItem a byte belongs to. That is not
what the discs on hand do — theirs have one distinct span per clip — but
the check conflates "the spans tile" with "the spans distinguish", and
only the second justifies placing a frame by its byte offset.
A seamlessly branched title re-references one clip file from several
PlayItems with different mark ranges, and those references share a single
feed span because the bytes are read once. A byte offset alone cannot
then tell them apart.
Whether that is happening on a given title is the fact that decides
whether provenance can identify a clip on its own, and it was not
observable. distinct_spans < clips says it directly.
Whether a title was placed by the seam plan or fell back to inference is
the single most useful fact about a branched rip, and it was invisible.
The two looked identical in the output, so telling them apart meant
rebuilding and re-ripping — which is exactly what it cost to find that
most branched discs were silently on the inference path.
Log both flags once at plan construction, and log each refusal with the
reason that caused it.
A playlist whose clips each restart their own STC has marks that all
cover the same low values, so they are not points on one title-wide
clock. from_clips refused those tables outright and fell back to PTS-jump
inference.
That refusal predates provenance, and it is only about inference. With
the feed spans tiling the title the clip comes from the frame's byte
offset and the marks are never read across clips: each clip's offset_ns
maps its own private clock onto the output timeline, which is exactly
the right operation for a restarting STC.
Refusing unconditionally therefore turned the seam plan off on most of
the branched discs on hand — the very titles it was written for. They
ran on inference with a timeline minutes past the title's real length:
one 2h29m title came out with its audio and video both spanning 3h15m.
So the refusal now applies only when the spans cannot be trusted, where
it is still load-bearing: without a usable clock or a usable byte offset
there is nothing to place with.
Accepting these tables opens one hole, and it is the same stranding the
refusal guarded against. A frame arriving with no byte offset would fall
through to the mark heuristics, which on this kind of table are
meaningless and can strand a track on a clip it has already left,
dropping the rest of the title. Such a frame now stays on its track's
current clip; the cursor only ever advances under provenance, which is
never wrong about which clip a byte came from.
The provenance lookup runs once per frame per track. A linear scan over the
clip list is fine for the 11-PlayItem fixture and is not fine for the real
hoard: two discs there carry 900 PlayItems in one title, and several carry
250-450, so a scan would be hundreds of comparisons on every one of millions of
frames.
spans_trusted already guarantees the spans tile the feed contiguously and in
order, which makes this a partition point. A repeated clip reuses its first
reference span, so the search walks back to the FIRST entry sharing it — the
bytes are read once, so the material is emitted once, at that entry offset, and
the answer is stable regardless of which duplicate the partition lands on.
Also adds a test at the real 900-clip scale: every clip places, output stays
monotonic across all 900, an arbitrary byte resolves to the right clip out of
ascending order, and a byte past the end belongs to no clip.
Four audit rounds each fixed one rule in SeamPlan::place and broke another,
because the question the rules were answering has no answer. Inside a
seamless-branching overlap clip k OUT comes AFTER clip k+1 IN — 57.8s of
overlap on the real fixture table — so a single timestamp is legitimately
inside two clips, and a clip file is not trimmed to its marks, so it also
carries material from before its own IN. No rule over timestamps can say which
clip a frame came from, and each attempt was right for one disc layout and
silently wrong for another: 65s of rewind, 17 minutes stranded, 28 minutes
dropped, 55s refused.
Frames already carry the byte offset they were read from (PesFrame::source,
stamped by the TS demuxer). Clip now carries the byte span its stream occupies
in the title feed, recorded while the extents are gathered. So the clip is a
LOOKUP: the offset falls in exactly one span. There is no decision to get wrong.
Every track of a clip lives in the same stream file and therefore shares one
span, so video, audio and subtitles agree by construction. Divergence between
them — each track guessing separately under its own tolerance — is how audio
and video ended up on different clips and drifted apart in the first place.
spans_trusted gates the whole path: unless the spans tile the feed contiguously
from zero, an offset means nothing and provenance is ignored in favour of the
mark heuristics, which is the 1.6.0 behaviour. A broken map degrades instead of
confidently selecting a wrong clip for every frame. A clip referenced twice
reuses its first span (the bytes are read once) and is still trusted.
Sources that stamp no provenance — a mkv:// remux, the deserialize hop — take
the heuristics, which is what they have always used and where they have always
been right, because they have no overlapping clips to be ambiguous about.
36 timeline tests, six of them new and covering: the overlap case marks cannot
see, all tracks agreeing, out-of-marks material dropped AND counted, a holed
span map, a discontiguous one, a repeated clip, and no provenance at all.
A symlink to a directory — a normal way to keep tens of gigabytes of streams
off the system disk — was dropped from the plan silently. `entry.file_type()`
reports the LINK, so such a subtree never enters `dirs`, and `metadata()` then
follows it and reports a directory, so it fails the is_file() test and was
skipped with nothing said.
The result is the worst class this release exists to close: PLAYLIST and
CLIPINF still synthesize, so the folder scans and enumerates titles, every clip
resolves to no extents, the mux takes its clean-EOF path, and a near-empty MKV
is written at exit 0.
It is still skipped rather than followed — following link targets invites
cycles and escapes from the folder — but at the same volume as the
unrepresentable-name skip eight lines above.
Round 7 enumerated the state space of SeamPlan::place and found three more
holes in the mark heuristics — the third consecutive round to find a defect
here, each time in the guard the previous round added:
- a clip file opening ON its own IN mark, after a tail frame past the previous
clip OUT had already advanced the cursor, read as another crossing and jumped
a clip too far: 28 minutes of clip 6 dropped;
- the last clip was unguarded entirely (`clip + 1 < len` skipped the check), so
the 9->10 seam rewound 75.6s with dropped_total() at 0 — the exact round-5
signature, reachable without any corrupt input;
- a glitch mid-clip satisfies "inside the current clip", so the round-6
stranding survived for the rest of that clip.
Three patches to the same heuristic, three wrong in a new way. So stop patching
it. Everything above the placement is still heuristics over marks, but the
property they exist to serve is now checked DIRECTLY: a track output must not
run backwards. A placement that would rewind by more than
DISCONTINUITY_BACKSTEP_NS is refused; the scan then looks for a LATER clip
containing the frame that does not rewind, which is exactly what a genuine
overlap crossing looks like; and if none exists the frame is dropped and
counted, because emitting it anywhere moves the track backwards.
The tolerance is the backstep constant because B-frame reorder legitimately
emits out of order by a fraction of a second, while a rewind that matters is
orders of magnitude larger.
The cursor moves ONLY when a later clip actually accepts the frame. My first
version of this advanced on a failed search, which strands a track exactly as
before — caught immediately by the round-6 regression test.
All 29 previous timeline tests still pass, including
continuity_preserves_forward_gap. The three round-7 scenarios now have a test
each, with the real 00801.mpls numbers.
The round-5 fix advanced the clip cursor on ANY backward step over 3s. That
also fires for a corrupt PTS, and for a legitimate STC discontinuity inside one
clip — and nothing moves the cursor back, because a forward step matches
neither past_out nor stepped_back. Every later frame then sits below the new
clip IN and is dropped: on the fixture table that is ~17 minutes of one track
gone, and the only volume gate compares total drops against ALL tracks frames,
so it exits 0.
The branch now also requires the frame to be INSIDE the current clip marks,
which is the only case that would otherwise be silently placed at the old
offset — the rewind. A frame outside them needs no help: the containment check
drops and counts it, the cursor stays put, and the next good frame is placed
normally. Both behaviours have a test, each confirmed to fail without the guard.
Separately, the dir:// PES input path scanned the folder and never applied the
encryption verdict scan_dir exists to produce, so the same folder ripped
through one door and failed through the other asking for a key it does not
need. That logic now lives in one function, session::apply_folder_encryption_verdict,
called by both.
The previous commit bounded the crack scan inspection by the bytes actually
read, which stopped it examining stale buffer bytes from an earlier batch. But
the cursor still advanced by the REQUESTED count, so those sectors were skipped
outright — trading "scans the wrong data" for "silently scans less than it
thinks", on exactly the damaged media where a title key is hardest to find.
The cursor now advances by what was read, so the next iteration resumes where
the read stopped. Floored at one sector so a source returning Ok(0) cannot spin.
Found by reading the fix again rather than by the next audit round.
read_sectors returns the number of bytes written and a source may return Ok
with fewer than asked — a recovery read over a damaged region does exactly
that. The crack loop discarded the count and inspected all n sectors of a
buffer that is REUSED across batches, so the tail still held the previous
batch sectors.
Cracking a key from those means cracking from data belonging to a different
extent, possibly a different VTS, while crack_span records the CURRENT one. A
key that opens nothing in this region is then installed, and its wrong
descrambles are only partly caught by the per-sector crib.
descramble_region was fixed to require the MPEG-2 pack start code, but
descramble_sector — the PUBLIC per-sector entry point, and the one the
module-level example tells callers to use — still keyed on the byte 0x14 flag
bits alone. The crate own documented guidance therefore led straight back into
the defect this release exists to fix: a VIDEO_TS.IFO sector holding 0x15 at
0x14 while starting 00 26 00 00 loses 1912 of its 2048 bytes, and because that
sector carries TT_SRPT the disc enumerates 38 titles while an image decrypted
from it enumerates 10, at exit 0.
It has no callers inside the crate, which is exactly why it survived three
rounds: nothing exercised it. Putting the guard inside the function rather than
in each caller is what keeps the safe path the easy one.
The integration test that covers it built its sector from the flag byte alone,
which no real scrambled sector looks like, so it stopped representing the path
it names — the same fixture-realism gap already fixed in four other places this
release.
A hole in the seam fix itself, found in audit round 5 and confirmed by
measurement against the real 00801.mpls marks already in the test file.
Nothing upstream trims a clip stream to its marks, so a clip file can open with
material from BEFORE its IN. With the cursor on clip 5 (7708.99..7910.79) a
frame at 7845.00 — clip 6 pre-mark lead-in, 8s below clip 6 IN of 7853.00 —
sits further from that mark than the 250ms tolerance, so neither crossing rule
fires. The cursor stays on clip 5, and 7845.00 IS inside clip 5 range, so the
frame is PLACED with clip 5 offset.
Measured: output went from 3714.51s to 3649.51s. Backwards 65 seconds, with
dropped_total() still 0 — no counter, no gate, nothing noticed — while the
whole 57.8s overlap band was emitted a second time over clip 5 written tail.
That is the collide-and-flatten symptom this type exists to remove.
A track own PTS only ever runs forward inside a clip, so a backward step larger
than DISCONTINUITY_BACKSTEP_NS is always a clip change. Such a frame now
advances to the first clip that could contain it — which terminates on its own,
rather than running to the end of the list, which is what the next_in bound
exists to prevent — and a frame still below that clip IN is material the
playlist excludes, so the containment check drops and COUNTS it.
All 27 pre-existing timeline tests still pass, including
continuity_preserves_forward_gap, which is the one that catches an over-eager
heuristic here. The new test asserts the output never moves backwards at a
join and was confirmed to FAIL before the fix.
The same gap as the MKV muxer: DemuxSink builds its timeline with
TimelineContinuity::with_clips, but no test ever handed it a title carrying
usable PlayItem marks, so frames_mapped and dropped_total() were always zero
and both gates in finish() were unreachable.
Deleting either left the whole suite green while a demux:// export of a
seamless-branching title wrote a directory of zero-byte track files beside a
populated chapters document and reported success at exit 0.
The test gives the title two clips with marks that exclude every frame, and was
confirmed to FAIL with the gate removed.
Audit finding: both seam gates in MkvMuxer::finish were dead code under test.
set_clips was never called anywhere in the suite, so continuity.dropped_total()
was always zero and neither branch could be reached.
The ordering of the two zero-frame checks is load-bearing and the comment says
so: a title the seam plan emptied also has a zero frame count, and MkvInvalid
is classified by is_skippable_title_stub as an empty nav/menu stub — so
reporting it that way makes an all-titles rip drop a real feature and finish the
rest at exit 0. Swapping the checks left the entire suite green.
The new test builds a two-clip plan (from_clips needs at least two, strictly
increasing) whose marks exclude every frame, and asserts both that the error is
SinkWroteNothing and that it is NOT classified as skippable. Confirmed to FAIL
with the two checks swapped.
Auditing my own fixes found all three. None were in the original code.
The VTS crack sort was byte-wise case-SENSITIVE while the filters that select
those files (vts_group_of / is_title_vob) are case-insensitive. On a
case-sensitive volume a set holding vts_01_1.vob beside VTS_01_2.VOB sorted
part 2 first, because V (0x56) precedes v (0x76) — reintroducing exactly the
budget-exhaustion the ordering exists to prevent. Now sorted on the same
uppercase normalisation the filters apply.
Refusing a name that round-trips to empty aborted the WHOLE plan. A sidecar
folder named with a single emoji made a backup un-rippable that 1.6.0 handled
fine, and reported it as a collision with a file that does not exist. It is now
skipped with a warning: the entry is unaddressable either way, but one
irrelevant file should not cost the user their rip.
The mtime check now applies only to files whose CONTENT the plan read — the
IFOs, whose bytes 0xC0/0xC4 place every VOB. Everything else is planned from
size alone, which is already checked, so comparing mtime there bought nothing
and risked a real false positive: disc backups commonly live on exFAT/FAT32,
which stores local time, so a long rip spanning a DST transition would see a
whole-hour shift on an untouched multi-gigabyte VOB and abort hours in.
Two dirimage findings, both silent-wrong-output.
The per-directory uniqueness check compared raw host names, but the reader does
not see raw host names: parse_udf_name trims leading and trailing whitespace
and drops any code unit char::from_u32 rejects — which is every half of the
surrogate pairs the encoder emits for non-BMP characters. So " 00000.m2ts" and
"00000.m2ts", or "A<astral>.m2ts" and "A.m2ts", were two entries at plan time
and ONE name at read time. find/read_file take the first match, so a title
resolved to the wrong file extents and muxed the wrong bytes at exit 0 — the
exact shadowing DirNameCollision exists to prevent.
The key is now derived by round-tripping the name through the very encoder and
parser that will be used, so it cannot drift from them. A name that survives
that round trip as empty is refused outright: it would exist in the image and
be addressable by nothing. The new test builds a real folder, calls plan, and
was confirmed to FAIL with the fix reverted.
Separately, the plan-vs-read revalidation compared file LENGTH only, while the
plan depends on CONTENT: a DVD VOB placement comes from bytes 0xC0/0xC4 of its
IFO, and IFOs occupy a whole number of sectors so an in-place rewrite keeps the
length. A re-authoring tool touching the folder mid-rip would pass the size
check while every title extent pointed at stale sectors. mtime is now compared
alongside size, and only when both sides report one, so a filesystem without
timestamps falls back to the old behaviour rather than failing every read.
probe_folder_encryption sampled AACS units starting from the largest extent
anywhere in the disc. AACS units are 3 sectors and a unit boundary is only
guaranteed at the START of a clip, so that anchor is only correct when the
largest extent happens to be a clip first extent.
For any clip over ~2 GiB it is not. The planner caps an allocation descriptor
at MAX_AD_BYTES = 524287 sectors, so every full piece of a split file ties on
sector_count and max_by_key returns the LAST tie — an extent beginning
(k-1)*524287 sectors into the file. 524287 % 3 == 1, so that start misses the
unit boundary for two file sizes in three.
The 6144-byte sample windows then begin mid-source-packet and the byte read as
the CPI flag is content. Both verdicts fail in a costly direction: a decrypted
folder is rejected as encrypted (DirImageEncrypted on something perfectly
rippable — the exact case scan_dir was added to rescue), or real ciphertext
reads as clear and the mux writes it out as video at exit 0. is_unit_aligned
cannot catch it: it measures against the same wrong base.
Now the largest TITLE first extent, which is unit-aligned by construction and
is also the more meaningful sample — the main feature rather than whichever
fragment happened to be biggest.
The existing tests could not reach this: their fixture m2ts is 786,432 bytes,
a single extent, which is always its own first.
Round 1 removed a largest-first sort from resolve_vts_key because
largest-first is the 1.5.1 garbage bug. Auditing that fix showed it was only
half right: `planned` comes from walking the UDF directory, which yields File
Identifier Descriptors in on-disc authoring order with nothing sorting them. So
deleting the sort did not restore playback order, it left the order undefined —
whatever the disc happened to list first.
DVD-Video numbers a title sets VOBs in playback order by spec (VTS_xx_1.VOB ..
VTS_xx_9.VOB, single digit), so ascending filename IS playback order and is
deterministic regardless of how the directory is laid out.
Order decides correctness here: crack_key shares one sector budget across the
whole extent list, a CSS DVDs biggest cell opens with a long clear run, and CSS
recovers the title key from scrambled data itself. Starting in the wrong place
can exhaust the budget without ever meeting scrambled data, whereupon the
caller falls back to the disc-wide key and the whole VTS is descrambled wrongly
— corrupt PES behind an intact header, written out at exit 0.
Four fixes from the first audit round. Every finding was verified against a
pinned tree and read directly before being accepted.
resolve_vts_key sorted a VTS title-VOB extents largest-first. That is the 1.5.1
garbage bug, and it grew back in a new code path: the comment claimed it
"matched the scan heuristic", but that heuristic WAS the bug and had already
been fixed in decrypt_keys_for_title, which documents the rule (PLAYBACK ORDER,
never largest-cell-first) and pins it with a regression test. A CSS DVDs biggest
cell opens with a long clear run and crack_key shares one sector budget across
the extent list, so starting there can exhaust it without ever MEETING
scrambled data — and CSS recovers the key from scrambled data itself. The crack
then returns None, the caller falls back to the disc-wide key, and every VOB in
that VTS is descrambled wrongly: corrupt PES behind an intact header, written
out as a complete extract at exit 0.
parse_pgcit dropped titles silently in THREE places — an unparseable PGC, an
out-of-range PGC index, and a truncated entry table. The finder caught one; the
other two turned up on reading the function. parse_vmg already counts and warns
per skipped title SET for exactly this reason, and this was the last place a
disc could quietly report fewer titles than it has.
write_image called flush() and returned Ok. flush() only pushes bytes into the
page cache and promises nothing about durability, so a 6-90 GB image could be
reported complete while still unwritten — a crash or an unmounted volume then
leaves a truncated file the caller was told was finished. Now into_inner (so a
buffered-write error surfaces instead of being dropped by BufWriter::drop)
followed by sync_all.
timeline used abs() on a saturating_sub result. Every other comparison in that
module is saturating because the timestamps come off a disc and are not
trusted; abs() panics on i64::MIN, which saturating_sub can produce.
The 1.6.1 DVD fix removed the USE of the loose predicate but left the
predicate sitting there with the better name. Anyone asking "is this sector
scrambled?" finds is_scrambled before is_scrambled_pack, and reintroduces the
defect that destroyed 1912 bytes of a real VIDEO_TS.IFO — the sector carrying
TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it
enumerated 10, silently, at exit 0.
Byte 0x14 only means "scrambling control" inside an MPEG-2 pack. In an IFO,
UDF or ISO 9660 sector it is whatever that format stores there. The new name
says what the function actually tests and nothing more, so the honest question
has the obvious name and the dangerous one has to be asked for deliberately.
Its doc comment also claimed a caller, decrypt::decrypt_sectors, that does not
exist — so the name was an invitation and the documentation was an argument
for accepting it. It has no production callers at all; it stays public because
an integration test asserts the flag extraction directly.
The first cut ran release-profile tests on all three platforms everywhere,
which invented coverage no crate had ever claimed. bdemu is Linux-only — its
ci.yml has no macOS or Windows job at all, and its tests call
ExitStatus::signal(), which does not exist on Windows, so they do not compile
there let alone run. The rest follow the policy ci.yml already set: tests
execute on Linux and macOS, Windows compiles them so the cfg(windows) halves
still get codegen before release time.
That policy is now explicit in qa.yml as a windows-build job rather than
implied by which jobs happen to exist.
Recorded, not papered over: running the suites on Windows DOES find real
defects — autorip has five Windows path-handling failures in mover.rs and
freemkv-engine has three tests hardcoding /dev/null. freemkv ships a Windows
GUI, so they matter. They are pre-existing rather than 1.6.1 regressions, and
fixing them is its own piece of work.
Also build the binary before cli-parity.sh. cli-integration.sh builds its own;
cli-parity.sh only checks for one and exits 2 with "build it first", so both
legs died before running a single case.
Three release-profile failures on macOS and Windows, all found by the qa gate
on its first run. Release-profile tests on those platforms had never run
before it existed, so none of these were regressions — they had simply never
been visible.
available_space returned None on every non-unix target. That did not merely
skip a test, it skipped the GATE: a Windows user extracting a disc to a full
volume got a confusing failure part-way through instead of a clear refusal up
front, and Windows is where the GUI ships. GetDiskFreeSpaceExW is declared
directly against kernel32, matching how scsi::windows already reaches Win32
rather than pulling in a binding crate for one call. It asks for
FreeBytesAvailableToCaller, which accounts for per-user quotas — the same
question f_bavail answers on unix.
The other two asserted on wall-clock timing with no margin:
- sleep_until_halted_wakes_mid_sleep bounded the wait at 350 ms and measured
377 ms on a loaded runner. That bound measures the scheduler, not the wake.
What the test is for is distinguishing "woke because the flag flipped" from
"woke because the 10 s timeout expired", and 2 s does that just as well.
- abandon_loses_to_a_close_already_committed released at 600 ms against two
300 ms grace windows plus a 250 ms poll cadence, so the windows could expire
first and the caller abandoned — a race, not a defect. The intervals are
scaled up so jitter is small relative to them; the ordering under test is
unchanged, only the margin.
Every push to qa now tags v<version>-rc<N>, N incrementing, before the gates
run. That answers "which build is on qa, and is it the one I tested?" without
anyone having to remember it.
The tag lands whether the run goes green or red, deliberately. A red candidate
needs a name more than a green one does: "rc3 failed release-tests on windows"
is a sentence you can act on, "qa is red" is not. Red on qa is the gate doing
its job — the branch saying this is not production worth yet.
release.yml now excludes v*-rc*. Its trigger was v*, which matches the
candidate tags, so without this every push to qa would have built and PUBLISHED
a GitHub release — including for the candidates that failed.
dev -> qa -> main, across every public repo.
dev is where work lands and is meant to be pushed to often, so ci.yml stays
the fast answer: fmt, clippy, unit tests, leak-guard. qa is the release
candidate, and qa.yml is the claim that a commit is production worth —
release-profile tests on all three platforms, and the Linux cross-target
clippy that the local precommit gate has always run but CI never did.
Release profile matters as its own gate: overflow checks are off, debug_assert
is compiled out, and inlining changes what the optimiser can prove. A test
that only passes in debug never guarded the binary anyone ships.
qa.yml checks siblings out at qa rather than dev. This is not a monorepo, and
a qa run resolving its dependencies from unreleased dev tips would go green on
a combination that is not the one shipping — the exact mismatch the branch
model exists to prevent. ci.yml now tracks whichever branch triggered it for
the same reason, since it also fires on qa.
The consequence is that repos move to qa together, in dependency order. qa is
backfilled from main so an unchanged crate still presents working code to the
crates built on it.
css::is_scrambled reads bits 4-5 of byte 0x14 and nothing else. That is a
sound test once a caller has committed to a title's VOB data, where every
sector is an MPEG-2 PS pack and byte 0x14 always means what it says.
descramble_region is not such a caller: it is handed arbitrary regions of a
disc, so it also sees IFO, UDF and ISO 9660 sectors — raw structures where
byte 0x14 is whatever that format happens to store there.
Measured on a real disc: the second sector of VIDEO_TS.IFO holds 0x15 at
offset 0x14 while starting 00 26 00 00, which is not a pack. The flag test
read it as scrambled, descrambled it, and destroyed 1912 of its 2048 bytes.
That sector carries TT_SRPT, so the title table went with it — the disc
enumerated 38 titles and an image decrypted from it enumerated 10, silently,
at exit 0.
is_scrambled_pack already existed with the right predicate. Use it here. It
costs nothing: a genuinely scrambled VOB sector always carries the pack start
code, and no IFO sector does.
Verified end to end — the decrypted image's `info` output is now identical to
the source disc's, 38 titles both, differing only in the CSS: Encrypted line.
The fixtures moved with it. Four of them built a sector by setting byte 0x14
alone, which no real scrambled sector looks like; they now build packs.
The diagnostic hashed IFO contents but not where they came from, so a
content difference could not be told apart from a path resolving to a
different place. It now prints file_start_lba and file_extents next to the
hash.
That measurement is what inverted this investigation: both a CSS disc and
its decrypted copy resolve VIDEO_TS.IFO to the same extent, and only one
sector of the fourteen differs — by exactly a descrambled payload with the
scrambling-control bits cleared. So the decrypted copy holds the correct
bytes, and an iso:// scan of a CSS disc is parsing a still-scrambled IFO
sector, because scan_iso opens a plain FileSectorSource.
Two opt-in dumps, driven by FMKV_IMAGE, that narrow where a decrypted DVD
image loses titles:
dump_title_sets_for_an_image reports what survives parse_vmg. On one disc
the CSS image yields 13 title sets and 38 titles; its decrypted copy
yields 8 and 10. Sets 8, 9, 10, 12 and 13 are dropped outright, set 11
parses but returns no titles at all, and set 7 returns 3 of 5.
dump_vts_ifo_reads_for_an_image reads every VTS IFO and hashes the
CONTENT. All thirteen are byte-identical across the two images, so
parse_vts is handed the same bytes and the same TT_SRPT info and still
fails on one of them — the divergence is in what it reads from the READER
afterwards, which is file_start_lba and the PGCIT.
An earlier version of the second dump compared only length and magic and
so wrongly reported the images as identical; it hashes the bytes now.
parse_vmg skipped any title set whose parse failed, with no log and no
counter. A real disc enumerated 38 titles from one image and 10 from
another, and the 28 discarded failures were invisible — the symptom read
as a scan difference rather than as dropped reads, which is most of why
it took so long to localise.
Behaviour is unchanged: a disc may legitimately carry placeholder TT_SRPT
entries, so one failure is still not fatal. It now warns per skip with the
title set and the error, and once at the end with kept-versus-declared.
Also adds an opt-in diagnostic that reads every VTS IFO from an image, to
separate a read failure from a parse failure. It reports all 13 sets
reading identically from both a CSS image and its decrypted copy, which is
what proves the 38-to-10 loss is downstream of these reads.
dvd_placement_invariant_on_a_real_folder checks, per title set, the sum
ifo.rs relies on: file_start_lba(VTS_nn_0.IFO) + vtstt_vobs must land on
VTS_nn_1.VOB. It reports all 13 sets correct on a real DVD folder, which
is what excluded placement as the cause.
dump_titles_for_an_image prints every title a scan produces with the
numbers canonical_title_order sorts on. It is what showed the real shape:
the same disc scans to 38 titles as a CSS image and 10 once decrypted,
with identical capacity and a byte-complete image.
Both are #[ignore]d and read their target from the environment, so they
cost the gate nothing and are there for whoever picks the defect up.
The zero-frame check ran before the seam gate, and its error is
classified as a skippable nav stub — so a title the plan dropped
ENTIRELY was reported as an empty stub and an all-titles rip would omit
a real feature and finish the rest at exit 0. The seam case is decided
first now, with a code that is not skippable.
The demux sink read a frame's track kind out of the FILTERED slot, which
is empty for a class the export drops. On an audio:// or sub:// export
the video track was therefore called non-video and handed the permissive
crossing rule — the same defect round 4 fixed for a Dolby Vision layer,
reintroduced one file over. Video tracks are now recorded before the kind
filter, beside the primary-video reference that exists for this reason.
Its frame counter counted frames PLACED, not written, while its name and
doc claimed otherwise. Renamed and documented for what it is, including
that it cannot see a single lost track among many.
Placement: files in a subdirectory of VIDEO_TS were never given data.
They were declared at full size with no extents, so they appeared in the
tree and read as nothing. The same folder under BDMV was always placed
correctly. And the duplicate title-set guard keyed on the constraint maps,
so an IFO declaring no offsets inserted nothing and a colliding second IFO
went undetected — it keys on the groups seen now.
Display for SeamPlanDroppedMost and ShortImageRead discarded their
payloads, and four new variants were missing from the code-uniqueness
test.
The cap was added without a test. Its companion — a modest oversize is
honoured as a gap — already existed, so this pins the other side: an
offset past the ceiling is refused rather than grown into, which is what
keeps a rewritten IFO from planning a multi-terabyte image.
Round 3 counted frames the clip marks excluded and reported them at
finish. Counting is not bounding: the only other gate was a global
zero-frame check, and its error is additionally classified as a skippable
nav stub, so a title whose marks do not line up with its PES clock could
discard almost all of itself and still exit 0 — a two-hour feature
emitting seconds, which is the defect this change set already shipped
once. Dropping more than was kept is never a real join, so it now fails.
The demux sink had no zero-output guard at all, so a fully-dropped title
finished cleanly: a directory of zero-byte track files beside a populated
chapters document. It now refuses, keyed on frames having been OFFERED —
a chapters-only export, or a track class the title does not carry,
legitimately writes none, and two existing tests correctly said so.
A title set's placement group is the parsed number, so VTS_01_0.IFO and
VTS_1_0.IFO land on one key and the second silently overwrote the first's
constraint, placing a VOB where the IFO the reader uses does not point.
Refused rather than resolved by arrival order.
A Dolby Vision enhancement layer is a second video track: it does not
drive epochs, but it does carry B-frame reorder. The crossing rule was
keyed on driving epochs, so the EL took the branch whose premise is that
the track has no reorder — and its ordinary reorder dip near the end of a
clip, which during an overlap also lands inside the next clip's range,
was read as a join. The EL was then placed on the next clip's offset, out
of step with the base-layer frame it must be co-timed with by the width
of the overlap: the same desync the per-track cursor was added to remove,
reintroduced for one track.
The property the rule actually depends on is whether a backward step can
be reorder, so it is now keyed on that. Both sinks derive it from the
track kind rather than from the epoch driver.
Two findings from the same escalation, both silent-wrong-output.
The plan was built for every multi-clip title. Only a Blu-ray PlayItem's
IN/OUT are positions in the clock the PES PTS runs on. HD-DVD fills the
same fields from the XPL's title-relative times and a DVD's come from
cell tables, so a plan built from them is an identity map with a drop
filter: it suppresses the layer-break rebase inference performs, and
drops whatever falls outside marks the PTS was never measured against. An
earlier reading of this called HD-DVD safe because its marks are
contiguous and every computed offset was zero — true, and irrelevant,
because they were zero in the wrong clock. Gated on the content format,
with a test using an HD-DVD-shaped table that the clock check alone
accepts.
The crossing test was also one-shot. A table whose clips restart their
own bases could miss it, and a missed crossing STRANDS the track: every
later frame falls outside the stranded clip's marks and is dropped for
the rest of the title. Counting drops, which is all the previous round
added, does not bound them. A table that is not one advancing clock is
now refused outright and falls back to inference, which is the documented
safe path for those titles.
Also from the same round: read_sectors added an unchecked lba + i, where
callers deliberately saturate their LBAs — a wrap folds the read back to
a low sector and hands the muxer another file's bytes. classify added 1
to two numbers parsed verbatim out of a filename. read_head used a single
read() where a short read on a network mount silently records no
placement constraint at all. And the page-cache eviction added last round
released only the read that crossed its threshold rather than everything
accumulated, so seven eighths of what was read stayed pinned.
The round-1 tests asserted arithmetic about MAX_CS0_NAME_BYTES and never
called plan(), so both would have passed with the guard deleted — which
is the failure mode this audit exists to catch, committed by the audit's
own fix. They now build a real folder containing a 255-byte name and
require the planner to refuse it, plus a companion proving a name at the
cap is still accepted so the guard is not merely refusing everything.
The subdirectory cap keeps its arithmetic-only test — creating 65,535
directories is not reasonable in a unit test — but now says so instead of
implying coverage it does not have.
The folder encryption probe returned "not encrypted" when it had sampled
nothing at all — a title shorter than one aligned unit skipped the loop
entirely. That verdict CLEARS the structural one an AACS directory
raised, so a genuinely encrypted folder would have been ripped as clear
and written ciphertext as video at exit 0. With no evidence it now keeps
the structural verdict, and its bounds arithmetic no longer trusts
disc-derived values not to wrap.
Reading an IFO header swallowed every I/O error and returned an empty
buffer, which sent each placement offset through unwrap_or(0) and
recorded no constraint at all — a permission error on one file produced a
silently misplaced VOB. The directory walk swallowed the same class while
claiming to skip only vanished files. Both now propagate; only NotFound
is skipped.
Four things the round-1 changes left inconsistent: two new error codes had
no doc comments, were absent from the io::Error mapping, printed no path
in Display, and were missing from the test that proves codes are distinct.
The demux sink dropped frames silently while the MKV muxer reported them.
And set_clips had been inserted INTO write_frame's doc comment, leaving
write_frame undocumented and its paragraphs describing the wrong function.
uid/gid used 0 as "not specified"; UDF's sentinel is 0xFFFFFFFF, and 0 is
root.
A DVD title set records where its VOBS begins as an offset inside its own
IFO, and the planner honours that offset because honouring it is what
makes a real backup readable. Nothing bounded it: a regenerated .BUP or a
hand-assembled folder naming an offset far past the content grew the
image to wherever it pointed — a u32 sector count reaches ~8.8 TB, and
writing that to an iso:// destination fills a disk with zeros before
anything notices. Capped at 128 GiB, which clears BD-100 with room.
Metadata is materialized up front and held for the life of the image, at
a 2 KiB File Entry per node, so the 100,000-entry cap alone permitted
~205 MB of it for content of no size at all — and the mux holds two
images at once while probing. The module claimed a budget of a few MiB;
that budget is now enforced rather than asserted.
Host reads had no page-cache eviction. The ISO source documents what that
costs, measured: an 85 GB read pins the whole file, starves the writer,
and collapses the mux to 2.7 MB/s against 70 MB/s isolated. A folder
source reads host files the same way, so it now uses the same eviction —
the hints move from private-to-that-module to crate-internal rather than
being reimplemented.
The placement code skips that field for the Video Manager, which is
right, but said it was skipping it because the field is the menu cell
address table. It is the title search pointer table, and it is an offset
inside the IFO rather than a pointer to another file — which is the
actual reason it constrains nothing. ifo.rs reads the same offset under
the correct name, so the two would have drifted.
A sparse passive track — a subtitle with no event near a clip's mark —
was held to the dense-video crossing window, so it stayed on the previous
clip's offset until its PTS passed that clip's OUT and every event in
between was mistimed by the overlap. Video keeps the tight window,
because its backward steps are also B-frame reorder; passive tracks have
no reorder, so any backward step into the next clip's range is a join.
Frames the marks exclude were dropped without a trace. Dropping is right
at a join, but this codebase has shipped complete-looking wrong output
before, so the count is kept per track and reported when the mux
finishes, alongside the pre-cluster counter that exists for the same
reason.
A File Identifier Descriptor records its name length in one byte, and the
length was narrowed with a cast: a 255-byte name — POSIX NAME_MAX,
entirely ordinary — encodes to 256 and wrote zero, which would read every
later entry in that directory from the wrong offset. A directory's link
count is 16 bits and was computed as 1 + subdirectory count, which the
global entry cap alone permits overflowing. Both are refused while
planning, where the tree can still be rejected cleanly.
The module and struct docs described inference as the whole algorithm;
they now say which path decides what.
The folder reader produces a wrong title list on a real DVD — the title
table is read at a sector offset relative to VIDEO_TS.IFO's start, and a
synthesized layout does not put that file where the offset resolves the
same way. Output is silently wrong at exit 0, so the feature is not
wired to the CLI in this release. The library module stays, unreachable,
and ships once both disc families are verified against real content.
The first cut placed every track with the cursor the video had moved. At
an overlap join the previous clip's audio is still arriving after video
has crossed, and those tail frames sit inside both clips' mark ranges —
so they took the new clip's offset, jumped forward by the overlap, and
collided with the new clip's own audio, which the muxer's monotonic nudge
then flattened. A remux confirmed it: the timeline length was already
correct and the original symptom was still there, 169 audio packets on
the tick floor.
A track's PTS only runs forward inside a clip, so its own backward step
to the next clip's IN is its crossing. That is per track, so the cursor
is too.
A seamless-branching title's PlayItems do not chain contiguously: one
clip's OUT can sit after the next clip's IN, where the disc stores the
join twice, or before it, where the playlist skips material. The mux
never saw those marks — its own header said so — and inferred seams from
PTS jumps instead.
Inference cannot recover this. A forward jump is ambiguous: it means the
playlist skipped, or it means frames were lost to damaged media, and
compressing the latter would falsify timing on exactly the rips that most
need it faithful. An overlap smaller than the B-frame reorder threshold
is invisible to inference entirely, and its duplicate content then
collided in the muxer, where the monotonic nudge flattened a run of audio
onto the tick floor and put sound ahead of picture for the rest of the
film.
Measured on one 11-PlayItem title: the file declared 7893.385 s, which is
what the playlist says the title is, and carried packets to 8029.298 s.
Both numbers came from the same program on the same disc. Four skips
totalling 135.9 s became dead timeline, and a 1.79 s overlap put audio
1.8 s ahead at the half-hour mark. Five of forty-seven titles were
affected; every single-clip title was exact.
So the marks are read. Each clip contributes exactly out - in, laid end to
end, so the output runs as long as the playlist says and a join never
rewinds. Titles without usable marks — DVD, HD-DVD, file sources — keep
the inference path unchanged, and clips that already chain contiguously
produce a constant offset, which is pinned by a test.
Users keep discs as extracted folders — a DVD VIDEO_TS or a Blu-ray BDMV,
usually a backup that is already decrypted. dir:// could only ever be a
destination, so those folders could be produced and never read back.
Everything above the sector layer wants a UdfFs over a SectorSource, and
every UdfFs read re-reads the ICB off that source at call time, so a
folder has to present itself as sectors. It does: dirimage plans a block
layout over the real files, encodes a UDF 1.02 filesystem for the
metadata, and serves data straight from disk. read_filesystem then parses
it exactly as it parses a disc, so nothing above changes — and the
iso:// arm of input() is now shared rather than duplicated, so dir://
inherits its decrypt gates, title selection and stream pruning.
The encoder is validated by more than its own reader: macOS mounts the
synthesized image and the mounted files compare byte-identical to the
originals. A round-trip through our own parser could not have shown that
— the tag CRC seeds at zero, and a wrong seed would satisfy us and no
real driver.
DVD placement is not free packing: a VTS IFO records where its title
VOBS begins relative to itself, so the VOB has to land exactly there.
Unsatisfiable marks fail loudly rather than misplace the file. 3D folders
are refused for now: the scanner detects SSIF and the planner cannot
alias its extents yet, so accepting them would produce quiet nonsense.
Left for later: metadata capture, HD-DVD, FMTS, encrypted folders.
Writing an iso:// meant a raw sector copy off a drive, so an existing
image could only ever be a source. Decrypting one you already had meant
finding the disc again.
The engine's copy is the recovery path — mapfile, multipass, damage-jump,
auto-resume — and all of it exists because optical media returns read
errors. A file does not, so a non-drive source gets write_image instead:
sectors in, bytes out, once, no recovery machinery. Keeping them apart is
not just tidiness. The mapfile identity check compares AACS unit keys and
the VID, both empty for an already-decrypted source, so identity passes
for any such source — a second run with different input to the same
output path would resume over the previous image and report success.
A short read is an error rather than a zero-fill: padding a truncated
source yields an image that looks complete and is not, which is the worst
outcome for a copy someone means to keep.
`dvd_time_t` stores H:M:S:F as non-drop-frame timecode with a rate flag,
not elapsed wall-clock time. On NTSC the seconds field advances every 30
frames, but a frame lasts 1001/30000 s, so 30 frames occupy 1.001 s of
real time. Reading H:M:S as literal seconds under-reports real time by
exactly 0.1% — 3.6 s per hour, growing with elapsed time, which is what
made chapter marks drift ~4 s by the 67-minute mark.
Convert the whole timecode through an integer frame count and apply the
exact 1001/30000 fraction once, instead of reading H:M:S literally and
dividing only the frame remainder by a decimal 29.97. Chapter marks sum
per-cell frame counts as integers and convert once per chapter, so every
mark lands on an exact frame boundary instead of accumulating f64
rounding across a long title.
PAL is arithmetically unaffected: 25 frames = 1.000 s exactly, so the
old and new paths agree to within one ULP. Sweeping all 900k H:M:S:F
combinations under 10 hours, 12 differ, by at most 8.9e-16 s — the new
path divides one exact integer instead of adding a rounded fraction to a
large one, so where they differ it is the more accurate of the two.
Five existing NTSC fixtures asserted the old literal-seconds values and
were updated to real seconds (each is exactly 1.001x its old figure);
`bcd_secs` now documents that its argument is timecode, not real time.
Fixes#1
Reported-by: AnimeFN <admin@animefn.com>
Entries had grown into investigative narratives — root causes, disc
counts, PIDs, percentages, function names — which is the right level for
a post-mortem and the wrong one for a public changelog. Each entry is now
one to three sentences leading with what a user observes.
Halves the file (1538 -> 735 lines) without dropping a version. API
signature changes stay explicit, since a consumer migrates against those.
A vendor label's `stream_number` is a slot in the one stream table its
config blob describes. The labels merged in from the playlists — to
cover streams the vendor named nothing for — carried a different number
entirely: a dense counter over every distinct stream found while
scanning the whole disc in directory order, related to no playlist's
slot numbering at all. Two coordinate systems, one field name. The
merge matched them by equality and the binder then counted streams
against the result.
Measured over the 44-image corpus: 22 discs merge such labels; of the
566 places one lands on a stream, 443 (78%) are a stream it does not
describe — the label states the PID it read itself from and it is a
different one. 142 of those are stopped by the language check 4da6d3a
added; 301 are applied. Those labels carry no editorial payload, so the
direct damage is confined to codec text — but the polluted list is also
what the anchor gate reads, and on 11 disc/stream-type pairs it is what
decides the anchor, which is how it reaches the vendor's forced and SDH
flags. 53 anchor facts are harvested off a merged slot. The clip-info
orphans had the same shape, numbered from `max + 1` of a list they share
no coordinate system with.
A label now either NAMES its stream — `StreamId { clip, pid }`, read out
of the very table `disc::bluray` builds the stream from — or it does
not, and only the ones that do not are ever reached by counting:
* `label_at` returns vendor labels only, so the two numberings can no
longer be confused by construction.
* Playlist and clip-info labels bind by id. Exact, no anchor, no
sequence, no ordinal.
* A named stream outranks a guessed one, so an editorial flag reaches
a stream only where the disc's own numbering puts it there.
* The presence of an id is the provenance the anchor gate was missing.
It reads the vendor's slots alone now, so a slot the vendor never
named no longer breaks the sequence — under-yield is the normal
shape of these blobs — and a title with fewer streams than the list
has slots is no longer eligible to hold it. Ranking prefers the
title that positively confirms most of the list.
* An orphan is in no playlist, hence in no title, so it binds to
nothing rather than to whatever counted its way.
The MPLS floor is one label per physical stream keyed by `(clip, PID)`,
not by `(type, language, codec, pid)`: the same PID in two clips is two
streams, and the old key collapsed them. Its `stream_number` is now the
entry's real slot in its own playlist's table.
41 of 44 images are byte-identical; all three that move lose a label
they should not have had, and no feature title changes on any image. A
dozen featurette playlists stop reporting a feature subtitle's SDH
marking on their own unrelated subtitle; eleven menu and bonus titles
stop advertising the feature's object-audio format on plain stereo; and
on a disc whose every title carries a single audio stream — too short a
table to anchor anything — a regional-variant tag asserted on all
seventeen titles is now asserted on none, in exchange for every title
stating the codec it actually carries, which none of them did.
Six tests had been asserting the invented numbering, including one
pinning the disc-global counter as a deliberate property.
Also fixes a defect in the same family that the corpus work surfaced:
`pgs_forced_probe::apply_verdicts` set `forced` but left `qualifier`,
so a demoted track shipped with a metadata sidecar calling it forced
next to a Matroska header saying it is not. Only a forced claim is
cleared; an SDH marking is not the probe's to touch.
One vendor's playlists.xml carries a per-subtitle-slot cell that looks
like a boolean, and it was parsed as one: value 1 meant forced, anything
else meant not forced. Across every image in the corpus that uses this
format the cell takes four values, and 1 is not the forced one.
Decoding three of those discs and counting every PGS display set:
* 1 marks a FULL dialogue track that additionally contains some
forced-narrative signs. All nine cells bearing it on one disc are
full tracks of 949-1411 display sets; all seven on another are full
tracks of 1602-1651. Neither disc has a small track among them.
* 2 and 3 mark a DEDICATED forced-narrative track, in its own trailing
stream slot, duplicating a language that already holds a full track.
The two 2 slots measured are 15 and 10 display sets with every one
flagged forced; the four 3 slots are 7, 14, 23 and 59 against
1216-2655 on the tracks they duplicate.
So the old reading was wrong in both directions — it flagged full
dialogue tracks forced, which is how one language came to present as two
identical full subtitle tracks with one of them marked forced, and it
threw away the cells naming the real forced tracks.
Content could not have corrected this afterwards. Clearing a wrong
forced label needs a disc whose authoring sets forced_on_flag, and on
the measured disc carrying four genuine forced tracks not one display
set anywhere sets it — there, the vendor cell is the only evidence there
is. The classification is now explicit: only a dedicated forced slot
earns the flag, an unrecognised value never does, and the
contains-forced-signs value is dropped rather than weakened into a
forced label, since a wrong forced flag on a full dialogue track is the
user-visible defect while a missing hint costs nothing.
Four of the crate's own tests had been asserting the boolean reading;
their subject was positional alignment, so they keep it and now use a
real forced value. The other two parsers that emit a forced qualifier
from vendor metadata were audited and are structurally immune — in both,
the forced marker names a slot of its own rather than hanging off a full
track's entry, so the failure has no encoding there — and each is now
pinned by a test saying so.
A vendor label list describes ONE playlist's stream table, but
apply_labels re-numbered it from 1 inside every title. Where sibling
playlists cover the identical feature clip and enumerate different
subtitle sets, identical ordinals resolve to different PIDs, and the
same physical stream came out flagged forced in one title and not in
the other. On the title such a disc offers as its rip target that put
`forced` on an 873 MB full-dialogue English subtitle track — the
reported "I see English and English (Forced), they are identical".
The blobs are not wrong; the binding was. The list carries no playlist
id, but it carries a language per slot, and that sequence is a
fingerprint: on the corpus exactly one title's per-type language
sequence reproduces the list position for position, and content
confirms that title's binding is the correct one. So binding is now
two-tier:
* The title whose whole per-type language sequence sits on the list
(>= 2 streams, longest wins) is the ANCHOR — the table the list is
describing. Each of its slots yields a `(clip, PID) -> label` fact,
and a PID is the same elementary stream in every playlist that
plays that clip, so sibling playlists bind through the map. A slot
the anchor never showed us is not bound at all.
* Streams no anchor fact reaches still bind by the STN ordinal, but a
label whose language contradicts the stream it would land on is
dropped. Subtitle labels carry nothing but the qualifier, so an
unverifiable one is all risk and no gain: off the authoritative
path they additionally require both sides to STATE a language and
state the same one. Unlabelled beats mislabelled — the muxer's
demotable() guard can only clear a wrong `forced` on discs whose
authoring uses forced_on_flag, and half the measured discs never
set it.
Measured over 44 disc images, 9 change and every cross-title label
conflict goes away: 171 forced flags that contradicted a sibling
playlist are cleared, 61 correct ones are recovered on playlists that
had been missing them, 5 subtitle qualifiers and 1 audio purpose bound
against a contradicting language are dropped. No title gains a label it
did not have.
Known residual: on one disc the featurette playlists keep two forced
flags (down from nine) where a shifted list happens to coincide on
language. Ruling those out needs the list's provenance — which slots
are the vendor's and which were merged in by the MPLS gap-fill — and a
whole-sequence gate without it costs correct flags on discs whose
vendor slots are interleaved with gap-filled ones.
The same scrub as the previous commit, over the files it did not reach:
the variant-MKB layout notes, the 2.1 segment index observations, the
PPS-revert regressions and the playlist-twin tiebreak.
Measurements keep their numbers — "a v70 `0x2d` body = 46_100*2 + 16"
is the useful part, and the title it came from never was.
A forced verdict is an absence claim, and a sampled run sees a fraction
of a track. Measured on real discs: tracks exist that flag about a
quarter of their display sets and leave the rest unflagged, so a sample
that catches one flagged set and nothing else promotes a full dialogue
track to forced -- which players then force on screen. A sampled run now
needs two display sets; a run that read every extent end to end has no
unread gap and may still promote off one.
Fifteen references across six files named the discs a defect was first
seen on. The parser leak found earlier was not an isolated slip — the
same habit runs through the mux comments, the changelog and the AACS
content verdict, where a title name was standing in for the shape of
the problem.
Every one is replaced with the property that actually mattered: a
multi-clip title, a UHD Dolby Vision profile 7 dual-layer stream, a
disc carrying an authored-bad TS packet. The comments are more useful
for it — the reader needs to recognise the shape on a disc they have,
not the one we happened to have.
`SEG_MainFeature` stays: the parser matches on that literal, so it is
a format token rather than a title.
Measured on a real title: the same 256 MiB budget cut into sixteen
windows per extent took 41.8s against the head-first read's 6.6s, because
every jump collapsed the source's read batching into three-sector calls.
Expected observations depend on total bytes read, not on how finely they
are cut, and measured subtitle density (a display set every ~30-50 MB of
clip) makes a 32 MiB window about even money on its own.
The pixelogic walk finds the feature playlist's section by name and ends
it at the next `SEG_`/`SF_`/`FPL_` marker. Those markers are section
NAMES, and a project's trailing sections — the per-language notice,
disclaimer and dub-credit cards — carry none. On 8 of the 11
affected-format discs in the corpus the feature playlist is the last
NAMED section in the blob, so the terminator never fires and the walk
consumes the whole tail of the file as more of the feature's stream
list.
The card names are `{lang3}_{card}`, which passes `is_stream_token`, so
each one advances an STN counter, and a card whose name collides with a
catalogued component emits a label outright. Measured on the worst disc:
95 entries past the end of a 9-audio/21-PG list, five phantom audio
labels at STN 10-14 from `*_AC` notice cards (`AC` reads as the AC-3
codec), and 94 uncatalogued-component occurrences — which also took the
parse from High to Medium confidence and fired the vocabulary-gap
warning on four components that are deliberately not catalogued. A
second disc fabricated one subtitle label from a token in a following
playlist section named `FP_SingAlong`, which `FPL_` does not match.
What every section has, named or not, is a stream list that opens with
its video slots. So a `Video Stream N` entry repeating one this section
already listed is the first entry of the NEXT section, and ends this
one. Distinct video entries are kept, since a section may legitimately
list a secondary video stream; the memo of them is bounded at the BD STN
table's ceiling so disc bytes cannot grow it.
Replaying all 11 blobs through `assign_labels` before and after: the two
discs above lose exactly their phantom labels (11→6 and 5→4), the other
nine are byte-identical.
One residue is pinned rather than papered over: a card's name precedes
its own section's video slot, so a forward-only walk can still count the
FIRST card after the last real slot. It sits at the tail of a list
nothing follows in, so it can renumber nothing — at worst it costs a
parse its High confidence.
No other parser in src/labels/ walks a flat entry sequence with a
terminator set; the rest scope each stream to a structural range or read
its number off the entry itself. paramount and criterion gain immunity
pins for the boundary property specifically: a stream list cannot run
into the next element's, and a missing element boundary shortens the
list rather than extending it.
A track that flags some of its own display sets and not others proves the
authoring house makes the distinction, so it needs no sibling to
corroborate the flag being in use -- but it still has to look like a full
dialogue track before a forced label is cleared. A small track with a
couple of flagged signs is a forced track, and demoting it is the mistake
the shape test exists to prevent.
A title cut into 50-odd clips gives each extent one window's worth of
budget. At the extent's head, every clip is sampled at the same relative
position and the first clip's window lands on the opening of the feature
-- the one stretch with no subtitles in it.
Two halves of the same subtitle-labelling bug class. The numbering half
landed already: an entry the parser could not parse still occupies an STN
slot, so skipping it shifted every later label onto the wrong stream. This
is the other half — an entry the parser counts correctly but cannot
INTERPRET.
`DUB` names the forced-narrative subtitle track authored to accompany a
language's dubbed audio presentation: the signs and on-screen-text pass a
viewer still needs once the dialogue itself is dubbed. It is the same
editorial class as `*_TXT_FOR_`, spelled differently by some authoring
runs. Uncatalogued, it matched no component arm, so the token signalled
neither audio nor subtitle and the whole stream record was dropped at the
domain guard — a genuine forced track left with no forced qualifier even
after the numbering was right.
Evidence, from two independent discs in the corpus: the token appears only
inside the PG list, embedded in an otherwise contiguous run of
`{lang}[_{region}]_TXT_FOR_` siblings — one forced-narrative slot per
localized language — and takes exactly the slot where that language's
forced entry belongs. Both discs also carry that language's FULL subtitle
track as a separate, separately-labelled slot, so the DUB entry is not it.
The stream it lands on is a sparse PG track, the signature of a forced
pass rather than full dialogue. Deliberately not added to vocab::qualifier:
that maps free-form English label text, where a bare "dub" means dubbed
AUDIO. The forced-subtitle reading is specific to this token grammar.
The corpus sweep that found it also produced four components that are
deliberately NOT catalogued. They are per-language notice and disclaimer
clip names that merely collide with the `{lang3}_{component}` token shape;
each occurrence sits in a one-video-stream section next to the disclaimer
entry it names. They carry no editorial meaning, and mapping them would
attach a qualifier to a stream on the strength of a filename.
Which is also why an unmapped component now reports once per parse rather
than once per occurrence. It was a debug line nobody reads, and that is how
this gap survived to a user complaint; but a per-occurrence warn would bury
the signal under dozens of routine collisions on an ordinary disc. One
bounded, deduplicated line names the distinct components and says plainly
that any forced/SDH/commentary meaning they carry went unapplied. The
backing set is capped and truncates by chars, not bytes — the components
come from untrusted disc bytes, and a byte-offset slice can split a
multi-byte sequence and panic.
Two existing tests encoded the wrong behaviour and are corrected: the STN
numbering test expected the forced run to skip the DUB slot, and the
unclassifiable-slot test used DUB as its example of a token with no
meaning. The latter now uses a genuinely uncatalogued component.
A window's last PES stays open in the demuxer until the next PUSI, which
under sampling is in a different window or nowhere — so the last display
set of every window was discarded, worst where the sample is thinnest.
Flush the demuxer at the end of each window.
A playlist that lists the same clip twice now merges the second read into
that extent's memo (facts OR'd, display count MAXed, coverage the larger
of the two) instead of overwriting it.
The content-based forced-subtitle probe spent its whole 256 MiB budget
on the first sectors of a title. A feature's subtitles begin minutes in,
so the probe read the opening logos, hit the budget, observed no display
set at all and contributed nothing to any verdict — the vendor label was
always the only input.
The forced predicate is asymmetric: one non-forced display set disproves
forced permanently, while proving forced needs the whole track, and
genuine forced tracks are tiny where full tracks are huge. So the same
budget is now SPREAD over each extent in ~16 MiB windows placed on the
AACS unit grid, sized in proportion to the extent, ending at the extent's
end. Cost is unchanged; placement is not.
Also:
* Per-track early exit. A track that is disproven (and whose label
needs no correcting) stops asking for budget; an extent that owes
evidence only for such tracks is skipped outright, and evidence
already in the cache is never demuxed a second time.
* Content may now DEMOTE a wrong vendor forced flag, in the probe and
in the muxer, behind one shared guard: absence of forced_on_flag
only means something if some other track demonstrably uses it, and
the track must have the shape of a full dialogue track rather than
of a forced-narrative one. On a disc where no track sets the flag,
nothing is demotable.
* A sampled or budget-cut extent's evidence is memoised with the
COVERAGE behind it. It used to be filed under the extent's full key
and replayed to playlists that would have read far more of the clip,
turning a prefix into an absence claim about the whole extent.
Sweep of every parser in src/labels/ for the numbering bug fixed in
pixelogic: a blob lists one entry per STN slot, but the parser advances
its per-kind counter only for entries it can use, so every entry it
skips shifts all later labels onto the wrong stream.
Three parsers were affected; the rest key each label off a number the
blob states outright and are immune.
* paramount — `aud` / `sub` are the STN-ordered stream lists, and
`forced_sub` / `*_com1_idx` index those same cells. A cell with an
empty language was skipped without consuming its slot, so every
label behind it bound one stream early while the vendor's own
positional indices still pointed at the raw cell. `stream_number`
is now the cell's 1-based position. The forced flag is the payload
here, so the shift lands `forced` on a full-dialogue track.
* mpls_universal — its counters must agree with the stream list
`disc::bluray` builds from the same STN entries, since that list is
what `apply_labels` counts against. They disagreed twice: a
`coding_type == 0` padding entry was counted here and dropped
there, and a PG coding_type in an audio STN slot (a layout
`mpls::parse_stream_entry` has a dedicated arm for) was counted as
audio here and built as a subtitle there. Both rules now live in
one `label_type_for`.
* deluxe — a binding construction whose Language `getstatic` did not
resolve was skipped outright. It is still an STN slot; it just has
nothing to label. It now advances the counter, with the list it
belongs to taken from its CodingType argument or, failing that,
from what its binding type's resolved siblings showed.
Two paramount tests asserted the renumbering as if it were the spec
(`empty_middle_slot_does_not_inflate_stream_number`,
`audio_stream_numbering_skips_empty_slots`) and are rewritten. Immunity
pins added for ctrm, dbp and criterion so the property cannot rot.
Also scrubs two commercial disc titles from the menu-graphic filename
examples in png_filenames and vocab.
Two comments identified the disc the STN-numbering bug was found on by its
vendor project name. The reproduction does not need it: what matters is the
SHAPE of the token list — placeholder slots, region-only tokens, an
uncatalogued component — not which release happened to exhibit it. Both now
describe the shape.
The remaining `SEG_MainFeature` references stay. That is a vendor section name
the parser matches on at pixelogic.rs:88, not a disc identifier — it is the
format's vocabulary, like `FPL_` or the `eng_MLP_` stream tokens beside it, and
removing it would break the parser.
Also drops the last prohibited citation from the changelog: an mp4:// bullet
said "no ffmpeg". The website changelog page is REGENERATED from this file at
release time, so a scrub of the site alone would have been reverted by the next
release.
A pixelogic feature section lists one entry per STN slot. Only the
entries that parse were advancing the counters, so every surviving
label was renumbered 1..N and applied to the wrong stream.
Three kinds of entry were being skipped:
* `PG Stream N` placeholders — the subtitle twin of `Audio Stream N`,
which was already counted. The old comment claimed the corpus showed
subtitle tokens align without counting them; on a disc that has both
placeholders and later editorial tokens they do not.
* region-only tokens (`fra_CF_`, `spa_LS_`) — REGIONS sets `variant`
but neither `is_audio` nor `is_subtitle`, so the token is dropped.
* tokens whose distinguishing component is uncatalogued (`jpn_DUB_`).
On UHD_Crime101_WW_150728 the PG list is 18 slots: five placeholders,
four region-only tokens, and `jpn_DUB_`, with the seven `*_TXT_FOR_`
forced-narrative tokens at STN 11-16 and 18. Counting only the ten
that parse put `eng_SDH_` on STN 1 and the seven forced markers on STN
2-8 — the disc's FULL subtitle tracks. `labels::apply_labels` turns a
Forced qualifier into `SubtitleStream::forced`, which the muxer writes
as Matroska `FlagForced`, so the output offered an "English (forced)"
track carrying 1731 display sets of complete English dialogue while the
real 19-set forced track went unflagged.
Unclassifiable entries carry no stream type, so they advance the list
currently being enumerated; sections run video -> audio -> PG and
`domain` follows the last entry whose type was known.
The fvi provenance fix, the key-service outage codes, the settings BOM
data-loss fix and the per-track-kind picker rows all landed on dev without a
changelog line. A release that ships undocumented fixes is one nobody can
audit later.
The online key service returned HTTP 502 for about seven hours. Every rip in
that window ended with
key: online > no entry > NO KEY
Error: E7022 No key source has a decryption key for this disc (id: 422EB...)
which reads as "this disc is not in the key database". Operators went hunting
for a VUK that was never missing; the correct action was to wait.
E7022 is a claim about the WORLD: every source answered, and none holds a key
for this disc. A source that could not be reached made no such claim -- nothing
at all was learned. `resolve_and_apply_traced` collapsed the two anyway, with
its own comment naming the incident and pointing at the fix.
Three codes for the three different operator actions, each a variant with a
number and no English (the library ships none):
E7028 KeyServiceUnavailable unreachable / DNS / timeout / 5xx -> wait
E7029 KeyServiceUnauthorized 401 or 403 -> fix token
E7030 KeyServiceRateLimited 429 -> back off
None carries a payload: the key-service URL and its resolved address are
operator-confidential and must not ride out in a Display an operator pastes
into a bug report.
`resolve_and_apply_traced` now keeps `Err` and `Ok(empty)` apart. A source that
answered and holds nothing still records `KeyNode::NoEntry` -- that is the one
true "I looked, it is not there". A source that FAILED records an empty path,
and its reason is stamped onto `Disc::aacs_error`, the channel
`ensure_decryptable_keys` already reads for the E7017-vs-E7022 split. The gate
gained three arms alongside `AacsVidUnavailable` and raises the source's own
code.
`KeyOutcome` deliberately gains no variant. It is matched exhaustively by every
front-end's trace renderer (freemkv's `pipe::render_resolution_trace`,
autorip's `keysource::render_resolution_trace`), and this fix must not become a
breaking change across four repos to say something the error code already says
precisely. Dropping the false `NoEntry` node is enough for the trace line.
The three codes join `is_disc_level_no_key`: a service that is down, refusing
the token or throttling is down for every title, so a rip loop must stop rather
than issue N doomed requests -- and on 429, dig the hole deeper.
`FetchOutcome::errored`, documented as unreachable-in-production, now fires for
real: the negative-result cache stops memoising a transient outage.
Red before green: `key_source_failure_is_not_reported_as_a_missing_disc_key`
drives KeySource -> resolve -> aacs_error -> ensure_decryptable twice over the
same disc and asserts the verdicts differ. With the old conflation restored it
fails on the trace node, and with that assertion removed it fails
`left: 7022, right: 7028` at the gate.
The `fvi://` arm of `output()` passed the destination `.fvi` path as
`FviSink::create`'s `source_path`, so every index named itself as its
own source. `SourceInfo::default()` supplied the rest, making
`source.medium` always "file" and `source.title` always 0 — three
header members wrong, where FVI_FORMAT.md §6.2 defines `source` as
describing the input.
Beyond the wrong data, it made the output unreproducible: two machines
indexing identical bytes emitted different files purely from where
they wrote them, and a local filesystem path leaked into a shareable
file.
`output()` cannot see the source, so thread the provenance down from
the driver, which can: `mux_stream` derives a `SourceInfo` per
`MuxInput` arm and passes it through `drive_mux` to `output()`. Per the
one-method-per-action rule this is a signature change, not an
`output_with_source()` variant; the parameter is `Option<&SourceInfo>`
so a caller with no provenance declares none rather than back-filling
the destination. `SourceInfo`/`Medium` become public API.
What each arm can honestly reach:
- Session: everything — device path, the caller's title index, the
title's playlist, the scanned volume id.
- Url: the source URL, its scheme's medium, `title_index`, and the
playlist off the opened stream's scanned title.
- Iso: the image path and playlist. The title index is not in
`MuxInput::Iso` (it carries a scanned `DiscTitle`, which has no
index), so it stays 0.
- Live: medium and playlist. The reader is an opaque
`Box<dyn SectorSource>` with no path, and again no title index.
Unreachable members are left empty rather than guessed — the sink
already omits the empty ones.
Every other sample-table parser (stco, stsc, stts, ctts) had a "count lie"
test proving the declared count is bounded by what the box actually holds,
and stco/stsc had their own arithmetic pinned. parse_stss had neither - no
test in this file ever called it with real entries, only with a too-short
buffer. Added the same two: two distinct entries read from their own
offsets (catching the o = 8 + i*4 arithmetic and the per-entry bounds
check), and a declared count of 3 backed by only 2 real entries (catching
the same "trust the box, not the count" contract the other four parsers
already had).
parse_stco, parse_stsc, parse_stts, parse_ctts and parse_stss all open with
the same "if b.len() < 8" guard before reading count = be32(b, 4), but no
fixture anywhere in this file ever called any of them with a buffer shorter
than 8 bytes - every test builds a complete box. A <-to-== mutant of that
guard only rejects a buffer of EXACTLY 8 bytes and lets everything shorter
fall through to an out-of-bounds be32 read, and nothing was exercising that
fall-through to notice. One test now drives all five through every length
from 0 to 8.
Also: co64's 8-byte offsets were never actually built by any co64 fixture
in this file (only the 32-bit stco path was exercised), so its manual
byte-by-byte u64 assembly was unconstrained the same way parse_elst's was.
And sample_offsets's sidx only advanced correctly by coincidence in every
existing fixture, because none of them placed three or more samples in a
single chunk back to back - the only shape where reusing the wrong sample's
size becomes observable.
Documented the <=8 boundary as equivalent across all five parsers (and the
matching start < end in sample_offsets): the per-entry guard immediately
below always breaks on the first entry at that exact boundary, so both
branches converge on the same empty result. Confirmed by re-running each
mutation against the full suite.
asc_len == 0 and end > b.len() reject for different reasons - a useless
zero-length ASC, and a truncated one - and nothing distinguished the || from
an && that would only reject when both are true simultaneously, or the >
from a < that would reject the common case of an esds with bytes after the
ASC (more child boxes, padding) instead of only a genuine truncation.
Documented the | in read_descriptor_len's accumulator as the same
shift-then-mask equivalent already recorded in audio.rs's BitReader::read:
the shift always vacates exactly the bits the mask fills, so | and ^ can't
disagree.
mdhd_language's guard was written as "b.len() < off + 2" (reject too short)
but nothing distinguished that from "reject anything not exactly off + 2" -
a buffer one byte longer than the minimum has to keep working, and a buffer
missing the field entirely has to return None rather than read past the end.
parse_stsd's mp4a path had no test at all: the AAC codec_private extraction
depends on both codec == Aac and body.len() >= 28 being true together, and
with no mp4a fixture anywhere in this file neither half of that condition,
nor the boundary itself, was constrained. Also added the same header-length
boundary check parse_elst already had (< 8 vs <= 8 - proved equivalent this
time, since the very next guard on the empty slice catches the <= 8 case
too), and one test walking every recognised audio fourcc (ac-3/ec-3/mp4a/the
four dtsX variants) so a deleted match arm for any of them fails loudly
instead of silently dropping that track.
Nothing asserted the scan actually STOPS at cap rather than one match past
it, or that the declared box size is decoded from its own four bytes rather
than an adjacent one - every existing fixture used sizes small enough that
all but the last size byte are zero, so an index slip reading the wrong byte
would read the same zero and go unnoticed.
Stream::read has its own s.size > MAX_ALLOC_BYTES cap, a separate call site
from read_moov's over the same policy - checked they agree (both reject
strictly greater than the cap, exact cap allowed) and they do, so this is not
one of tonight's one-policy-two-copies bugs. But the boundary itself and the
one-byte-over case were unasserted, and Mp4Reader::write returning an error
(mp4:// is read-only) had no test either.
Building Mp4Reader directly in the test (its fields are private but visible
within this module) over the existing FakeBigReader avoids a real 256 MiB
backing file for the boundary case.
Nothing asserted that a hdlr other than vide/soun gets dropped rather than
folded into the audio branch, that the per-track PID formulas
(0x1011/0x1100 + track_idx) use the right operator and the right operand,
that a sample-less track still advances track_idx for the next one, or that
the cross-track sample_budget is actually decremented (as opposed to grown or
divided) by each track's real count. All four were reachable with a single
track_idx == 0, which made every existing fixture blind to +/-/* confusion on
these sites - track_idx never moved past 0 in any of them.
Also let audio_trak_missing omit stsz, needed to build a sample-less track for
the track_idx test.
parse_elst's existing tests used segment_duration/media_time values that were
almost all zero or 0xFF bytes, so an index slip in the version-1 byte
extraction (reading a neighbouring byte, or one outside the entry entirely)
could return the same value by coincidence and the test wouldn't notice. Added
a fixture with every byte distinct and nonzero so any wrong offset is caught.
elst_offset_ticks's `empty_movie_ticks > 0` guard on the Some(mts) arm looked
like a pure optimisation, but shifting its boundary lets an
empty_movie_ticks == 0 call fall into the division instead of skipping it -
and a zero movie timescale (unreachable through from_reader, which filters it,
but not through this function's own contract) makes that division panic.
Pinned the boundary directly so the function stays safe on its own terms.
Documented nine further mutants as equivalent rather than chasing them:
media_edits/odd_rate and the None-arm's empty_movie_ticks check only gate a
tracing::warn!, never the returned offset, and parse_elst's b.len() < 8 vs
<= 8 boundary computes the same empty Vec either way once available = 0 is
worked through. Confirmed by re-running each mutation against the full test
suite.
The remaining triage items after tonight's HIGH fixes: 1,290 lines, almost
all tests. Covers disc/mod.rs's DVD scan path (with real minimal VMG/VTS IFO
fixtures rather than mocks), drive/mod.rs, labels/class_reader.rs and
labels/mod.rs — the two biggest untriaged survivor clusters in the crate —
plus hevc.rs and ps.rs.
One production change, and it is an extraction rather than a behaviour
change: MacScsiTransport::open mapped the shim's negative failure sentinels
to typed errors inline, where nothing could reach it without a real IOKit
FFI call. It is now map_shim_open_error, so the mapping can be pinned. It
matters because collapsing -5 into the DeviceNotFound catch-all turns
"another process holds the drive" into "no such drive", and an operator
chasing the wrong problem is worse than a blunt error.
Gate green on the pinned toolchain including the secrets scanner.
read_moov's forward-progress guard (box_size < header_len, OR'd with the
EOF check) and the MAX_ALLOC_BYTES cap were only exercised on inputs well
away from their boundaries, so a mutation testing pass found the exact
edges unasserted: a size-8 (header-only) moov, a box that overruns the
file by exactly the amount the OR/AND distinction can see, and a payload
of precisely MAX_ALLOC_BYTES. Added a shared FakeBigReader (lifted out of
an existing test's local struct so a new test can reuse it) to exercise
the MAX_ALLOC_BYTES boundary without a multi-hundred-MiB backing file.
Every mutation-testing survivor in this file is a | with ^ flip inside a
bitstream packer: BitReader::read's accumulate step, the push closures in
dac3_box/dec3_box/ddts_box, and the multi-field extractions in parse_eac3
and parse_dts. All nine are the same shape: shift an accumulator left by
exactly the width of the next field, then OR it in, so the two operands
never share a set bit and | and ^ agree on every input. Confirmed by
running cargo-mutants against just these nine mutations after the existing
test suite (which already exercises each function's field values) - all
nine still survive, as expected for a genuinely equivalent mutant. Recorded
the reasoning once at BitReader::read so nobody spends time chasing it
site by site.
Three pack_language mutants and two detect_rate boundary mutants survive
mutation testing with no test able to close them, and it's not for lack
of trying: they're equivalent by construction. Recording the proofs next
to the code so nobody re-chases them:
- (b[0] - 0x60) as u16, shifted << 10 then truncated to u16, is congruent
mod 65536 to (b[0] + 0x60) as u16 shifted the same way, because
0x60 * 2 * 1024 is an exact multiple of 65536. The same swap on the
second letter (shifted only << 5) is NOT equivalent, which is why only
the first letter's mutant survives.
- The two | with ^ mutations that OR the three packed fields together
are equivalent because the fields (a lowercase letter minus 0x60, so
1..=26) always fit in 5 bits and never share a set bit once shifted
into their 0/5/10 positions.
- detect_rate's tolerance and tie-break comparisons only diverge from
their <= mutants on an exact 0.5 fps distance or an exact tie, and a
brute-force search over every achievable integer-nanosecond median
found no case that lands on either boundary bit-exactly.
MP4 track timing was numerically unasserted. Every operator in the
PTS-to-ticks, duration, tkhd_dur and ctts chain could be flipped and the
whole suite stayed green, because no test decoded an output file and checked
a concrete number — the existing tests assert box presence and gross
container shape only. That is the crate's worst failure mode: a title muxes
"successfully" with silently wrong A/V sync or total duration, and nothing
above can tell.
The new tests build tracks with known PTS deltas and compare the emitted
stts, ctts and tkhd.duration against computed values.
Also lifted the faststart slack rule out of the match guard into
faststart_fits(). A leftover hole of 1-7 bytes cannot be expressed as any
ISO-BMFF box, since a box header is 8 bytes, so finish() must fall back to
moov-at-end rather than write a free box that lies about its own size. The
condition now has a name and a test instead of being an unexplained
`g == 0 || g >= 8` inside a pattern guard.
Drive::read and read_fua split any request larger than the transport's
transfer limit into chunks and slice the caller's buffer by count * 2048.
The up-front length check is the only thing between an undersized buffer and
a "range end index out of range" panic out of a public API, and the comment
above it records that this was once a live panic.
Every existing read test stays on the single-chunk path, where an undersized
buffer is already tolerated and returns Err(DiscRead), so the guard itself
had no coverage at all and a mutation run flipped its arithmetic freely.
The test drives a mock transport with a small transfer limit and asserts the
two paths agree: an undersized buffer is an error either way, and behaviour
on a caller mistake does not depend on the drive's transfer limit. Confirmed
by hand that both the reported * -> + mutation and a < -> > flip now fail.
The escape-stripper comments named a third-party decoder as the authority
for the cumulative-zero rule. This repo is public and does not cite other
implementations; the rule is specified in ITU-T H.264 §7.3.1, which is the
citation that belongs here anyway.
No behaviour change — comments only. The scan-secrets gate caught it.
A mutation run over mux/mkv.rs and mux/mkvstream.rs left 148 survivors.
Reading them turned up no wrong code, but a lot of code whose output
nothing ever looked at. Most of that is on the read side: parse_track
had a dedicated arm for Language, TrackName, FlagForced, Video and
Channels and not one of them was checked, so a re-mux could have lost
the audio language, the subtitle forced flag, every track label, the
resolution and the channel layout with the suite still green. Ten of
the eleven CodecID comparisons were unasserted too — only HEVC was
pinned — so any of them could have been mis-wired and the stream would
have gone to the wrong parser. The round-trip test now writes a real
three-track title through the muxer and reads it back through the
reader, and a separate test walks every registered CodecID.
The BPS statistics tag was the worst of the write side. Its test
asserted `file_bytes.contains("800")`, which a wrong bitrate passes
trivially — 80000 contains "800". Both the tag and the back-patched
Segment duration are now decoded and compared to a computed number, on
a title that declares no duration so the whole max_block_ticks →
seconds → bits chain is exercised. Cue points get the same treatment:
their CueTrack and CueClusterPosition were never read back, on either
the keyframe path or the i16-forced-split path, which are two hand-
written copies of the same three fields.
The rest closes arithmetic that only a bad disc reaches: a zero frame
rate or zero display-aspect denominator (both divisions), a
TimestampScale that does not fit an i64, a cluster timestamp of exactly
i64::MAX, a TrackNumber of 65537 that truncates onto the valid track 1,
and the shortest legal Block at both VINT widths. Two tests separate a
clean end of stream from a device failure: swallowing the second one
truncates the output at a bad sector and reports the rip complete.
Also pinned: only the first video and first audio track may be default
(the de-duplication lives in MkvStream::create and had no test at all),
the activation trigger is the first VIDEO track rather than track 0,
the measured field order reaches the file rather than just the helper
that computes it, and a Blu-ray 3D base/dependent pair builds the merge
instead of shipping two unrelated H.264 tracks.
96 of the 148 mutants verified killed by hand. Of the remainder, most
are equivalent — disjoint-bit `|` that `^` cannot change, delete-arm
mutants whose fallback is the same constant, guards on tracing calls —
and the write_frame branch at 1452 is unreachable: a cluster is always
open by the time it can be entered.
The mux/codec parsers (startcode, h264, hevc, dts) had 300 surviving
mutants between them, and it turned out to be for the reason you'd
fear: the exp-Golomb readers and the AU-boundary bitstream scanners had
essentially no direct unit coverage, only indirect exercise through
full-frame parse() calls that never touched the actual edge cases.
Direct fixes to test gaps:
- The shared BitReader's read_ue truncation guard (`leading_zeros >
31`) and skip_start_code's 4-byte-vs-3-byte boundary check had no
test at their exact boundary. Added tests that hit the boundary
precisely; a `>=`/`==`/`<=` typo either rejects a legal 31-leading-
zero code or reads one byte past the buffer.
- H.264's private SpsReader duplicates the same read_bits/read_ue
shapes with no tests of its own at all (only reached through
multi-field SPS parsing, several fields deep). Added direct tests.
- HEVC's per-AU trailing-zero strip after the last NAL (no start code
following) walks `end` down to trim padding; a wrong-direction typo
there walks off the end of the buffer instead of terminating -
exactly the "loop must make positive progress on malformed input"
class. Added a test with a zero-padded trailing NAL.
- HEVC's SEI match guards (`sei_mastering.is_none()` /
`sei_content_light.is_none()`) implement "first HDR10 value in the
title wins" - untested, and a naive test using both-messages-per-AU
can't even exercise the guards because the whole-scan early return
above them already handles that case. Split into single-message-
per-AU tests that actually reach the arms.
- parse_mastering_display/parse_content_light_level's length guards
were `< N` with no boundary test; one-byte-short input now confirmed
to return None instead of indexing out of bounds.
- DTS's drain_front collapses duplicate offset-0 PTS markers after
rebasing; untested, and the visible effect (front_pts()) can't tell
a working collapse from a broken one since it already returns the
right marker either way - the actual defect is unbounded growth of
pts_marks over a long recording, so the new test asserts the bound
directly across repeated drains.
- DTS's dts_core_samples/dts_core_sample_rate header-length guard and
next_core_boundary's syncword-length guard got exact-boundary tests
the same way; also caught a nblks `<<`/`>>` direction bug candidate
in the mutant (confirmed the real code is correct, just untested).
Real bug found and fixed, not just a test gap:
H.264's parse_sps_high_profile_ext re-implemented emulation-prevention
byte stripping inline (a window scan: match `00 00 03` at position i,
advance 3, else advance 1) instead of calling the existing
unescape_ebsp_prefix used by slice-header parsing. On a run of 3+ real
zero bytes ahead of an 0x03 - non-conformant, but this is disc bytes,
not a spec-clean encoder - the two disagreed: unescape_ebsp_prefix's
cumulative zero counter (matching the H.264 reference decode process
and libavcodec's RBSP extractor) treats it as an escape and drops the
0x03; the window scan treats it as real payload and keeps it,
corrupting the SPS bits read after it. Extracted the shared rule into
`unescape_ebsp` (parameterized on output length so both the 16-byte
slice-header prefix and the unbounded SPS case can share it) and
pointed both call sites at the one implementation. Added a regression
test pinning the shared function's behaviour on the input that used to
separate them.
All new tests hand-verified against the actual mutation (operator
flipped or guard replaced by hand, confirmed red, then restored) per
the mutation-testing brief, not just written and trusted.
A 12,330-mutant run left 159 survivors across these three files, all from
missing assertions rather than wrong code — every gap here is a test, no
production logic changed.
Two shapes accounted for most of them:
- Buffer-cap constants (MAX_PES_BUFFER_TOTAL, MAX_PS_BUFFER,
MAX_BD_PES_PAYLOAD, PES_BUFFER_INIT_CAP) were only ever read by tests
through their own symbol, so a mutated `*`/`-` in the constant's
definition changes what the symbol itself evaluates to and every
self-referential assertion still passes. Pinned each against a literal
computed independently in the test.
- Several `>`/`==` boundary checks on framing lengths (MPEG-2 pack header,
system header, BD-TS adaptation field) were only ever exercised with
slack in the buffer, never at the exact byte the check exists for.
Added exact-fit cases for the pack header, system header, and
psi_payload_base's AF-consumes-everything boundary.
Real, higher-value gaps closed along the way:
- ts.rs's per-PID discontinuity_flag and the NULL-TS concealment marker
both require adaptation_field_length > 0 before trusting the AF flags
byte; neither branch had a test proving af_len == 0 (no flags byte at
all, ordinary payload underneath) is left alone.
- header_remaining (PES header spillover across TS packets) only had
single-continuation-packet coverage, which can't distinguish `-=` from
`+=`/`*=` because the corrupted value never gets read again. Added a
case spanning two continuations.
- ps.rs's parse_stream_id_extension (used for HD-DVD 0xFD routing) walks
nine optional PES-header/extension fields with a `pos +=` each; only
the PTS/DTS pair had ever been exercised. One test now arms every
field and checks the walk lands on the right byte.
- find_ps_boundary's `sc + 3 >= len` guard had no test at sc == 0 with a
bare 3-byte start code, the case a `+` -> `-` mutation turns into a
debug-mode subtract-overflow panic on ordinary tail-of-buffer input.
- tsmux.rs: an oversized video access unit must go out as a single
unbounded-length PES; the `is_video || small-enough` guard that
enforces this had no test with a video frame actually over the
bounded-PES threshold, so a `||` -> `&&` mutant survived (it would
silently split a keyframe across several look-alike-independent PES
units). Also pinned the PES-length and PTS big-endian encodes at
values above 255 / with bit 29+ set, where a `>>`/`<<` swap first
becomes observable.
Every test above was verified by hand: applied the exact mutation,
confirmed the test fails (or the specific panic fires), then reverted.
Left unclosed, all confirmed equivalent by hand-tracing rather than
just left alone:
- Every `<<8 | byte` PID/length bit-combine (ts.rs pid/PAT/PMT parsing,
ps.rs dvd_audio_pid/hddvd_extended_pid/parse_pts): the two halves
never share a bit, so `|` and `^` produce identical output for every
input - no test can tell them apart.
- ts.rs's `af_len > 183` check in process_packet: fully subsumed by the
`payload_start >= TS_PACKET_BYTES` check three lines later for every
af_len that could trip it.
- ts.rs's out-of-range `pid_index` sentinel (-1 vs 1): unreachable, since
a TS PID is masked to 13 bits (max 8191) and the table is always sized
to at least 8192.
- A cluster of "push an empty slice on an exact boundary" mutants in
tsmux.rs's write_pes_chain (offset < hdr_len, af_bytes stuffing
guards): the guarded write becomes a length-0 write_all, a no-op
either way.
Not reached this pass, for lack of a clean seam within the time
available - ps.rs's extract_packets bounded-PES-length exact-fit
checks (lines 278/282/303, the `sc+6>len` / `sc+6+pes_len>len` /
force-flush cap arithmetic). The first two need a scenario where
"proceed vs. wait one more byte" is observable in the packet list, and
the third only shows up at a start-code offset (sc) that survives to
the moment the cap check runs - in this code path sc is always 0 once
an unbounded PES buffer starts accumulating, since nothing before it
ever drains. Didn't find a construction in the time available; flagged
rather than papered over with a self-referential assert.
DriveId::from_drive issues three data-in commands. The two GET CONFIGURATION
calls both clamp on bytes_transferred, with a comment noting it is
device-reported and untrusted. INQUIRY, three lines above them, discarded it
and decoded bytes 8..43 unconditionally.
The buffer is pre-zeroed, so a drive answering GOOD status with a short or
empty data phase — a USB-SATA bridge mid-wedge does exactly this — produced
blank vendor, product and revision strings and a byte 0 of 0x00. Every
platform enumerator gates on raw_inquiry[0] & 0x1F == the optical peripheral
type, and 0x00 is DIRECT ACCESS, so the drive silently disappeared from the
device list instead of reporting that its identity probe had failed. The
operator sees no drive at all rather than an error.
Anything shorter than the SPC-4 standard 36-byte header is now
E9058 DriveInquiryShort, and the buffer is truncated to what actually
arrived so nothing decodes past it. Exactly 36 bytes is still accepted: the
vendor-specific tail is optional.
This is the same defect as the READ CAPACITY short-transfer bug fixed
earlier today, in the same crate, found the same way.
mod.rs had two independent implementations of "pick the winning label
parser". select_result owns the rule — highest confidence wins, and on a tie
the earlier entry in PARSERS wins — and carries a regression test for a past
bug where analyze() picked the LAST equal-confidence parser instead of the
first. extract(), the path that actually ships, re-derived the same rule with
its own inline scan and had no test of its own.
The tie-break is load-bearing rather than incidental: array order encodes a
trust ordering, with the hand-vetted parsers registered ahead of the ones
that detect on any BD-J disc. A later parser winning a tie means the disc
gets labels from a less trusted source, silently.
extract now collects its candidates and calls select_result. Confirmed by
flipping the tie-break to prefer the later entry: the shared test fails,
where before the fix that mutation was invisible to the whole suite.
This is the seventh instance tonight of one policy implemented twice with
only one copy hardened, and the second in this file after the chapter
mark_type filter.
An FMTS forensic segment interleaves two variants at the unit level: the
disc carries both halves and we hold the key for exactly one parity.
Decrypting the alternate half with our key produces garbage; leaving it as
ciphertext is correct, because the muxer drops untouched ciphertext cleanly.
The decision lived inline in apply_aacs_map's per-unit closure, where no test
could reach it. A mutation run flipped the subtraction to an addition and the
parity comparison, and every test still passed — so the gate that decides
which half of a forensic segment we decrypt was entirely unasserted.
It is now unit_is_our_phase(). The index arithmetic also saturates and
clamps: both inputs come from the key map, so a unit below its own range
start or a zero unit size means a malformed map, and neither may panic on
debug overflow or a divide by zero inside a library a long-running service
depends on.
Four of the five surviving mutants now fail. The fifth, dividing by the unit
size versus multiplying, is equivalent rather than uncovered: aligned offsets
are exact multiples of the unit size, so the two differ only by a factor of
unit_sectors squared, and unit_sectors is the constant 3 — odd, so parity is
preserved on every reachable input. The proof is recorded in the test instead
of a test that pretends to cover it.
fill_extents skipped an exhausted or zero-sector extent by calling itself,
which costs a stack frame per skipped extent. Nothing filters
sector_count == 0 out of a UDF or MPLS extent list, so a malformed disc
declaring a long run of empty extents recursed once per extent before
reading a single sector. Rust does not guarantee tail-call elimination, so
that overflows the stack — which aborts the process rather than returning
an io::Error, taking a long-running service down with it.
The skip is now a loop. The regression test runs on a 256 KiB stack, where
the recursive version dies and the loop finishes immediately.
Two cases of the same shape: one policy implemented twice, with only one
copy hardened.
Disc::read_capacity decoded buf[0..4] from READ CAPACITY (10) without
checking that the transport actually delivered four bytes, even though its
comment claims to mirror decode_read_capacity — which has exactly that
check, and documents why. A drive answering GOOD with an empty data phase
leaves the buffer zeroed, so last_lba decodes to 0 and the probe reports a
one-sector disc instead of an error. It now calls the shared decoder rather
than re-deriving it.
collect_chapter_summary filtered chapters on mark_type <= 1, counting the
reserved type 0. PlaylistMark's own doc says filters must test == 1, and
disc/bluray.rs did; the labels path did not, inflating the public
chapter_count and letting a playlist whose only marks are reserved pass the
chapter_count == 0 skip. Both sites now share PlaylistMark::is_chapter_mark
so the copies cannot drift again.
Both fixes were confirmed red before green.
Every job above proves libfreemkv builds; none proved anything built on
it does. That gap bit today one level up — an engine signature change
broke autorip and went unnoticed, because consumer CI only fires on a
push to that consumer, and nobody pushed one.
libfreemkv sits underneath all five dependents, so a break here costs
more than a break anywhere else in the project. It is now the place the
question gets asked, since it is the place the change happened.
cargo check --all-targets only: each dependent has its own suite for its
own behaviour. This answers the narrower question that went unanswered.
Every other repo's CI now runs clippy with --all-targets. libfreemkv,
the crate the other seven build against and the one held up as the
reference workflow, was the last one still linting the library only — so
its ~3,000 tests, by far the largest body of test code in the project,
had never been linted at all. Turning the flag on surfaced 74 findings.
Most were mechanical and applied with clippy --fix. The rest, by hand:
- Four discarded Results in decrypt.rs. css::descramble_region returns a
Result and four CSS tests threw it away, so a descramble that FAILED
would have surfaced as a confusing buffer-comparison mismatch instead
of the actual error. They expect() now.
- A dead `kp` field on the PlantedWalk fixture. The test deliberately
asserts Kp as the explicit AES-G3(dk, 1) relation from [C] §3.2.4
rather than against a stored value — its doc comment says so — which
makes the field not just unused but a trap: the obvious "fix" of
asserting against it would quietly weaken the test to comparing the
fixture with itself. Removed.
- Two hand-rolled ICB counters in the HD-DVD fixtures, a needless mut,
three vec!s that only ever needed arrays, a filter_map whose every arm
was Some, and a Vec::new()+push chain.
- Doc list indentation in mkv.rs and mp4/read.rs, which was mis-rendering
in the generated docs.
- A five-[u8; 16]-tuple return type named FourLevelParts.
Three lints are allowed at the specific sites, with reasons, because
they are wrong for this domain: the underscores in the bitstream-header
literals mark BITFIELD boundaries, not digit groups, so regrouping them
uniformly would satisfy the lint by destroying the only thing they
encode; and in three table-validation loops the loop variable is the
domain value under test (a DTS SFREQ code, an AMODE value, a palette
entry number), which is what the assertion messages name.
io_error_code was private, so the predicates built on it (is_halt,
is_skippable_title_stub, is_disc_level_no_key) were the only way to ask
anything about an io::Error's origin. A consumer that needs the code
itself — to report WHY a title failed rather than to branch on one of
three known cases — had no route to it: mux_stream returns an io::Error,
the typed Error is gone by then, and only the E<code> string prefix
survives.
That left every front-end to re-implement the prefix parse by hand,
which is precisely the string-matching 1.5.x spent its time removing.
One parser, exported.
No behaviour change: the function is unchanged and the three predicates
still call it.
CI checked out one repo, but libfreemkv path-deps ../freemkv-unlock on
the branch tip — release.sh swaps that to a git tag only inside the
TAGGED commit, then restores the path dep on the branch. So the branch
tip has never been buildable in CI by construction, and every green run
we have ever had was a tag build. Windows and Linux were first compiled
at release time, which is the worst moment to discover a build error.
Both repos now check out into subdirectories (actions/checkout refuses a
`path:` outside $GITHUB_WORKSPACE, and ../freemkv-unlock is outside), so
the path dep resolves exactly as it does on a developer's machine. Every
cargo step runs with working-directory: libfreemkv, and rust-cache is
pointed at the same workspace.
The sibling is taken from `dev`. On main/tag builds the Cargo.toml in
that commit carries the git-tag dep instead, so the extra checkout is
simply unused there.
This is what makes the new branch policy's "dev must be green"
achievable rather than aspirational.
Work now lands on dev and CI must be green there; main only moves at
release time, to the tagged commit, so a push to main is the release
validation run rather than day-to-day feedback.
leak-guard already runs on every branch (on: [push, pull_request]) and
release.yml stays tag-triggered, so neither needed a change.
I broke DVD ripping earlier today and the real-media acceptance gate
caught it on its first full run. Greenland.iso failed with E7013
"Decryption failed"; reverting only this change made it rip clean in 8
seconds. That is a regression I introduced, not a pre-existing defect.
WHAT I GOT WRONG.
Round 9's crypto lens reported that descramble_region "descrambles with
a key it just proved wrong" when the crib check rejects the cached key
and the re-crack also fails. I agreed, and made it Error::DecryptFailed
to match the AACS path, on the reasoning that CSS has no external key
source so a failed crack on a readable sector should never happen.
The premise was wrong. `attack_crib` is a HEURISTIC, not a proof: it
finds a periodic run in the unscrambled header and predicts the run
continues past 0x80. When that prediction does not hold, the crib
reports a mismatch even for a CORRECT key — and the re-crack then fails
BECAUSE the crib was never valid. So crib mismatch plus crack failure is
the signature of a crib false positive, not of a stale key. The cached
key is not proven wrong; it remains the best available evidence, and on
a real DVD it is very probably right. Real discs hit this constantly.
The deeper error was treating "no key" as one thing across schemes. An
AACS unit key either opens a unit or it does not — the Verify-Media-Key
relation decides it, and a wrong key is provable. A CSS title key is
recovered from the data itself by an attack whose success varies sector
by sector, so "the crack failed here" says something about THIS SECTOR's
plaintext, not about the key. Unifying the policy was right for the
schemes that can prove a key wrong. CSS cannot, and I folded it in
anyway.
decrypt_span keeps its shape and the cross-scheme test keeps its two
AACS arms, with CSS now explicitly excluded and the reason stated.
Three tests asserted the wrong behaviour and are corrected, including
one I rewrote earlier today to pin exactly this. Every one of them
passed the whole time the code was broken — because none of them had
ever seen a real disc.
The lesson is the one I kept stating and then did not act on: 3,013 unit
tests, ~400 mutants killed and nine audit rounds did not catch this, and
one acceptance run did. Synthetic media cannot reproduce what a real
disc does.
All 59 measured survivors in mp4/audio.rs: 50 killed, 9 proven
equivalent, none left unaddressed. No production change — every
extraction reads correct against ETSI TS 102 366 (5.3.2, 5.4.2, Annex
E.1.3, F.4, F.6.1) and TS 102 114 5.3.1.
It was a fixture gap, not a code defect, and a specific one: the
existing fixtures gave several fields the SAME value (fscod=0, bsmod=0,
lfeon=1, acmod=7) and asserted only derived channel counts. Nothing
asserted the emitted dac3/dec3/ddts payload BYTES at all, so the
packer's shifts and masks were entirely unconstrained. A wrong mask
there does not crash — it writes a box declaring the wrong channel
configuration, and a player believes it.
Several kills needed fixtures designed to discriminate rather than
merely exercise:
the reduced-rate branch needed fscod == 3, which no test in the file
reached — and `== -> !=` survived on an fscod=0 fixture only because
the reduced table happens to return 48000 there too
the DTS LFF mask needed a value where XOR and AND differ in MEANING:
on the 5.1 fixtures LFF flips 1 -> 2 and BOTH codes mean "LFE present"
the acmod mix-level skips needed three different acmods, because `^`
is a no-op unless acmod == 4 exactly
the ddts bitrate needed 44100/512, which does not divide evenly — at
48000 the rounding is invisible and `/ -> %` on den/2 survives
The 9 equivalents are one pattern: OR-ing a shifted high part with a
masked low part on disjoint bit lanes, where the mask is on the same
line as the OR. Since cargo-mutants applies one mutation at a time, no
single mutant can break both. Each was applied and observed green.
FILED, not fixed: reserved sample-rate codes are silently guessed as
48 kHz (audio.rs:133, :20/:153) while a reserved DTS AMODE is refused
with a comment explaining why. Same silent-wrong-metadata class,
opposite answer. Refusing would make such a stream unmuxable, which is a
product call.
There were TWO top-level decrypt paths: decrypt_sectors_impl for CSS and
clear media, whose AACS arm was a bare `return Err` stub, and a wholly
separate decrypt_sectors_mapped for AACS. Each scheme therefore decided
its own answer to "there is no key for these bytes", and nothing held
them to the same one.
They drifted, in opposite directions, within a single release:
css::descramble_region descrambled with a key the sector's own crib had
just proven stale — garbage behind an intact clear header, reported Ok.
the mapped path returned early for any LBA outside every range, before
ever asking whether those bytes were ciphertext, so an unkeyable
encrypted unit passed through and extract counted it as good.
Both were fixed individually earlier today. This removes the shape that
allowed them.
decrypt_span is now the single orchestrator: it owns the loop, the
refusal, and the loss count, and each scheme supplies only what
genuinely differs. apply_aacs_map is a scheme step that reports what it
could not open; it no longer decides what that means. The public
wrappers (decrypt_sectors, _in_content, _mapped) are unchanged in
signature and all funnel through it.
Adding a scheme now means adding an arm here, which means answering the
refusal question. That is the point.
The new test asserts ONE verdict across all three schemes — AACS with no
map, AACS with an encrypted unit outside every range, and CSS whose
re-crack failed — plus that clear media is NOT a refusal. A per-scheme
test cannot hold this: each would keep passing while the two disagreed.
Flipping the AACS arm back to pass-through reds it.
Also removes the last of the tracing-capture scaffolding. Serialising
those captures crate-wide did not fix the 1-in-10 flake, and asserting
the predicates directly made the helper, both capture subscribers and
an unrelated dead OrderSink unused. Deleted rather than left behind.
drive() and drive_mut() had ZERO callers — not in libfreemkv, freemkv,
autorip, bdemu, keysources or kdb. Deleted rather than converted: dead
public API that panics is not an API worth preserving the shape of.
into_drive() had two callers and now returns Result. The empty-slot
state is reachable through ordinary public use — stage_drive_as_reader
moves the drive into the reader slot, and calling into_drive twice moves
it out — so the panic was not guarding a caller error. identify() was
converted for exactly this reason in this same release; the fix went to
one of four public sinks and the other three were left.
I deferred this on the assumption the blast radius was large. It was
three call sites. Checking beats assuming.
Also fixes a REAL FLAKE in the gate, which is worth more than the above.
resolve_vid_only_bus_key_gate_reports_true_has_volume_id... failed about
one full-suite run in ten while passing every time in isolation. It
installed a capturing tracing subscriber to read back the has_volume_id
field of a warn.
That cannot be made reliable: dispatcher::set_default is THREAD-LOCAL
while tracing's callsite-interest cache is GLOBAL. The original author
knew, and called rebuild_interest_cache() — necessary but not
sufficient. I first serialised every capture in the crate behind one
lock (harness::with_captured_tracing, which also removed the same
hand-rolled dance from three other sites). Still 1-in-10, because the
cache can be re-evaluated against the process-default dispatch rather
than the thread-local one.
So the predicate is now a named function, handshake_has_volume_id, and
the test asserts the VALUE. A boolean does not need a subscriber to
check. The gate's hard-error behaviour keeps its own test.
Measured: 14 consecutive full-suite runs, 2994 passed, 0 failed.
A flaky gate is worse than a missing one — every green after it means
less, and this one had been eroding trust in the whole suite.
a_signalled_discontinuity_survives_a_backstop_discard asserted that a
SOURCE-signalled discontinuity on discarded bytes still reaches the AU
that follows. It did not test that. Deleting the disc_marks push, or the
mark-retirement loop inside discard_gap_before, left it passing.
The mechanism: the `discontinuity = true` rode the FIRST over-cap push,
which still has the next AU's delimiter at buf[0] — so it force-flushes
as an over-long AU rather than discarding, and THAT AU consumes the
mark. The assertion's `.find(|x| x.data.contains(&0x22))` then filters
it out, and the flag it reads comes entirely from `pending_gap`, set by
the second push's backstop. Behaviourally identical to the test 40 lines
above it, under a name promising something else.
I wrote it this morning, in the same commit that fixed a different test
for having a fixture that never reached the code it named, while
cataloguing that exact shape. Third instance today of writing the bug I
was hunting.
The two mechanisms cannot be isolated in one fixture — a fragment that
trips the backstop sets pending_gap regardless — so they now get one
test each. The replacement drives disc_marks end to end with no backstop
involved: a flagged fragment that carries a complete AU and is emitted,
not discarded. Nothing else pinned that path. Removing the disc_marks
push reds it.
Found by the round-9 opus escalation over test quality, dispatched
because the sonnet pass over the same 17,000 lines of new test code
returned zero findings.
The three bounded-fsync failures returned bare io::ErrorKind values —
TimedOut, Interrupted, Other/EIO. `is_halt()` matches on
`io_error_code(e) == Some(E_HALTED)`, i.e. the "E<code>" prefix that
`From<Error> for io::Error` mints, and that is documented as the ONLY
recognised shape. A bare ErrorKind carries no prefix.
So cancelling a rip while sync_all / finish was inside the bounded fsync
made `is_halt()` return false, and the CLI reported a clean user cancel
as a hard I/O failure at the end of an otherwise complete mux.
All three were also mutually unclassifiable, which is the same
information-loss the numeric-code scheme exists to prevent: a caller
could not tell "cancelled" from "NFS wedged" from "worker died", and
should not retry the third the way it retries the first. And their
Display text is std English — "timed out", "operation interrupted" —
reaching a user from library code, which this crate does not do.
Now Error::Halted, Error::SyncTimeout (E_SYNC_TIMEOUT 9056) and
Error::SyncWorkerLost (E_SYNC_WORKER_LOST 9057), on both platforms.
E_HALTED also now maps to ErrorKind::Interrupted rather than falling into
the 6000..=6999 InvalidData bucket. A stop is an interruption, not
invalid data. Nothing branched on the old kind — every consumer uses
is_halt() — so this is safe as well as more accurate.
Two things worth recording. I first placed the E_HALTED arm AFTER the
6000..=6999 range arm and wrote a comment claiming it preceded it; match
arms are ordered, so the range won and the comment was simply false. The
test caught it. And the macOS test asserted only that each arm was
non-Ok with a particular ErrorKind — it passed throughout the period the
three were indistinguishable. It now asserts they can be TOLD APART,
which is the property that actually matters.
Found by the round-9 opus escalation over the API contract.
Regression from the round-8 change that started RETAINING ECMA-167
4/14.14.1.1 type-1 (allocated, not recorded) descriptors. Retaining them
is correct — dropping one slides every later extent's data down by the
hole's length, corrupting the file silently. But read_icb_extent still
took extents.first(), so the value it returns can now be a hole.
A type-1 extent's lba is where SPACE is allocated, not where bytes live.
IcbExtent's own doc says exactly that. file_start_lba hands the value
out as "the absolute starting LBA of a file's first data extent", and
ifo.rs uses it as the base for every VTS VOB extent:
file_start_lba(IFO) + vtstt_vobs + cell.first_sector
So a DVD whose IFO's first descriptor is type-1 reads its entire video
title set from the wrong place on disc. No error anywhere — the reads
succeed, they just land on unrelated sectors. Verified: the test reports
2900 instead of 2040 with the old code.
Same shape as file_extents/extents_abs_at dropping the recorded flag,
and the same root cause: one change taught read_icb_extents about a new
extent type and did not visit the accessors that consume its output.
Three of them; two are still open (task filed).
Found by the round-9 opus escalation over the API contract, dispatched
because the sonnet pass over the same scope returned zero findings.
Seven of its eight items were absences rather than wrong lines — the
class a wrong-line scan structurally cannot see.
The Main Volume Descriptor Sequence was selected from the anchor's
declared extent whenever that extent's SHAPE was usable — length >= 16
sectors (ECMA-167 3/10.2.1), non-zero location, no address wrap — and
the customary location was tried only when the shape failed.
Shape is a property of the field, not of what is there. An anchor can
pass all three checks and point at nothing: a mastering tool that wrote
the reserve location, a stale anchor on a rewritten volume, or
deliberate corruption on an untrusted disc. The sweep then finds no
Partition Descriptor, partition_start stays 0, and the volume is
rejected as UdfNotFilesystem — while the real sequence sits unread at
the location the old fixed sweep would have found.
So the branch a DAMAGED disc actually takes had no recovery path, which
is backwards: that is the branch recovery exists for.
Now both are candidates and the fallback is retried on OUTCOME. This is
the same principle the Metadata File Location chain in this same
function already uses — treat the recorded value as a candidate, fall
back when it does not pan out — applied one level up. The sibling was
added in 8e70f2a and the asymmetry has been sitting there since.
A read fault inside a sweep no longer aborts before the next candidate
is tried, but it does not vanish either: it is held and returned if no
candidate yields a partition. "Could not read" stays distinct from "read
fine and the bytes say no", because mux::resolve caches the second for
the whole disc.
Not addressed, and pre-existing: the RESERVE sequence at avdp[24..32]
exists in ECMA-167 3/8.4.2 for exactly a damaged main sequence and is
still not consulted. Worth doing; a bigger change than this one.
decrypt_sectors_mapped returned early for any LBA no map range covers,
on the reasoning that unmapped means clear filesystem or nav. "The map
has no key here" and "there is nothing to decrypt here" are different
statements, and only the second makes passing the unit through correct.
On a multi-CPS disc an orphan clip — referenced by no playlist, so in no
title extent and therefore in no range — hits the first and was treated
as the second. The early return fired BEFORE the aacs_unit_encrypted
gate below it, so nothing ever asked whether those bytes were
ciphertext. extract_tree then counted them as bytes_good, dropped the
.partial suffix, set complete = true and exited 0: a scrambled file on
disk with a clean bill of health.
FileResult's own doc already stated the intended contract — "unreadable
sectors AND undecryptable units both land here (extract fails a bad
decrypt loud)". The code did not implement it on this path.
The fix is one line of symmetry. The split-unit branch immediately above
already makes exactly this distinction, checking aacs_unit_seed_encrypted
before refusing. This branch did not, so the same question got two
answers eight lines apart — the duplication shape this release keeps
finding.
Deliberately NOT changed: the decrypt decision on an orphan is still to
refuse rather than guess. Blind trial-decrypt is what the keymap-only
model exists to remove, and extract has no CPS/forensic fetch source.
The defect was never that we declined to key it; it was that declining
looked like success.
The test asserts BOTH directions — a clear out-of-range unit must still
pass through byte-identical, because that is the ordinary whole-disc
read and breaking it would trade one silent defect for a loud one.
ResyncGate::dropped is zeroed the moment a keyframe disarms the gate,
and the only EOF warning fires for gates STILL armed. So a mid-title gap
that resolves left no trace anywhere — and most gaps do resolve. A rip
with several concealed gaps reported 0 errors and 0 lost bytes while
whole GOPs had been discarded, which disc.rs's own test comment calls
the ONLY channel through which loss is reported.
This is the other half of 41e0982. Arming the gate after an 8 MiB
backstop discard is right — a picture with dangling references must not
ship — but until the drop is counted that trades silent corruption for
silent loss.
The gate now carries dropped_total alongside dropped: per-run answers
"how expensive was this gap", cumulative answers "what did the caller
lose". errors() sums the gates.
Summed in ONE place rather than counted at the three admit call sites.
Three copies of the same increment is how the mux-flush path ends up
counting and the main path not, or the reverse — the duplication shape
this release has been removing. The gate already knows its own total;
the accessor just has to ask.
Found independently by two round-9 lenses, which is what raised it from
plausible to worth acting on.
Both halves are pinned: removing the dropped_total increment reds the
resync test, and removing the sum from errors() reds the DiscStream one.
The second test asserts the ACCESSOR rather than the gate's counter,
because a test on the counter would have passed throughout the entire
period the defect existed.
Resolution::pixels() returned (0, 0) for Unknown, and the MP4 sink wrote
it verbatim into tkhd (ISO/IEC 14496-12 8.3.2) and VisualSampleEntry
(12.1.3). Both fields are MANDATORY there, so unlike Matroska — which
omits the optional PixelWidth/PixelHeight elements — MP4 has nothing to
leave out. The result was a structurally complete file that passes every
container check, declares a 0x0 video track, cannot be rendered, and is
written with no error anywhere.
WHY IT WAS POSSIBLE, which is the part worth keeping:
pixels() previously fabricated 1920x1080 for Unknown. That was wrong but
playable, so this sink never needed a guard and the absence of one was
invisible. Changing the sentinel to (0, 0) moved the defect instead of
removing it — a zero PAIR still reads as a usable value, so the sink
stored it and serialised it.
The accessor's doc comment then ENUMERATED the callers it believed were
safe: "the Matroska sink omits the optional elements, the VobSub writer
omits its size: line, and no caller divides by either dimension." Two of
those three are true. MP4 was not on the list because MP4 has no guard
at all, and a prose list cannot enforce itself. mkv.rs's own comment
even states the principle — "the check belongs in the one accessor
rather than in each caller that remembered to write it" — and
labels/mod.rs still carried its own duplicate Unknown test long after
the accessor took that job over.
So: pixels() now returns Option. Not because Option is tidier, but
because every caller genuinely needs a DIFFERENT answer and the compiler
is the only thing that reliably makes them choose one. Matroska and the
metadata sinks take unwrap_or((0, 0)) with the reason stated at each
site; the VobSub path degrades to a palette-only .idx; MP4 fails with
E_MP4_UNKNOWN_RESOLUTION (9055).
Six call sites, not the five my first grep showed — I piped it through
`head` and acted on a truncated list. The compiler caught the sixth.
That is the same mistake as trusting a lens that reported silence.
descramble_region descrambled with the key a sector's own crib had just
proven stale, whenever the re-crack from that sector also failed. The
clear header is not scrambled, so it survives intact: the sector still
opens with a valid pack start and passes every structural check the PS
demuxer applies. Only the payload is corrupted — exactly where nothing
looks. Ok(0) dropped, exit 0.
CSS has no external key source. The title key comes only from cracking
the data, so on a READABLE sector "no key" is not a missing input, it is
recovery failing on bytes we can see. That should never happen, and when
it does the answer is not to emit something.
Now Error::DecryptFailed — the same verdict the AACS path already gives
for a unit no held key opens. Both alternatives to failing are bad data
reported as success: descrambled with a rejected key it is garbage
behind a valid header, and passed through untouched it is ciphertext
where plaintext is meant to be.
WHY IT WAS POSSIBLE, which matters more than the fix:
There is no single place that owns "what do we do when there is no key".
decrypt_sectors_impl looks like the central dispatch, but its AACS arm
is a `return Err` stub — AACS decrypts entirely through
decrypt_sectors_mapped, a separate top-level path. So CSS decided its
own policy inside css/, AACS decided in decrypt.rs and mux/resolve.rs,
and nothing held them to the same answer. The asymmetry was not an
oversight; it was structurally permitted.
How a disc decrypts is one process — resolve a key for this data, apply
it, refuse if it cannot be proven. Only the resolve-and-apply step is
scheme-specific. Filed as a task: the policy belongs in one orchestrator
with the schemes supplying only what genuinely differs.
Two tests changed rather than added, both of which pinned the old
behaviour: the unit test asserted the sector was descrambled, and the
integration test asserted the scramble flag was cleared, which is what
descrambling-with-any-key does. Neither established that the result was
CORRECT — the fourth bad-test shape.
Regression I introduced in 8e70f2a earlier today. The Metadata File
Location fix replaced
read_sector(reader, meta_file_lba, &mut meta_icb)?;
with a candidate loop guarded by `.is_ok()`, which discards the error.
A transient read fault — marginal sector, drive re-read, an ECC recovery
that reports failure once — then falls through to the block-0 fallback,
the File Set Descriptor read there finds the wrong tag, and the volume
comes back as Error::UdfNotFilesystem: the deterministic verdict "this
is not a UDF disc" for a retryable I/O event.
mux::resolve MEMOISES that negative for the whole disc, so one flaky
read silently demotes every remaining title to the base-Unit-Key-only
path — the AACS 2.1 forensic units garble and the demux drops them. The
mux completes with less content and no error, which resolve.rs's own
comment says must never happen.
This file already draws the same distinction twice in prose, at the AVDP
check and the FSD check: read fine but the bytes say no is structural;
could not read is transient. The loop erased it.
Two things I got wrong on the way, both worth recording:
The first guard was `meta_tag == 0` — nothing read at all. That misses
the damaging case, which is the RECORDED candidate faulting while block 0
reads fine and holds some other descriptor. meta_tag is then non-zero and
not 266, and the fault is still laundered into a structural negative. The
guard is now `meta_tag != 266`: any unread candidate leaves the verdict
unproven, so the fault wins.
The first test denied BOTH candidate LBAs, and passed WITHOUT the fix —
the block-0 read fails too, so a read error propagates from further down
either way. A test whose fixture never reaches the changed line. That is
the eleventh bad-test shape this audit catalogued, and I walked straight
into it while fixing a defect found by looking for it. The fixture now
denies only the recorded location, and removing the guard reds it.
Found by the round-9 opus escalation over correctness, dispatched
because the sonnet pass over the same scope returned zero findings.
Silence from a cheap model is not evidence.
identify_master_enums picks, for each fingerprint, the best candidate
class out of CandidatePool. Its tie-break only prefers an exact ldc
count over an inexact one, so two candidates that are BOTH inexact but
both within LDC_COUNT_TOLERANCE are decided purely by iteration order —
and the pool was a HashMap.
Rust seeds HashMap per instance, so this is not merely unstable across
runs: the new test resolves the SAME jar to both Alpha and Beta within a
single process, across 16 iterations. The same disc could emit different
commentary/SDH/descriptive labels on consecutive rips of unchanged
input, with nothing in the output saying the choice was arbitrary.
BTreeMap fixes it by construction rather than by a sort someone can
forget to keep. The pool is capped at MAX_CANDIDATE_CLASSES, so the
ordering cost is irrelevant.
The test runs the whole identification sixteen times and asserts one
distinct winner. A single run cannot distinguish deterministic from
lucky, and the seed does not change within a process — so repetition is
what makes this a test rather than a hope.
Found by the round-9 labels pass, which was dispatched specifically
because every one of the ten lenses had reported leaving deluxe.rs
unread. 1,159 new lines that nobody had opened.
Round 9 findings, triaged and verified against the pinned tree.
writeback_file: a bounded-fsync WorkerLost returned bare ErrorKind::Other
on Linux where macOS returns EIO. Round 8 fixed the Linux arm to return
Err at all — the right fix — but stopped short of matching the value, so
a consumer distinguishing timeout / halt / lost-worker had nothing to
branch on for the third case on one platform. Now EIO on both.
Three doc comments described the pre-fix behaviour, one of them for
longer than the bug existed:
linux.rs durable_sync still said "all three fallbacks return Ok(())"
mod.rs sync_all still said Linux silently swallows fsync failures and
callers must not treat Ok(()) as a durability barrier
mod.rs SequentialSink::finish repeated the same caveat
All three now say what the code does: a bounded-fsync failure is an Err
on every platform, so Ok(()) IS a durability barrier. A doc that
describes a fixed bug is worse than no doc — it tells a caller to write
a workaround for something that no longer exists.
au_assembly: discard_gap_before duplicated drop_marks_before's
mark-retirement body verbatim and added one statement. Mine, from
earlier today. It now calls it. Two copies of the same retirement loop
is exactly how the two call sites would drift back together.
clpi: ClpiStream's audio_format / audio_rate / video_format / video_rate
are decoded from untrusted on-disc bytes on every parse and read by
nothing. The identically-named fields consumed in disc/bluray.rs belong
to mpls::StreamEntry, not to this struct — checked, because an earlier
round wrongly called a live function dead. Deleted, along with the seven
test assertions that pinned them; the tests that pin pid, coding_type
and language remain. Also removed a section-header comment orphaned by
the get_extents deletion, describing a fixture that no longer exists.
Thirteen agents triaging src/labels and src/disc died on a saturated
machine, leaving 5,836 insertions across 28 files uncommitted in a
worktree. Recovered by 3-way apply onto twelve commits of drift; zero
conflicts. The diff was archived to private-repo first, because a
worktree is not a backup and this one had already nearly been lost.
One production change, and it is the right one: mpls_universal::parse
read every playlist off the disc AND converted the entries to labels in
a single function, so the conversion — stream-type mapping, dedup key,
the dense global counters — could only be reached through a synthetic
UDF image. Extracted to build_labels(&[Playlist]), which unit tests can
drive from already-parsed values. Behaviour-preserving: same iteration
order, same skip-on-error.
Two collisions resolved by hand:
A second mod pass_progress_tests, written independently against the
same survivors as the one committed in 337b57e. Kept mine — it covers
the distinct-counters case and the Progress blanket impl, which theirs
does not — but theirs had three clamp tests mine lacked: good_pct,
bad_pct and pending_pct also clamp an overshoot, and I had only tested
that for work_pct. Merged those in as one test and proved each of the
three clamps load-bearing by removing them individually.
An unused_parens warning in a new fixture.
Method note, recorded because it cost real time: git apply --3way
STAGES its result, so `git diff` reads empty and the tree looks
untouched. I nearly concluded the patch had silently failed. Worse, the
first attempt piped through `head -20`, so `echo exit=$?` reported
head's status rather than git's — the same mistake this audit has
already documented once. Check the real exit status, and check
--cached, not just the working tree.
Second pass over src/aacs and src/css. No production change; the only
non-test edits are two fixture bytes and one test rename.
Five latent panics on untrusted data, every guard correct and none
tested — so each was free to be deleted:
variant.rs:224 a 0x04 record not a multiple of 5 indexes p_uv[0..4]
off a one-byte tail
variant.rs:269 a 0x0c record shorter than the 0x04 slot count
slices past the cvalue table
stevenson.rs:177 short sector read -> index 138 into a 129-byte slice
stevenson.rs:208 a crib longer than the 1920-byte encrypted region
-> index 2058 into 2048
stevenson.rs:272 a header periodic all the way to offset 0 ->
subtract with overflow
That last one is reachable from ORDINARY DVD data — constant or padding
bytes are periodic. Verified on HEAD: widening the guard to <= 0x80
passes all 64 css tests unmutated.
media_key_variant_from_kp had only a soft-correction test, so every
step past that early return was unexecuted. The new two-slot fixture
puts the covering slot at index 1, so the uvs[1 + 5*idx] and
cvalues[idx*16] strides stop multiplying by zero.
derive.rs:319 + -> - confirmed killable, as the first pass predicted:
p == 0 makes (p-1)..32 underflow. Every prior fixture used a uv whose
lowest set bit was 4, 10 or 11, so trailing_zeros() was never 0.
One fixture bug caught and fixed rather than papered over: a |= mutant
first SURVIVED because mk[14]'s 0x04 bit happened to be set, making OR
and XOR agree. The byte is now clear and an assert_eq! pins it, so the
fixture cannot drift back into agreeing with the mutation it exists to
catch.
walk_mkb_be24_high_byte_is_honored renamed to
walk_mkb_be24_middle_byte_is_honored. Its 0x00_0110 length exercises
the << 8 term only, which is why << 16 -> >> 16 survived it. The name
was the lie; both framings are worth having, and the comment now points
at the genuine high-byte test at 0x01_0004.
derive.rs 146:32 and 154:30 stay untested, now with a proof rather than
a judgement: bit_pos == -1 requires current_v_mask == 0xFFFF_FFFF, and
calc_v_mask can never return that — its loop condition holds at
!v_mask == 0, so it always shifts at least once. Both branches are
reachable only after the walk has gone non-convergent and is heading
for the bounded exit, where the return value is undefined. Termination
is already pinned.
Equivalents proven by observing green, including six more OR/XOR pairs
on provably disjoint bit fields, and the two KEY_CORRECTION_DATA sites
where the constant is the documented all-zero placeholder so x ^ 0 ==
x | 0. Those become killable only if a real per-licensee KCD is wired
in.
A partial confirmation sweep (138 of 415 mutants before the box
saturated) found 135 caught, one timeout that is itself a detection,
and exactly one survivor — the KEY_CORRECTION_DATA equivalent above.
read_filesystem hardcoded the Metadata File's File Entry at block 0 of
the physical partition — 'the metadata file ICB is at physical
partition lba 0'. UDF 2.50 2.2.10 records where it actually lives, as a
partition-relative Uint32 at offset 40 of the Metadata Partition Map.
That field is the only thing on the volume that says where the entry
is; block 0 is merely where authoring tools usually put it.
On a conformant volume that recorded it elsewhere, block 0 holds
something that is not a File Entry, metadata_start falls back to
partition_start, the File Set Descriptor read there carries the wrong
tag, and the volume is rejected as UdfNotFilesystem. Worse, a volume
with a decoy file set at block 0 — as a rewritten or dual-structure
volume can have — does not error at all: it mounts a different
filesystem and reports success.
Verified on HEAD: reverting the lookup reds four tests, e.g. the
metadata partition beginning at 2000 where the map records 33754069.
The recorded location is trusted only when the map's partition type
identifier reads '*UDF Metadata Partition'. A Virtual (2.2.8) or
Sparable (2.2.9) map is ALSO ECMA-167 3/10.7.3 Type 2 and records
unrelated fields at offset 40, so its bytes must never be read as a
location. Deleting that guard reds its own test.
Block 0 stays in the candidate chain, so a volume whose map is absent
or wrong but whose Metadata File does sit there keeps mounting exactly
as before. This is additive, not a behaviour swap.
Also 30 tests and ~55 more mutants across read_icb_extents,
read_file_limited, read_inline_data, the prefetch stubs, parse_dstring
and parse_udf_name. The metadata-partition branch — the branch EVERY
real BD-ROM takes — had no test at all; nothing in the crate built a
two-partition-map volume.
Closes the max_bytes gap flagged earlier: 259 > -> == and > -> < now
die on both the declared-size and the inline-ICB paths.
Equivalents proven by application, notably two guards that read as
protective but are unreachable: pm1_len is a single byte so
440 + pm1_len < 2048 always holds, and ad_offset + l_ad <= 2048 is
enforced upstream so off + ad_size never exceeds the block.
Bit 0 (Existence) stays unread, deliberately. ECMA-167 4/14.4.4 makes
it a display hint, not a statement that the file is absent, and UDF
2.50 2.3.4.2 carries it through as the DOS hidden attribute. For a
ripper the consequences are asymmetric: honouring it can silently drop
a real .m2ts from the title list, ignoring it costs an extra name in a
listing.
Known structural limit, not fixed: read_filesystem takes only the FIRST
extent of the Metadata File, so a fragmented metadata partition would
map every sector past that extent to the wrong place. metadata_start
being a single base LBA is what forbids the fix.
Third pass over src/mux/. 40 survivors killed, no production change.
resolve.rs — the deleted-statement cluster is now fully constrained.
All 14 deletable statements probed; 9 were already caught, 5 survived:
c.sort_unstable() in fill_base_key_gaps. Every existing case handed
it cuts already in LBA order, but IndividualSegment.tbl is a record
list. Verified on HEAD: deleting the sort passes all 54 resolve
tests. The mutant lays a base-key fill straight over a forensic
segment.
last_idx = idx (FMTS gap fill) and last_idx = hit (multi-CPS cache
hit). An extent with nothing to sample must inherit its neighbour's
CPS unit; the mutants fall back to the first unit's key. Exactly the
shape this file's own comments name — wrong key, no error,
lost_bytes == 0.
Both check_halt()? polls in probe_fmts_index_keys. These cannot be
killed by outcome, since a later poll returns Halted too. The tests
count reads instead, which is what the don't-hammer-a-struggling-
drive rule actually says: after a Stop the drive is asked for zero
content sectors.
The four unresolved += 1 arms each got a test, and deleting each fails
exactly one — one-to-one, so no fixture passes for the wrong reason. A
control test pins that the baseline table resolves, so an expect_err
cannot succeed for an unrelated reason.
ts.rs::scan_streams was never entered. Six killed, two of which return
wrong answers that look right: reading the PAT/PMT CRC as a table entry
invents a stream on PID 546 out of CRC bytes, and dropping the
ES_info_length skip decodes a descriptor as an entry and loses the one
after it. Every existing PMT fixture declares ES_info_length = 0; a
real BD PMT carries a registration descriptor on essentially every
entry. Also ISO/IEC 13818-1 2.4.4.3 program_number == 0 is the network
PID, not a program.
mp4/read.rs — 13. Height read as the width beside it; channelcount;
the 4-byte base-128 descriptor varint (every existing esds fixture uses
a single byte); all three optional ES_Descriptor fields, whose loss is
silent (an AAC track just loses its CodecPrivate); first-vs-last media
edit, which is A/V desync of the difference; and the version-1 mvhd
timescale offset, emitted by any writer whose duration exceeds 32 bits.
dts.rs — 7, from a real cargo-mutants run over the file rather than
guesswork. Including a buffer that IS the syncword, which is the state
a sync split across PES packets lands in the moment its last byte
arrives.
Equivalents proven by application, not argued: the sample_encrypted_units
guard pair is mutually redundant by construction (total*p/9 < total for
p <= 8), so either alone is equivalent and both together are not; the
PMT section_len guard is dead code where its PAT twin panics; three of
the seven EXSS_HEADER_MIN_BYTES arithmetic mutants still sum to 10.
drop_marks_before retired discontinuity marks alongside timing marks at
both of its call sites. At stream start that is right. At the
MAX_AU_BUFFER backstop it is not, and the two are now separate.
The backstop fires when 8 MiB accumulate with no AU start code in them
— corrupt or hostile input — and throws the run away. There IS a prior
AU in that case, and whatever emits next definitively does not continue
it: a decoder handed that picture resolves its references against
frames separated from it by megabytes of discarded data. Retiring the
flag meant the resync gate (resync.rs, driven from mux/disc.rs) never
armed, so the broken picture went out looking sound. Silent corruption
is the one class of loss this crate refuses to have.
At stream start the opposite holds. Bytes ahead of the first
access-unit delimiter are the tail of an AU that began before sync, and
there is no prior AU to be discontinuous from. Marking it would arm the
gate at the head of every title and drop its opening GOP. That risk is
why this was a decision rather than a fix, and splitting the call sites
is what avoids paying it.
Recorded as a sticky flag, not an offset mark. A mark placed at the new
base is retired moments later by the pre-sync trim that follows resync
— the gap has to outlive the bytes that caused it. I found that by
writing the test first and watching it fail with the mark approach.
The discard is a discontinuity whether or not the source signalled one,
and a signalled one on discarded bytes still reaches the AU that
follows; both directions are tested.
Note the first over-cap run is NOT a discard: the next AU's delimiter
is still at buf[0], so it force-flushes as an over-long access unit and
loses nothing. Only a run with no opener at all reaches the backstop.
The tests push twice for that reason — the single-push version passes
without the fix.
Swapping either call site for the other fails: reverting the backstop
reds the two gap tests, and arming the gate at stream start reds the
third.
163 of 322 surviving mutants across src/aacs and src/css. No production
line changed — every function read correct; the finding was always an
absent test.
Two structural holes, both verified against HEAD before landing.
variant.rs had no test that ever produced a Media Key. Every terminal
assertion in the module was an Err classification — NotVariantMkb,
SoftCorrectionRequired, OnlineChallengeRequired. So the entire 2.1
success path (VARIANTS lookup, VKD selection, Kpnew, the final unwrap,
the verify gate) was pinned by nothing, and that path produces the
Media Key that becomes the VUK that decrypts every byte of a 2.1 disc.
Built the first complete planted variant MKB: the VARIANTS entry is
chosen as Kvn ^ 1 so the real VKD sits behind a decoy at table index 1,
making the lookup load-bearing rather than incidentally correct. That
one fixture kills 23 operator mutants across three functions.
aesg3 — the subset-difference tree node function — was in the survivor
list as replaceable by [0; 16], meaning every device key in the crate
would derive the same Processing Key. It is caught today only as a side
effect of a negative test added after the mutation run; nothing asserted
the relation itself. Pinned now via the spec relation ([C] 3.2.2) using
the FORWARD primitive, with s0 transcribed independently rather than
read back from AESG3_SEED, so the test cannot agree with a mutated
constant.
Same shape in derive.rs: plant_mkb was one slot with zero descent, so
slot indexing was the identity permutation and the ancestor-descent
branch never ran — which is why 39 of recover_dk_position's mutants
survived. Added a 3-slot fixture keyed at index 2 and a four-level
descent fixture whose expected Processing Key is written out as an
explicit aesg3 chain rather than computed by calc_pk_from_dk; a fixture
built by the function under test moves with its own mutations.
Two latent panics on untrusted input now have tests: a 0x05 cvalue
table shorter than the 0x04 slot index, and a drive declaring more
payload than the 32772-byte response buffer holds.
23 equivalents claimed with reasoning, and confirmed empirically where
possible — all eight css/lfsr mutants were run and exactly the seven
disjoint-bit-lane ones survived.
Explicitly NOT claimed equivalent: derive.rs 146:32 and 154:30 are
reachable, but only on the non-convergent bounded-exit path where the
function's sole contract is termination. A test there would pin
defined-but-meaningless output.
Noted for the next pass: the pre-existing walk_mkb_be24_high_byte_is_honored
used total length 0x0110, whose high byte is zero — it exercised the
middle byte only, which is why << 16 -> >> 16 survived it. Left in
place; a real one was added at 0x01_0004.
get_extents had no caller anywhere in the ecosystem, and neither did
anything feeding it. Removed: get_extents, resolved_ep_map, full_pts,
full_spn, parse_cpi, EpCoarse, EpFine, the ep_coarse/ep_fine fields,
the unused version field, and the 35 tests that exercised them.
This reverses the fix in f2ca0e4, deliberately. That commit corrected a
real truncation — out_time past the last EP entry resolved to
last_ep_spn + 1, dropping everything from the final I-frame to EOF —
and the fix stands in history as the record of what was wrong. But the
defect existed for eight audit rounds precisely because the code had no
caller: nothing exercised it, so nothing noticed. Keeping speculative
infrastructure alive on the strength of a comment saying it is reserved
for a path that does not exist is how that happens again.
ClipInfo keeps source_packet_count (read by disc/bluray.rs) and streams
(the CLPI/MPLS cross-validation in labels/clpi_audit.rs). 1,139 lines
out, gate green in debug and release.
ClipInfo::get_extents fell back to `last EP SPN + 1` whenever out_time
lay past the last entry-point. EP entries mark I-frames (BD-ROM Part 3,
CPI / EP map) and a clip's final GOP lies after the last one, so a
PlayItem covering a whole clip — whose OUT_time is the presentation end
— always lands in that arm. The extent then stopped one source packet
after the last I-frame.
Measured on a fixture with 200,000 source packets and the last EP at
SPN 131,072: sector_count came back 12,289 where covering the clip
needs 18,750. Everything from the last entry point to EOF is outside
the returned extent.
Scope, stated plainly: get_extents has NO callers anywhere in the
ecosystem today — it is #[allow(dead_code)] and documented as reserved
for the timestamp-range read path. Nothing ships this loss. It is
fixed now because a latent truncation in extent arithmetic is far
cheaper to correct before it has callers than after.
The SPN at-or-after an out-of-range out_time is the end of the clip,
source_packet_count, with .max(last + 1) so a disc that under-declares
its own packet count against its own EP map still yields a sane bound.
Also 174 mutants killed across clpi, mpls, ifo and ebml — the first
time any of these four files has been examined. And ebml's 8-byte VINT
back-patch was duplicated verbatim in end_master and end_master_buf
with its top four payload octets unreachable through either (they need
a 16 MiB..256 TiB buffer); extracted to fixed_width_vint8 and tested
across the full 56-bit payload, no behaviour change.
38 of ebml's 46 survivors are one equivalence cluster: every | in
write_size / read_id / read_size / read_uint_val ORs into disjoint bit
lanes, where ^ is the identical operation. Applied all 38 at once —
green — then spot-checked four individually.
Five surviving mutants in AacsKeyMap::read_plan, all in the arithmetic
that decides which half of a forensic segment this disc's key opens.
The existing coverage used a forensic range starting exactly on the
extent's first unit. Under that shape several wrong formulas agree with
the right one by arithmetic accident: (lba + range_start) / us and
(lba - range_start) * us both produce the correct kept set.
Real ranges are not shaped like that. A range start comes from a source
packet number — start_spn * 192 through clip_byte_to_lba in
mux/resolve.rs — and 192-byte packets bear no relation to the 3-sector
aligned unit, so range_start % 3 is whatever the disc says.
Getting the parity wrong does not crash. It reads and decrypts the
ALTERNATE variant's half: the units this key does not open decrypt to
garbage, the units it does open are skipped. AACS 2.1 forensic marking
is precisely what makes the two halves differ, so the failure is silent
— a full-length rip carrying the wrong variant.
Two fixtures are needed because no single one kills both: an unaligned
range start with the extent beginning on it inverts the halves under
the + form, and an extent offset one unit-remainder from the range
start inverts them under the * form.
Also pinned the short-tail guard. is for a remnant
SMALLER than a unit — bytes with no following unit to desync. Widened
to <=, the last whole unit of every extent bypasses the phase gate, so
a forensic segment ending at an extent boundary contributes one
alternate-variant unit to the rip.
Second mutation pass over src/mux/. 26 survivors killed, no production
change. Verified on HEAD before landing: each mutation below passes all
1,237 mux tests unmutated-suite.
The priority item was the honest-loss-reporting surface. Both
DiscStream::errors and DiscStream::lost_bytes could return a constant
with nothing failing — a rip that lost sectors would report zero loss
to the caller. This project has already shipped one defect of that
shape (a total decryption failure reported as an empty title, exit 0).
Driven now through two short-read fills so both land on values that are
neither 0 nor 1 and differ from each other; no constant and no field
swap survives.
MkvStream::finish -> Ok(()) also survived. MkvMuxer::finish has the
zero-frame MkvInvalid guard and two tests cover it, but the Stream
wrapper above it could return Ok unconditionally and bypass the guard
entirely — the empty-title defence was one layer thinner than it looked.
au_assembly: pinned au_opener_from behaviourally to the normative byte
values for all four modes, with negative cases for codes that are
explicitly not openers (MPEG-2 slice 0x01..0xAF, user data 0xB2,
extension 0xB5, sequence end 0xB7 per 13818-2 Table 6-1; VC-1
0x0A/0x0B/0x0C; H.264 SPS/PPS/IDR-slice). au_assembly and codec/ hold
independent copies of these constants; they agree today, and comparing
constants would not catch logic drifting apart, so both sides are now
pinned to the spec instead of to each other.
demux_sink::sanitize: every filename component demux:// writes comes
from disc-controlled text, so the path-separator arm is a traversal
guard. Deleting it now fails, including an end-to-end case where
base = "../evil/Title" must produce exactly one file inside the
chosen directory.
stts_and_ctts_expand renamed to stts_expands_runs_to_per_sample_deltas_in_order
and given runs with distinct deltas AND distinct lengths. Its old name
claimed ctts coverage it never had, which is why the composition-time
chain went unconstrained for eight rounds; the doc comment now points
at the tests that do cover ctts.
Correction to the previous pass: codec/truehd.rs flush -> vec![] IS
equivalent. Applied it, full mux suite green. TrueHD buffers across PES
but parse emits every complete unit immediately, so a residual buffer
at EOF is a truncated access unit and is correctly discarded. The
vec![Default::default()] variants are genuinely different and are
killed.
Deliberately not constrained: mkv::set_opening_capture (diagnostics
behind a process-global tracing check, flaky under the parallel
runner), and the three stdio.rs header paths (StdioStream holds
concrete io::Stdin/Stdout and cannot be driven without a production
refactor to injectable Read/Write).
Mutation testing over src/mux/. No production change — 49 survivors
killed, all proven red before green.
The MP4 composition-time chain was entirely unconstrained: VideoTiming::ctts,
build_ctts and parse_ctts could each return a constant and the suite
stayed green. Confirmed on HEAD: build_ctts -> vec![] passes all 1,220
mux tests. A demuxed B-frame title presenting in decode order would
have shipped.
The cause is a test whose name asserts coverage its body does not
deliver — stts_and_ctts_expand builds an stts box and never touches
ctts, and write_then_read_round_trip asserts sample sizes and keyframe
flags but not one PTS. Same shape as the set_speed forwarding finding,
different disguise.
mlp_num_substreams / mlp_substr_header_size: every TrueHD fixture in
the crate uses one substream and no extraword, so both could return a
constant and agree with all of them. These position mlp_parity_ok's
window over the AU header, so a constant mis-windows the parity check
on exactly the multi-substream AUs that carry 7.1 and Atmos.
CodecPrivate absent vs empty: mkv.rs writes Some(bytes) verbatim and
omits the element on None (RFC 9559 5.1.4.1.24), so a zero-length Some
emits a track header asserting the config IS empty. Four parsers could
return Some(vec![]) before any frame.
Also: mandatory ISO/IEC 14496-12 boxes (tkhd, vmhd, smhd, dinf, mdhd)
could each build empty; HEVC num_extra_slice_header_bits (H.265 7.3.2.3)
was never non-zero in any fixture, so the slice-type offset skip was
unexercised; chapter names from the disc go straight into
<ChapterString> and the & escape must run first; a stray 0x47 in a
payload must not latch a TS resync.
Documented as equivalent rather than killed: CodecParser::flush and the
three parser flush bodies that differ from the mutant only by a tracing
call, and DropTally::log_summary.
read_directory decoded file-characteristics bit 1 (Directory) and bit 3
(Parent) but never bit 2 (Deleted) — ECMA-167 4/14.4.4. It followed the
ICB of a descriptor naming a file that no longer exists.
4/14.4.3 permits a deleted FID's ICB field to specify an extent of
length zero, so it need not point at a File Entry at all. Following it
reads whatever descriptor occupies that metadata LBA:
- deleted DIRECTORY FID: recursion lands on the File Set Descriptor
(tag 256), hits the not-a-File-Entry arm and returns Err. One stale
descriptor in one directory fails enumeration of the entire volume.
- deleted FILE FID: read_file_size returns Ok(0) for a non-File-Entry
tag, so a deleted name is reported as a real zero-byte file.
Both directions of the same shape at once: a recoverable condition
becoming a hard failure, and a non-existent entry becoming a plausible
success.
Bit 0 (Existence) is still unread. Skipping hidden files could hide
real content, so it is left alone deliberately rather than folded in.
Found by mutation testing: 48 of the 51 surviving mutants in
read_directory are killed by the 17 tests added here, covering the
short_ad decode byte by byte (ECMA-167 4/14.14.1), the extent-type
mask, the AD bounds guard at the exact sector end, the tag-261 File
Entry directory arm that no test reached at all, the multi-sector
read offset, the FID stride including L_IU, and the nesting cap.
Three survivors are genuine equivalents and are documented as such:
l_fi > 0 vs >= 0 (the emptiness guard below reaches the same state),
and the << 32 / | in the visited-set key (meta_start is constant
across a walk, and the two halves are disjoint).
Mutation testing left both unconstrained.
sector/mod.rs — set_speed on the Box<dyn> and &mut dyn forwarding impls
could be replaced with an empty body and nothing failed. This one hides
better than the read methods because the trait's own default body is
already a no-op, so a forwarder that swallowed the call is
indistinguishable from a source with no speed control. Consequence is a
silently absent value, not a wrong one: the recovery path lowers read
speed through a damaged region, and a swallowed call leaves the drive
at full speed while the caller believes it slowed down. Routed through
a generic S: SectorSource bound, since a direct call on a &mut dyn
receiver auto-derefs to the vtable and never enters the forwarding body.
progress.rs — 42 survivors. All four percentage accessors could return
a constant, read the wrong byte counter, or have their divide-by-zero
guard inverted. Added exact-value tests (25%, not 'some percentage'),
both sides of each guard, the overshoot clamp, and one test setting all
three disc counters to distinct values at once — without it, a swapped
field still passes every single-counter test.
The Progress blanket impl for closures could return a constant true.
That return value is the cancellation signal, so a constant-true body
makes every closure-based consumer uncancellable.
Each mutation applied, observed red, reverted.
TROUBLESHOOTING step 2 said `freemkv info`, which needs a source URL;
the drive route is `freemkv info disc://`.
architecture.md quoted 1.5 s / 30 s for the read timeouts. The
constants are READ_TIMEOUT_MS = 10_000 and READ_RECOVERY_TIMEOUT_MS =
60_000 (src/scsi/mod.rs:72,94).
Fifteen surviving mutants killed, from the highest-risk class: functions a
mutant could replace wholesale with a constant while all 2,555 tests
passed. None of the code was wrong. In every case a test was absent, which
is why eight rounds of reading never found any of them.
The one that generalises is in sector/mod.rs. Its existing test READS as
covering `read_sectors` on the `&mut dyn SectorSource` forwarding impl —
it takes a `&mut dyn`, calls the method, checks the spy. But the receiver
auto-derefs and dispatches through the vtable straight to the spy, so the
forwarding body is never entered. An earlier round hit this exact trap on
`set_unit_base` and fixed it with a generic helper; the read path kept the
test that looked right. Verified by stubbing the forwarding impl to Ok(0):
the new test fails, the old one passes. That makes a tenth distinct shape
of bad test in this audit, and the mutation list is how to find the rest —
any forwarding-impl method in it has the same problem.
decrypt.rs's two existing gate tests assert only `dropped == 0`, which is
precisely what the `Ok(0)` mutant returns; one asserts nothing else at all.
A wrapper that decrypts nothing therefore looked correct while the caller
muxed scrambled MPEG. Now pinned by descrambling a real CSS sector and
comparing against the plaintext it was built from — not against a
re-derived descramble, which would only assert the code agrees with
itself.
css/mod.rs's `is_scrambled_uncracked` turns out to have no production
callers at all; the enum is matched directly. Its three tests all assert
only the true direction, which is exactly why the `-> true` mutant
survived. It is public API, so a consumer routing on it would, under that
mutant, refuse to rip every clear DVD.
aacs/inf.rs's MKB drive read had no test whatsoever. Now pinned
byte-for-byte across multi-pack concatenation, the single-pack case, a
genuinely empty response, and error propagation — an unreadable MKB must
surface as an error, not as an empty one.
aacs/derive.rs's nine mutants are killed with planted MKBs built by
inverting the AACS relations, so no real key material is involved. The
assertions land on the derived Media Key rather than the intermediate
positions: a recovered position that does not actually walk to the planted
key is no better than None. A fixture-guard test asserts the planted MKB
parses, since an unparseable one would make every `-> None` body look
right.
2570 lib tests, debug and release.
`km_verifies` is the gate deciding whether a candidate Media Key belongs
to the disc. The MK-pool brute force in resolve.rs runs every candidate
through it, so a version that said yes to everything would accept
whichever candidate it tried first and the rip would continue with a wrong
Media Key — wrong VUK, wrong title keys, garbage plaintext, and no error
raised anywhere.
Nothing tested it. Whole-crate mutation testing reported
`replace km_verifies -> bool with true` as SURVIVING: the body could be
replaced with `true` and all 2,556 tests still passed. A verification
routine whose verification was itself unverified, in the most
safety-critical function in the crate.
The implementation is correct — it matches the AACS relation
`AES-D(km, mk_dv)[0..8] == 01 23 45 67 89 AB CD EF`. Only the defence was
missing.
No real key material is required to test it. That relation means a valid
record for any chosen km is just `AES-E(km, <the constant> || anything)`,
so the fixture is self-contained. The test asserts three things: the key
the record was built for verifies; a key differing by ONE BIT does not,
which is the assertion that kills the mutant and is a near-miss rather
than a random key; and an MKB carrying no verify record does not default
to yes, because unverifiable and verified are different answers.
Confirmed by reintroducing the exact reported mutation and watching this
test fail.
This is the first defect found by mutation testing rather than by reading.
Eight audit rounds and a security lens that read this file in full all
missed it, because it is not a wrong line — it is an absent test, and only
an instrument that asks "would anything notice if this were broken?" can
see that.
Ten defects in code no previous round had ever scoped. `src/labels/`
identifies a disc's studio by parsing jar archives and JVM class files off
untrusted media, so every byte here is attacker-controllable — and 813 of
its lines were executed by no test at all.
The worst is a non-terminating loop. A fallback stream-number scan
advanced with `saturating_add`, and the comment says why: a crafted XML
"must not overflow (panic in debug, wrap-to-0 in release)". Once the
counter pins at u16::MAX and that number is taken, the loop cannot exit.
So a fix for an overflow panic produced an unbounded hang, which is
strictly worse — a panic is observable and catchable, and catch_unwind
cannot interrupt a live loop. Reachable from about 8 MB of XML.
Where the same overflow appears in the deluxe decoder the fix is
checked_add and stop, NOT saturation — twice wrong there, because
saturating would peg every stream past the ceiling at one number and
apply_labels binds on (type, number), silently mislabelling tracks. A
correctness bug wearing the costume of success.
Round 7 capped the ldc-string retention per class; nothing capped the
aggregate, so a 64 MiB jar held that budget for every class at once. Same
defect one level up, which is the shape that keeps recurring in this
directory. Four other amplifications are bounded the same way, each with a
stated headroom and a paired test proving real media passes untouched —
the tightest is 5x on a label length, the loosest 2000x on the stream
numbering space, against BD's 32-per-type STN_table limit.
Two are not caps at all: a quadratic membership scan became a set, and an
attacker-derived length added to a cursor without saturation now cannot
wrap. Nothing is excluded by either.
A `#[cfg(test)]` hand-copy of a shipping parser was the ninth bad test
this audit has found, and the first proven by mutation rather than
inspection: deleting the guard from the REAL function left all 26 tests
green, including the one named for that guard. Pointed at the real
function, the same mutation fails.
Separately, all three failure arms of the bounded fsync returned Ok(()) on
both macOS and Linux, so sync_all reported success for a durability
barrier that never ran. Only macOS was in scope; the Linux twin is fixed
here too, because a platform disagreeing with its sibling about whether a
failed sync is an error is the class that already produced an over-length
SCSI CDB macOS rejected and the other two truncated. Note the behaviour
change: a mux whose final sync times out on a wedged mount now fails
rather than exiting 0.
Three of the caps are proven by wall-clock deadline rather than an
operation count, with 18-80x margin on the passing side. On a heavily
oversubscribed machine those could flake.
as an empty directory
On an AACS 2.1 disc the non-forensic gap fill hardcoded pool slot 0 as
"the" base Unit Key, so every content LBA outside a forensic segment was
keyed with CPS unit 1's key even on a disc carrying several CPS units. It
does not fail loudly — it produces garbage plaintext. The gap fill now
resolves each extent's own base key from its ciphertext, sharing the
sampling and slot-picking the multi-CPS path already had rather than
adding a second copy, and memoised in the existing per-disc cache. A disc
with one base key still short-circuits with zero extra reads, which the
existing probe-cost test pins. Forcing the slot back to a constant fails
four tests, so the choice is load-bearing rather than incidental.
A directory ICB whose descriptor tag is neither File Entry nor Extended
File Entry (ECMA-167 4/14.9, 4/14.17) was turned into a successfully-read
EMPTY directory, indistinguishable from a genuinely empty one, while the
same tag on a file ICB was already a hard error. Fifth instance in this
audit of a failure converted into a plausible success value, and the
second in this very function — round 5 fixed a read error becoming a file
size of zero here.
An unrecorded extent (ECMA-167 4/14.14.1.1: allocated but not recorded,
logically zeros that still occupy file space) was dropped entirely rather
than contributing its length, so every later extent landed at the wrong
file offset. Silent corruption, not an error. Extents now carry a recorded
flag and the hole emits zeros without touching the media.
The Volume Descriptor Sequence was swept at hardcoded sectors 32..64 while
the anchor's own Main VDS Extent pointer was parsed into a comment and
ignored; ECMA-167 3/10.2.1 defines that extent by the field, not by
position, so a conformant volume placing it elsewhere failed to mount.
drive_status decoded byte 5 as Media Status without checking the event
header's NEA bit or notification class (MMC-6 §6.7), so a reply carrying
no media event descriptor decoded as "no disc". The drive is untrusted
input here, and this is the works-on-my-drive class.
Two more tests were found asserting the defects they sit next to — one
requiring unrecorded extents to be dropped, one that four drive fixtures
built non-conformant replies the corrected decoder rightly rejects. Both
rewritten. Combined with the DTS one in the previous commit that makes
three tests this round that locked a bug in as intended behaviour, which
is a different and worse failure than the tautological tests found so far:
a tautology fails to catch a regression, these actively defend the defect.
Not fixed, adjacent: extract_one_file streams extents sequentially and
will now READ an unrecorded extent's sectors rather than writing
guaranteed zeros. Offsets are right, and pressed media reads as zeros
there, but it is not zero-guaranteed the way read_file now is; that needs
a recorded flag through PlannedFile.
Seven defects in the code the test suite executes least — 913 lines of
disc/mod.rs alone are run by no test at all, which is why this round scoped
from coverage rather than from what previous rounds said they had read.
Disc::scan_image kept its own copy of the crack's extent ordering and fed
crack_key_outcome largest-cell-first. That is the fifth instance in this
audit of a local reimplementation drifting from the canonical one, and the
cost here is a key that does not descramble the feature: picking by sector
count bypasses the capacity gate and can select a different VTS entirely.
The copy is gone — which title comes from the canonical order the scan
already applied, and the extents are handed over in playback order,
exactly as decrypt_keys_for_title does. Its doc records why the duplicate
existed so it cannot grow back.
Resolution::pixels returned 1920x1080 for Unknown. That is the FOURTH
instance of one trap and the other three were in this same file, two of
them fixed hours earlier — without sweeping for siblings, which is the
whole reason this one survived. It now returns (0, 0), and the sweep was
done properly this time: every remaining Unknown arm across the crate is
honest, and the two ColorSpace sites that look like fabrication are
emitting H.273 code point 2, which is the spec's own "unspecified". Two
callers carried local Unknown-to-zero workarounds — precisely the cost of
making callers responsible for a lie — and one is now redundant.
BD-ROM Part 3 code 0xA2 is the lossy secondary DTS stream, not lossless
Master Audio. A test asserted the wrong mapping as intended behaviour, so
correcting the code failed it; the test is deleted with a note pointing at
its replacement. That is a NEW failure mode for this audit: not a test
that cannot fail, but one that locks the defect in. There is no
DtsExpress variant to map to, so it takes the lossy DTS-HD member and the
approximation is documented.
Also: DiscSession::identify could panic through drive_mut once the public
API allows an absent drive — two siblings were converted in an earlier
round and this one was missed; an extent end that added without saturating
where the rest of the crate saturates; a diag reason string restating the
comparator's sort keys and drifting from them, now derived from them; and
a short read that advanced the offset by the full request, silently
skipping the gap. That last one existed twice, in two reads with the same
shape, now merged so they cannot drift apart.
The short-read policy is a judgement call I could not derive from a spec:
no skip_errors is a hard error, with skip_errors zero-fills and charges
the loss. It deliberately does not retry mid-unit, because resuming inside
an AACS aligned unit would trade a silent gap for a silent decrypt
desync — the worse of the two.
Three copies of the same two mappings existed. The canonical accessors
returned 6 channels and 48000 Hz for Unknown; a third copy in diag.rs
returned 0. The honest one was the copy.
A plausible wrong answer is worse than an obvious one. Six channels at
48 kHz is indistinguishable from a real 5.1 track, so every caller became
responsible for remembering to check the variant first — and this crate
walked into exactly that: the json:// sink reported a confident 5.1 for
audio whose neighbouring fields said "unknown". That was fixed at the call
site earlier in this audit; this fixes it at the source.
The accessors now return 0, which is what both in-crate call sites already
coerced Unknown to by hand, so their guards are gone and the behaviour is
unchanged. Zero is also obviously wrong if it ever reaches output, where
six is not.
The diag.rs duplicates are deleted rather than corrected — a fourth copy
would have drifted too. Their only caller was a trace line in the same
file, now on the canonical accessors. Their tests moved across and gained
the Unknown case, which is the point: restoring either fabricated value
fails both.
Found by the round-7 correctness agent while fixing the json:// sink; it
flagged the third copy as out of its scope rather than touching it.
Five parsers that take bytes straight off a disc are now swept with
generated input asserting one property: they return Ok or Err and never
panic. That is this crate's own hard rule, and the class seven rounds of
reading is worst at.
Written in-crate rather than with cargo-fuzz, which needs a nightly
toolchain this project does not use, and without proptest or arbitrary,
because one dev-dependency is a deliberate posture and the parsers take
plain byte slices. What is given up is coverage-guided mutation, which is
the real loss. What is gained is determinism: the same seed replays the
same cases anywhere, so a CI failure reproduces locally verbatim.
Three generators, and the second is the one that matters. Pure random
bytes die at the magic check and exercise the entry guards only; prefixing
valid magic is what reaches the parser body; mutating a mostly-zero record
is what reaches the offset and count arithmetic a hostile image would lie
about.
That claim is MEASURED, not asserted. A harness whose cases all bounce off
the entry guards is the fuzzing equivalent of a test that cannot fail, so
one test counts how many generated cases parse to completion: 15,606 of
60,000, about 26%. If a future change to a guard drops that to zero, the
test fails rather than continuing to report a meaningless pass. Two further
tests pin that the three generators produce different bytes and that a seed
replays identically.
1.2M cases across all five targets found nothing. On this evidence that is
a real negative rather than an empty one.
The first version of this file was itself broken in the way this audit
keeps finding: its two meta-tests set FREEMKV_HARNESS_CASES and raced,
because the test harness runs them in parallel and env mutation is unsound
there. The budget is a parameter now, and the environment is read once at
the call site.
Two crate-internal parsers widened from private to pub(crate) so the
harness can reach them. No public API change.
The worst of the five was a regression suite that never touched the code
it guarded: nine batch-count tests called `safe_batch_count` and
`buggy_batch_count`, both defined in the test file itself. The u16
truncation they exist to prevent could be reintroduced in
sector/prefetched.rs with every one of them green. They now drive the real
producer through the public API, and reinstating the truncation fails five
of the nine. Worth recording that the symptom has changed since the
original fix: the unit-alignment clamp below floors a zero batch at three
sectors, so the bug is now a twenty-fold throughput cliff rather than the
stall it once was.
The MP4 reserve test's only numeric case was dominated by the floor and
the buffer, so BYTES_PER_SAMPLE could be zeroed without failing it. It now
has a case where the per-sample term dominates. The zero-count guard in
FileSectorSource was likewise unfalsifiable — seek-past-EOF and a
zero-length read both succeed — so the test now observes the file cursor.
The AACS media-key ambiguity guard had no test at all; the pool scan is
extracted so the verifier can be injected, because a genuine two-key
collision needs one ciphertext decrypting under two AES-128 keys to
plaintexts sharing a 64-bit magic, which is a 2^64 search and not a
fixture.
macOS implemented the documented cheap, side-effect-free presence probe by
building a full exclusive transport — which force-unmounts the disc. Linux
and Windows issue one TEST UNIT READY with no unmount; macOS was the
outlier. It now walks the IOKit registry for the media object instead.
The C shim's registry reads assumed CoreFoundation types the registry does
not guarantee, so a driver publishing a CFNumber where a CFString was
expected aborted the process from inside public API. Types are checked and
a wrong type treated as absent. The unbounded waitpid on the unmount child
is now a polled deadline, and the last-resort match gained the NULL check
its two siblings already had.
The empty-CDB guard existed only on Linux while a shared helper's comment
claimed all three backends had it. Moved into the helper, so the comment
is now true and macOS and Windows are covered.
One finding was REJECTED with evidence rather than fixed. The TrueHD
buffer-cap test was indeed bogus, but MAX_TRUEHD_BUF turns out to be
unreachable by any input: the parser only retains data when the buffer is
shorter than the declared AU, and that declaration is twelve bits, so the
worst case is 8189 bytes against a 256 KiB cap. An exhaustive sweep over
all 65536 AU headers confirmed it. The fixture now sits at the reachable
ceiling and asserts that instead. The cap itself is left in place as
defence, unreachable by construction, matching how the AC-3 resync guard
was handled earlier in this audit.
Two behaviour changes worth naming: Linux's empty-CDB error becomes
InvalidCdbLength rather than a transport failure, and an unknown device
now reports absent media rather than a not-found error, because the
registry cannot tell an empty drive from a missing one. The latter is a
conflation of the kind this audit has fixed three times; it is recorded
for the next round rather than left silent.
Five defects, four of them the same shape: a local reimplementation of
logic the crate already had, which had drifted from it. Each is now fixed
by calling the canonical version rather than by patching the copy.
DiscStream::new — the live disc:// path — built every parser through the
plain codec lookup and never asked whether a video stream was an MVC
dependent view, though resolve::build_demux_state does. The same 3D disc
therefore muxed correctly from an ISO and incorrectly ripped live. The
open-coded loop is gone; both paths now call build_demux_state.
collect_psi_section reimplemented the continuity-counter gap test and
disagreed with process_packet in the same file: it tolerated neither a
duplicate packet nor an adaptation-field-only packet, which per ISO/IEC
13818-1 §2.4.3.3 does not increment the counter. A spec-legal PMT
continuation was read as desync and the title's stream list came back
empty. Both callers now share one `cc_is_gap`, and a duplicate packet's
payload is no longer appended twice — doing so would have corrupted the
section the check exists to protect.
The json:// sink called the channel-count and sample-rate accessors
unconditionally, and both fabricate a concrete value for Unknown, so it
reported a confident 5.1 at 48 kHz for audio whose format was unknown
while its own neighbouring string fields said "unknown". The keys are now
omitted, matching mkv.rs. This matters more than it did: a sample-rate
ladder fixed earlier in this audit means Unknown now reaches consumers
that used to receive a wrong-but-concrete value.
For an audio:// or sub:// sink the reference video track's output is
filtered out, so its first PTS was never recorded and every delay was
computed against zero — baking a wrong DELAY into the filename. The
reference is now recorded whenever a frame is on the reference track,
independent of whether that track has an output, so a normal title gets a
correct delay; where no reference is ever observed the tag is omitted
rather than guessed.
A third copy of the channel/sample-rate mapping exists in src/diag.rs and
was left alone as outside the confirmed set. It is the same drift shape
and is recorded for the next round.
The Program Stream demuxer appended every fed byte and enforced its 4 MiB
cap only inside a branch reached once a start code had been found. Input
containing no start code anywhere therefore hit no cap at all, and since a
whole title is fed through this demuxer, a zero-filled or ciphertext VOB
extent buffered the entire title — up to ~90 GB. When the buffer holds no
start code, only a two-byte `00 00` prefix can begin a PS unit on the next
feed, so that is kept and the rest dropped. The bound is exact rather than
a heuristic: a start code can straddle a feed boundary by at most its
first two bytes, so no real byte is discarded, and a test feeding
`FF FF 00 00` then `01 E0 ...` pins that.
The existing test named for this case fed a real start code first, so the
cap it exercised was the in-PES one. Renamed to say what it covers.
The BD-J label path had a different shape to anything found so far: the
cap is on the COMPRESSED size of a disc file while the allocation scales
with the decompressed size. A `.class` gated only by a path prefix
inflates to the 64 MiB ceiling, yielding ~33M retained strings from `ldc`
operands or ~67M pushes onto a symbolic stack whose depth was unbounded
despite the Code attribute's own `max_stack` being parsed and then
ignored. Bounded both, the stack by `max_stack` itself (JVMS §4.7.3).
The VMG TT_SRPT title count is an untrusted u16 with no de-duplication,
so ~800 KB of crafted IFO re-parsed one PGC 65535 times. Capped at 99, the
DVD-Video maximum, so no conformant disc is clipped.
Every cap carries stated headroom against real media, and each has a test
locking that real media still passes.
I rewrote three of the new assertions before landing them. They compared
the result against the very constant under test — `total <= MAX_TT_SRPT_TITLES`
— which passes vacuously the moment someone raises the constant, the most
likely future regression and the seventh instance of this tautology shape
in this audit. They now assert literals derived from the spec.
The TT_SRPT fixture also had to change: with 65535 identical entries the
de-duplication collapsed them on its own and the cap was never what
bounded the result, so the test passed with the cap removed entirely.
Distinct entries defeat dedup and leave the cap as the only guard;
de-duplication now has its own fixture. Verified by raising each of the
three constants and confirming all three tests fail.
The bytes after a NAL header are EBSP, not RBSP: ISO/IEC 14496-10 §7.4.1
has the encoder insert 0x03 after any 0x00 0x00, and §7.3.1 removes it
before parsing. The measured-picture-type parse read the raw NAL instead,
on the stated reasoning that slice_type is too early for an escape to
intervene.
That holds only up to a point. first_mb_in_slice is ue(v), so a value of
65535 or more needs sixteen leading zero bits and opens the payload with
0x00 0x00, which an encoder must then escape. A UHD frame is ~32,400
macroblocks, so a conforming Blu-ray never reaches it — but 8K does, and
the disc is untrusted input. Such a stream decoded slice_type against a
byte the encoder had inserted and reported the wrong picture type: a wrong
result rather than an error, which is the class this lens exists for.
The prefix is un-escaped into a 16-octet buffer rather than the whole NAL:
the two ue(v) fields are at most 32 bits each, so nothing longer can be
needed, and it keeps a per-frame allocation proportional to the frame off
the path.
The test pins both directions. It asserts the un-escaped prefix decodes to
first_mb_in_slice = 65535 and slice_type = 2, AND that the raw EBSP does
NOT — without that second assertion the test would pass whether or not the
fix were present, which is the failure mode this audit has now found four
times. The bit string was derived independently rather than by hand: my
first attempt at the fixture was wrong by one nibble and the test caught
it.
Also covers the cases that must NOT be unescaped: a 0x03 not preceded by
00 00 is ordinary payload, and 00 00 03 03 keeps its second 0x03 because
the escape resets the zero run.
Both backends discarded the platform's own error code on the execute hot
path — the one every READ(10) of a rip goes through — and collapsed every
cause to the same status-0xFF transport failure.
On Windows, open() and reset() in the same file both capture the Win32
error; execute() did not. That left ERROR_INVALID_PARAMETER (a struct
layout regression, the exact class this file's SDK-layout tests exist to
catch), ERROR_ACCESS_DENIED and ERROR_GEN_FAILURE (a genuinely wedged
drive) indistinguishable, with nothing in the log to tell a code bug from
a hardware one.
On macOS the same, and worse: the file had no tracing calls at all, where
the Linux and Windows backends both log their execute failures. Its open()
carefully decodes the shim's sentinel into typed variants instead of
flattening them, but execute() threw the IOKit return away — so another
process taking exclusive access mid-rip and a real hardware wedge produced
identical, empty diagnostics.
Logged rather than added to the error type: the typed variant is public
API, and the recovery classification is deliberately the same for all of
these. What was missing is the breadcrumb, not the distinction.
Two further findings from the same sweep were rejected. Windows reset()
always returning Ok(()) and macOS ignoring timeout_ms are both already
documented in the code as deliberate, and the reporter flagged them for
completeness rather than as defects.
Neither fix has a test: reaching either branch needs a failing ioctl or a
failing IOKit call, and both files are compiled only on their own platform.
Four round-6 findings.
FileSectorSource::read_sectors guarded its output buffer with a
debug_assert, which is compiled out in release — so an undersized buffer
panicked with 'range end index out of range' instead of returning an
error, out of a public SectorSource impl where the length is caller input.
Drive::read_fua already carries this exact guard, with a comment recording
the same panic being fixed there, and PrefetchedSectorSource has a
regression test for the same case; this impl had been given neither. The
new test is red in release for precisely the predicted reason: 'range end
index 8192 out of range for slice of length 2049'.
parse_track's sample-rate ladder ended in an unconditional S48, so any
SamplingFrequency below 44100 was recorded as 48 kHz. A 32000 Hz AC-3 or
DTS track is legal and common in broadcast-sourced content, and the wrong
rate then propagated into the reconstructed AudioStream. Anything below
the lowest mapped rate is now Unknown, which is what the crate's canonical
SampleRate::from_hz already returned — the ladder disagreed with it. The
ladder itself stays, because the MKV element is a float and wants
tolerance rather than exact equality.
shim_open_exclusive used the mach port from IOMainPort without checking
the return; on failure the port is left untouched and every IOKit call
below ran against an uninitialised value. shim_list_drives in the same
file does check it.
build.rs treated cc and ar as successful if the process merely SPAWNED, so
a genuine compile error in the macOS C shim produced no object file and
surfaced later as an unexplained link failure against a missing symbol.
The shim is macOS-only and is neither linted nor compiled on the other two
platforms, so a mistake in it has exactly one chance to be noticed.
The last two have no test: one needs IOMainPort to fail, the other needs a
deliberately broken C shim, and neither is reachable from the test
harness. Both mirror a correct sibling in the same file, which is the
evidence available.
CI's clippy job runs on ubuntu-latest, so cfg(target_os = "linux") code is
what the gate actually compiles — and none of it is built by clippy on a
Mac. Five `-D warnings` errors were sitting in drive/linux.rs,
scsi/linux.rs, io/writeback/linux.rs and the Linux arm of drive/mod.rs:
four collapsible let-chains and one manual `% n == 0`. CI was red on the
lint job while the local gate reported all green.
Collapsed into let-chains, which the declared toolchain supports, and
folded drive/mod.rs's length precondition into its chain so the body no
longer needs a nested block.
Found because an agent working on the SCSI backends reported the lints in
passing while checking that a Windows-only file compiled. Worth noting how
it stayed hidden: every one of these files is cfg-gated to a platform this
machine is not, so no amount of local gating would have surfaced them. The
companion change to the precommit script closes that hole.
Two fixes from round 5.
The Linux and Windows backends truncated a CDB longer than 16 bytes
(`cdb.len().min(16)`) where macOS returned InvalidCdbLength. Under SPC-4 a
command's length is fixed by its opcode group code, so a shortened CDB is
not a shorter form of the same command — it is a DIFFERENT command, and
the drive executes it and answers GOOD with data for a request nobody
made. A silently wrong result on the layer everything else sits on.
Rather than mirror the guard a third time it now lives in scsi::mod as
checked_cdb_len, with all three backends routed through it, so it cannot
drift per platform again. That also makes it testable everywhere: each
platform module is cfg-gated to its own host, so a guard inlined into
linux.rs and windows.rs would have had no test coverage on any single
machine. The shared helper is the only place the behaviour can be
asserted on every platform's CI.
The two existing macOS tests were tautological — they replicated the
guard's logic inline instead of calling it, so they would have passed with
the guard deleted. They now call the real helper.
Separately, the H.264 keyframe parameter-set re-assert grew a
few-hundred-byte prefix buffer to the full access-unit size, copied the
whole frame into it, and dropped the presized buffer: one extra
whole-frame allocation and copy per keyframe. A UHD title is ~200,000
frames of 150-400 KB with a keyframe every second or two, so that is
thousands of avoidable multi-hundred-KB copies per title, each large
enough to go through mmap. It now splices into the reserved headroom in
place. This mirrors the identical fix already made in hevc.rs, which the
H.264 path had drifted from.
Byte-for-byte equivalence is pinned by a test whose expected literals
were captured from the pre-change implementation, and which I confirmed
still passes when the old build-and-copy code is restored. The
no-reallocation claim is measured rather than argued: a counter over 30
bare keyframes, which reports 30 of 30 against the old path and 0 with
the splice.
The reallocation test initially passed even with PARAM_REASSERT_HEADROOM
set to zero, because a small parameter set fits in the presize's
incidental slack — it proved the fixture did not reallocate, not that the
headroom prevented it. Its SPS is now large enough that the constant is
load-bearing, so zeroing it fails the test.
Not verified: no runtime behaviour on Linux or Windows: no drive, no
ioctl. Both files were confirmed to compile for their own targets.
Same shape as the mkv:// conflation fixed earlier in this round, found by
looking for it deliberately. E7023 carried two conditions with opposite
correct responses: one title on a multi-VTS DVD failing its own re-crack,
where skipping it and finishing the rest is right, and the main feature's
crack failing outright, which is disc-wide and dooms every title
identically. Because both raised the same code and that code is in
is_skippable_title_stub, an uncrackable disc walked all N titles printing
"title skipped, it was empty" and exited 0.
The disc-wide condition gets E7027 CssNoDiscKey, mirroring the AACS-side
E7022 NoDiscKey it is the analogue of, and joins is_disc_level_no_key.
The per-title raise keeps E7023 and stays skippable. Because the engine's
classifier already tests is_disc_level_no_key before the skippable
branch, this reaches the right outcome downstream with no change there:
such a disc now stops on the first title and reports no-key instead of
returning success with nothing written.
Disc::css_error deliberately still stores CssKeyMissing — autorip matches
that variant on the field to pick the CSS rather than AACS message, and
what consumers classify on is the gate's returned verdict, which is the
only thing that changed.
Two neighbouring CSS raises were examined and deliberately left alone:
the no-key branch in the same function is genuinely unreachable via
ensure_decryptable and documented as defensive, and resolve_dvd_title_key
is per-title on both of its call paths.
Verified by removing the new code from is_disc_level_no_key, which fails
both new tests; each pins both directions so neither can silently flip.
Not proven end to end against a real uncrackable disc — none available.
Five fixes. Three are real defects with regression tests; two are bounds
that were expressible but not expressed.
A fixed-size lace (RFC 9559 §10.3.4) whose body is empty declared n
frames and carried none. The divisibility check passed, because 0 % n is
0, and `chunks` yields nothing on an empty slice whatever width it is
given — so the clamp that existed to avoid chunks(0) returned zero frames
where the Lacing Head said n. The whole lace vanished with no error
raised and the caller saw a clean short block. A zero-size frame cannot
be a valid frame, so it is now malformed.
A disc read failure while fetching a directory entry's ICB became a file
size of zero rather than an error. Zero is indistinguishable from a
genuinely empty file, so an unreadable ICB on a damaged disc silently
changed which titles a caller saw as present — read_directory already
fails hard on its entry-budget guard, so propagating is also what the
surrounding code does. read_file_size still returns Ok(0) for an ICB
whose tag is neither File Entry nor Extended File Entry, which is a real
zero and not a failure.
The pending-frame buffer was capped at 4096 frames, which does not bound
memory: frames are arbitrarily large and a UHD video frame runs to a few
hundred KB, so the existing cap permitted over a gigabyte. Now bounded by
bytes as well, at 64 MiB.
round_up_grain overflowed for inputs within one grain of u64::MAX —
div_ceil then multiply — and the wrapped product is small, turning the
largest possible estimate into a negligible reserve. It saturates, and
the reserve is clamped to what a `free` box's 32-bit size field can
actually hold, since writing a larger one truncated the size and left
mdat beyond a box claiming to be far shorter. No real title comes close;
a 90 GB UHD title estimates a few MiB.
The AC-3 resync guard now advances the PTS cadence like both of its
sibling branches, so the three paths out of that block cannot disagree.
This one is defensive and has NO test: reaching it needs input that both
parses frames and leaves a megabyte of residue, and the parser's own
carry rules drop pre-sync junk and cap a partial frame at 8192 bytes, so
no such input was found. Stated here rather than covered by a test that
would pass either way.
Two findings from this round were rejected on inspection. A reported
panic in the .mpls suffix check does not exist: the `.get(..)` on the
line above returns None off a char boundary and `filter` never runs its
closure, so the byte index is unreachable. A test written for it passed
against the unfixed code, which is what surfaced the error.
E6008 meant two unrelated things: "this title produced no muxable
frames", which is a benign stub worth skipping, and "the source file is
malformed", which is not. Because a single code carried both,
is_skippable_title_stub answered yes to the second one — so feeding a
truncated or corrupt mkv:// input made the engine classify it
SkippableStub, print a notice saying the title was empty, and exit 0.
Silent data loss reported as success.
Split into E9053 MkvSourceInvalid for the read path (25 raise sites
across mkvstream.rs and ebml.rs's read primitives) and E9054
MkvUnencodable for the four write-side sites, which are the encoder
refusing to emit a body at or above the 56-bit VINT limit — an output
limit with no input involved, so calling it a corrupt source would be
wrong in the other direction. E6008 keeps only the zero-frame guard it
was documented to mean.
Kept one code for the whole read path rather than one per raise site:
nothing a consumer does differs between a bad VINT, a non-UTF-8 string
element, a truncated body and a child overrunning its parent. E9052 is
the model for when a carve-out earns its keep — laced blocks name one
specific RFC 9559 §10.3 feature with its own diagnosis.
Also fixed meta_sink.rs raising MkvInvalid for a serde_json encode
failure in the json:// sink, where no MKV is involved at all; it now
matches the identical guard in mux/meta.rs.
Reverting the split at the single code() arm reproduces the old
classification: 22 tests fail, including both new assertions. The
opposite direction is pinned too — dropping E6008 from the predicate
fails the genuine-stub test, which drives the real muxer end to end.
Twenty-six confirmed findings from the fourth audit round, landed as one
cluster because they were found by agents working over disjoint file sets.
The one worth calling out is a pair of AACS tests that could not fail.
Both asserted CBC behaviour against a hand-rolled expectation that
happened to be IV-independent, so replacing AACS_IV with sixteen zero
bytes left them passing — they were pinning the code's own arithmetic,
not the published constant. Replaced with a literal witness of the
published IV plus the NIST SP 800-38A F.2.2 CBC-AES128 vector, and
verified the other way round: zeroing AACS_IV now fails three tests.
The rest are allocation and correctness work on hot paths: the Annex-B
writer in demux_sink allocated and freed a whole-frame Vec per frame,
which for a UHD title is ~200,000 allocations over the mmap threshold
plus the page faults to first-touch each one; it now reuses a buffer on
the writer, and still takes the NAL prefix width from the configuration
record rather than assuming four.
Six findings whose real fix lives in a consumer crate are recorded for
re-filing rather than patched here.
Four conformance defects in the read paths, two of them silent corruption.
**Lacing was ignored entirely.** RFC 9559 §10.2 defines Xiph, EBML and
fixed-size lacing, where one Block carries several frames; the reader took the
Block payload verbatim, so a laced Block became a single "frame" consisting of a
lacing header followed by concatenated frames — garbage to the codec parser, no
error. Audio tracks from other muxers commonly use lacing, so an ordinary
foreign MKV was silently mangled.
All three modes are now parsed: Xiph 255-run sizes including the trailing-zero
rule for exact multiples of 255, EBML unsigned first size plus SIGNED VINT deltas
with the 2^((7*n)-1)-1 bias of §10.3.3, and fixed-size even division, with the
last frame's size deduced from the remainder. Laced timestamps follow §10.3.5:
the first frame takes the Block timestamp and the rest are spaced by the track's
DefaultDuration, else BlockDuration/count, else shared with a warn.
Parsing was chosen over refusing because refusal would leave freemkv unable to
remux common foreign audio at all, and each mode is about fifteen lines.
A malformed lacing header now raises a NEW code, E_MKV_LACING_INVALID = 9052,
deliberately NOT MkvInvalid — because is_skippable_title_stub classifies
MkvInvalid as a skippable nav stub, so reusing it would have recreated the exact
conflation that is still open as a separate finding. A test asserts the new code
is not skippable.
**TrackNumber was assumed to be 1..N in TrackEntry order.** RFC 9559 §5.1.4.1.1
only requires it to be non-zero and unique, so sparse or unordered numbers are
legal. Block routing and codec_private both computed track + 1. A real
TrackNumber map is now built, recorded only for TrackEntries that yield a stream
so dropped track types no longer shift the mapping.
Verified red here independently, and the failure mode is worse than mis-routing:
with track + 1 restored, a buttons track's payload was attributed to the AUDIO
stream — wrong payload into the wrong codec parser.
**The NAL length prefix was hardcoded to 4 bytes.** lengthSizeMinusOne lives in
avcC byte 4 and hvcC byte 21 (ISO/IEC 14496-15 §5.3.3.1.2, §8.3.3.1.2) and was
never read, so a source declaring 1- or 2-byte prefixes had its raw prefixed
bytes emitted verbatim with no start codes. All four conversion sites now derive
the width from the track's own configuration record.
**Edit lists were ignored.** No edts/elst was parsed, so the presentation
timeline an edit list defines (ISO/IEC 14496-12 §8.6.5/§8.6.6) was dropped —
which is how encoder delay is normally expressed. Leading empty edits and the
first media edit's media_time are now applied to both dts and pts, with the movie
vs media timescale distinction respected. A list needing more than a constant
shift applies the leading edit and warns rather than presenting the result as
faithful.
17 tests. I reproduced the lacing mutant independently: returning the body whole
kills five of them, including the exact-payload and malformed-header cases.
Still open and deliberately untouched: the MkvInvalid / is_skippable_title_stub
conflation across ~20 reader raise sites. It is a cross-cutting error.rs change
and E_MKV_LACING_INVALID is the template for it.
Two correctness defects in AACS 2.1 / FMTS key-map resolution, both order- or
anchor-dependent, and both able to abort a whole disc or silently garble it.
**The single-CPS short-circuit depended on which title resolved first.**
`pool_len` counted the WHOLE unit-key pool, and resolve_fmts_key_map appends the
disc's forensic index keys to that same caller-owned pool. The count was captured
before THIS call's FMTS branch but not before earlier titles', so once any forensic
title resolved, every later title saw a pool larger than one and fell into
multi-CPS sampling — 8 random reads per extent, and a whole-disc DecryptFailed if
no pooled key opened a menu extent's samples. A disc that ripped fine when a
non-forensic playlist sorted first failed when a forensic one did.
Forensic keys are now tagged FMTS_POOL_TAG_BASE = 1 << 24 and the short-circuit
asks single_base_key_slot(), which excludes them. The old 1000 tag was NOT kept,
and the reasoning is worth recording: base CPS ids are Unit_Key_RO.inf position + 1
and that count is a BE16, so 1000 sits inside a genuinely reachable id space. 1<<24
cannot collide. The id field is cosmetic — decrypt.rs indexes the pool by slot and
reads only the key — so widening the tag is safe.
**Forensic segment SPNs were anchored to the wrong clip.** They live in the
forensic feature clip's byte space, but were mapped through
clip_byte_to_lba(&title.extents, ..), which treats byte 0 as the start of the
title's FIRST extent. Any playlist not beginning with the forensic clip mapped
every segment to the wrong LBA: either the anchor probe sampled the wrong clip and
the whole-disc resolve aborted with FmtsKeyMissing, or — worse — a forensic index
key was applied to non-forensic sectors while the real forensic units kept the base
key, giving silently garbled output with no error at all.
The correct anchor turns out to be a DISC fact, not title data: an AACS 2.1 disc
names its forensic feature BDMV/STREAM/<clip>.fmts, and carries one
IndividualSegment.tbl, so the SPNs are in that one clip's byte space. A new
forensic_clip_extents() finds the unique .fmts in the already-walked UDF tree, and
those extents now drive the segment arithmetic, the addressability filter, and the
index probe — whose title parameter is gone, since its reads were mis-anchored too.
"Does this title carry forensic content" is now "does it read the forensic clip's
sectors" rather than "do the segment bytes land somewhere in the concatenation".
Where the clip is NOT identifiable — no .fmts, or several, making the SPN space
ambiguous — on a disc that does carry a non-empty table, the resolve now fails loud
with FmtsKeyMissing rather than guessing an anchor. That is a deliberate behaviour
change: a hypothetical disc with two .fmts clips hard-fails where it previously
produced a possibly-wrong map. Failing loud beats silently garbled output, and
inventing an anchor was not acceptable.
This site had been flagged independently three times — by the agent that added the
FMTS per-disc memo, by the round-4 correctness lens with a concrete scenario, and
by the round-4 conformance pass.
Verified red here independently: reverting single_base_key_slot to count the whole
pool fails both new tests. The agent's own evidence was probe_reads 48 vs 40 (the
8 extra sampling reads) and an E7013 DecryptFailed whole-disc abort, and for the
anchor an E7026 FmtsKeyMissing on a [trailer, forensic] extent list.
All six pre-existing FMTS tests pass unchanged through the new anchor, including
the exact-cost assertions (40 probe reads, one key-service call per disc, one UDF
walk for 60 titles), so the round-3 memoisation wins are intact.
Two defects in fixes landed the same day, both found by round 4 auditing round 3's
work rather than trusting it.
**The E-AC-3 grouping ignored substreamid, so the timeline still doubled.**
Confirmed independently by the correctness and conformance lenses and verified by
hand: substreamid appeared only in test helpers, never in the production path. Per
ETSI TS 102 366 (A/52) Annex E a frame set is independent substream 0 — mandatory,
always first — with its dependents, then the OPTIONAL additional independent
substreams 1..7 with theirs, all covering the same time period. Treating an
additional independent substream as a new access unit advanced the clock a second
time for the same 32 ms, which is exactly the doubling the grouping fix existed to
prevent. No fixture caught it because every fixture used substreamid 0.
is_dependent_substream becomes substream_role -> Starts | Extends: strmtyp 1
extends; strmtyp 0/2 with substreamid != 0 now extends (this was the bug); strmtyp
0/2 with substreamid == 0, legacy AC-3, and reserved strmtyp 3 start. Reserved 3
starts regardless of its id bits, because its BSI layout is undefined so those bits
cannot be trusted — an unknown frame is neither merged into an unrelated programme
nor silently discarded.
The frame set stays ONE sample rather than being split into a separate track for
the associated service, and the reasoning is in the module doc: a substream
numbered 1..7 with no substream 0 is not conforming, so extracting one would mean
renumbering ids and rebuilding frame sets — a transcode, not a remux. Programme
selection is the player's job.
A stream joined mid-frame-set (first sync is substreamid 3) is skipped with a debug
and resyncs at the next id-0, mirroring the orphan-dependent rule: its mandatory
id-0 substream was never seen, so it is neither decodable alone nor timeable.
MAX_AC3_BUF 128 KiB -> 1 MiB, because an AU is now a whole frame set: worst case 8
independent x 9 substreams x 8192 B = 576 KiB, which the old cap could have dropped
mid-hold.
**The forced probe's two round-3 fixes cancelled each other.** CHUNK_SECTORS = 1023
exists (with a const assert) so every read starts on a 3-sector AACS aligned-unit
boundary; the short-read fix advanced by actual bytes, making the advance a
non-multiple of 3. Every later read was then misaligned, DecryptingSectorSource
refused it before reading, the stop became ReadFailed, and no verdict was asserted
— so content-based forced detection silently fell back to the vendor label on
exactly the encrypted discs the 1023 change was written for.
A partially-satisfied read now advances only by whole aligned units and re-reads
the <=2 residue sectors from the next boundary, feeding only the aligned prefix so
nothing is double-fed and no partial unit reaches the parsers. A read that fully
satisfies its request still advances by all of it. When less than one aligned unit
comes back the bytes are fed and the same LBA is retried twice before stopping, so
a starved source cannot spin — verified by raising the retry limit and watching the
test hang.
Verified red independently here: reverting substream_role to strmtyp-only fails
eac3_additional_independent_substream_stays_in_the_frame_set (6 access units where
3 are correct — the doubling, literally) and the mid-frame-set resync test.
Reported, not fixed: dec3_box still hardcodes num_ind_sub - 1 = 0 and
num_dep_sub = 0, so it under-declares any stream carrying additional independent
or dependent substreams now that frame sets arrive whole. DolbyConfig has no
fields for either; a real fix needs the parser to surface observed substream
counts. That file is another lens's this round.
Unverified: no real multi-programme DD+ stream exists here, so defect 1 rests on
synthetic Annex-E fixtures. The retail DD+ check (No Time to Die, all substreamid
0) confirms single-programme discs are unaffected.
**The session panics are reachable, and my earlier triage of them was wrong.**
`DiscSession::scan` and `resolve_keys` both did
`self.drive.as_mut().expect(..)`. I previously downgraded these to LOW on the
grounds that no shipped consumer calls them after the drive has been staged into
the reader slot. That is the wrong test: `stage_drive_as_reader` is a PUBLIC
method that empties the drive slot, so the public surface permits the sequence,
and a library must not panic from public API regardless of what current callers
happen to do. Both now return Error::DeviceNotReady.
**A shipped doc comment cited a third-party source FILE** as the authority for
the CLPI ProgramInfo layout ("Layout per the BD CLPI spec clpi_parse.c"). Now
cites the Blu-ray Disc Read-Only Format Part 3 CLIPINF specification.
**The CHANGELOG justified a muxer decision by naming a commercial competitor**
("MakeMKV's rip of the same disc omits it", "matching MakeMKV"). Reworded to
stand on its own terms: the element is optional in RFC 9559, nothing requires it
for interlaced SD, and the 40 ms DefaultDuration is the frame rate the source
actually carries.
The leak gate is extended for both new classes — third-party `*_parse.c` /
`*_dec.c` / `*_demux.c` style filenames, and a competitor named as authority.
Narrowing that rule took two attempts, which is worth recording. A bare
`\.(c|cpp|cc)` pattern produced eight false positives: this repo has its own C
shim (`macos_shim.c`) that build.rs and the docs legitimately reference, and the
pattern also matched the Rust field access `p.cc`. It now matches only the
suffixes typical of third-party media-library sources. This is the second
false-positive round on this rule — the first flagged ffmpeg INVOCATIONS in the
test harness — so the lesson is that a hygiene pattern needs testing in both
directions before it lands, exactly like any other code.
detect_rate's nearest-match fix, the colr HLG/BT.470 fix and their four tests
were silently reverted. Cause: the r3fix-silent worktree was cut BEFORE the
conformance commit landed, and I landed its work by copying whole files into the
main tree. mp4/mod.rs was in both agents' file sets, so silent's copy — built on
the older base — overwrote the conformance changes wholesale. The gate stayed
green throughout, because reverting a fix and its tests together is perfectly
consistent.
Re-applied the conformance commit's diff for that file with a three-way merge;
both agents' changes to mp4/mod.rs now coexist (final_report, UndescribableAudio
and the max(track_id) id fix are all still present alongside RATE_TOLERANCE_FPS
and the colr resolver delegation).
Only mp4/mod.rs was affected. mp4/audio.rs and mkv.rs were in no other agent's
file set and were intact.
Process lesson, recorded because I would otherwise repeat it: NEVER land a
parallel agent's work by copying whole files, when its worktree was cut at an
older base than HEAD. Apply its DIFF (git apply -3), or rebase its worktree
first. Copying files silently discards anything committed to those files in the
interim, and no test can catch it because the tests disappear with the code.
The Windows UI needs current winsafe, whose real minimum is 1.89 (its manifest
under-declares 1.87 while it uses NonNull::from_ref). Rather than stop at the
minimum, this goes to current stable and fixes what that costs.
The counter-intuitive result: 1.97 is CHEAPER than 1.89. libfreemkv had 54
clippy errors at 1.89 and 6 at 1.97, because clippy tightened the noisy
collapsible_if lint in between. Stopping at the minimum would have been the most
expensive choice available.
Roughly 47 lints across the eight repos, the large majority auto-fixed:
libfreemkv 6, freemkv-engine 14, bdemu 8, freemkv-keysources 7, autorip 6,
freemkv-unlock 3, freemkv-i18n 3. The hand-fixed ones are a descending sort to
sort_by_key(Reverse), four manual checked-division sites, a loop counter replaced
by enumerate, and a loop whose first let-else became a while-let.
Worth recording for whoever bumps next: clippy is MSRV-AWARE. Those 54 lints only
appear once the crate DECLARES 1.89 or later, because let-chains become
available. A bare `cargo +1.89 clippy` against a manifest still pinned at 1.87
reports clean and is meaningless — gate with the real precommit script, which is
also the only thing that covers build scripts.
The pin still sits below the Mac default, so it keeps doing its job: catching
lint drift locally before CI sees it.
All 21 findings held up under verification; 15 fixed here, 6 deferred to files
another agent held this round, 0 rejected.
**A defect in my own round-2 probe fix.** CHUNK_SECTORS was 1024, and
1024 % 3 == 1 — verified — so every chunk after the first was misaligned against
the 6144-byte AACS aligned unit and would be REJECTED by
DecryptingSectorSource's alignment gate. On an encrypted disc the forced-subtitle
probe I added last round would have read almost nothing past its first chunk.
Now 1023 sectors (341 aligned units) with a const assertion that fails the build
if it stops dividing, plus set_unit_base per extent so the source's gate is
anchored where the extent actually starts.
**The same probe skipped sectors on a short read**, advancing by the REQUESTED
count rather than the bytes actually returned, so a partial read silently left a
gap in the middle of the evidence. It now advances by n/SECTOR_BYTES and clamps n
to the buffer.
**Its cache key omitted the PGS PID set**, so a playlist declaring an extra
subtitle PID got another playlist's verdict for a track that had never been
probed. And the key was the whole extent list, so partial clip sharing missed
entirely. Both fixed by keying (start_lba, sector_count, pid) — and per-extent
keying was shown SOUND rather than assumed: ForcedTracker is two monotone
booleans, so per-extent evidence composes by field-wise OR, order- and
grouping-independently. Making that honest required per-extent demux state, so an
extent's evidence comes only from its own bytes, and memoising only extents whose
read reached a designed stop.
**A reachable panic in the timeline.** mkvstream::parse_block accepts a
TimestampScale up to i64::MAX, so a video frame can set high_ns = i64::MAX and the
next passive frame panicked adding the backstep. In release it wrapped negative
instead, firing the straggler clamp for essentially every passive frame — audio
and subtitles rewritten onto the wrong point of the output timeline. All four
sites saturate.
**A public constructor divided by zero**: PrefetchedSectorSource::new_with_events
with unit_align == 0. Now InvalidInput, matching its batch_sectors sibling.
**Two Debug impls printed key material.** DiscInputs (volume_id, mkb, unit_key_ro,
samples) and UnitKeyFile both derived Debug. Nothing logs them today — fixed as
prevention, because the next tracing::debug! someone adds is the leak. A doc claim
that DiscInputs "contains no secrets" was false and is corrected.
**An env-var multiply could overflow** in file_sector_source; now bounded at 64 GiB
like its writeback sibling, with the parse split out so the bound is testable
without touching process env.
**The mp4 demuxer allowed one sample per file byte** — ~64x RAM amplification.
Now file_len/16, since only vide/soun tracks are indexed and the shortest legal
AC-3 frame is 128 bytes.
**Two pipeline concurrency defects**: a consumer apply() error was invisible to the
producer, and abandon/finalise had a TOCTOU where a caller could report an
unfinalised output. Both fixed with compare-exchange state rather than a bool.
**Two per-frame copies removed**, both MEASURED rather than reasoned: the AU
assembler now hands its allocation to the frame (same pointer, unchanged capacity,
proven by asserting the pointer) and tsmux reuses one Annex-B buffer across
frames. Both keep capacity deliberately — a naive split_off would have cost more
than it saved.
**A comment pointed at the wrong file** for a mirrored constant; the mirror is now
compiler-enforced with a const assertion converting 90 kHz ticks to ns, so drift
fails the build.
Deferred to another agent's files, all confirmed: detect_rate's fractional-twin
snap, the mp4 reserve's u32 truncation, round_up_grain's overflow, the quadratic
base-key gap fill, and MkvStream's frame cap counting frames rather than bytes.
Every fix verified red by reverting it. Also noted for later:
DecodeSampleSet still derives Debug over multi-MB of on-disc ciphertext.
All six confirmed against the code. The governing rule this cluster serves: a
lossy or degraded outcome is never silent, because a corrupt rip the user does
not know about is the worst failure available.
**A 3D MKV re-mux silently lost one eye.** The BlockGroup read path had arms for
BLOCK / BLOCK_DURATION / REFERENCE_BLOCK only, so BLOCK_ADDITIONS fell into the
skip arm — while the writer does emit BlockAdditions > BlockMore > BlockAdditional
for the MVC dependent view. Reconstruction was judged out of scope and the
reasoning is recorded: PesFrame has no side-payload field and the header parser
never reads BlockAdditionMapping, so there is no dependent-view track to route the
AU to. Instead the loss is now LOUD — counted in bytes and events, warned once,
and surfaced through MkvStream's errors()/lost_bytes(), which the driver already
samples into MuxOutcome. One detail in the finding was wrong and is corrected: the
re-mux does NOT still advertise the mvcC mapping, because the header parser
ignores that element, so the output is a plain 2D H.264 track.
**An all-titles rip silently skipped real titles.** The header-buffer-cap
overflow returned Error::MkvInvalid, and is_skippable_title_stub matches exactly
E_MKV_INVALID | E_CSS_KEY_MISSING — verified here — so a 512 MiB-of-frames title
was classified as an empty nav/menu PGC stub and dropped. It now has its own
E9051 / MuxHeaderBufferExceeded { bytes }, outside the skippable set.
**The public pre-mux report contradicted the file.** Mp4Sink::finish() drops an
audio track it cannot describe, which I chose last round over failing an export
whose video is fine — but mp4_fit_report still listed that stream as included, so
the application's plan and the actual output disagreed. Fixed at both levels:
Mp4SkipReason is now non_exhaustive with NoSamples and UndescribableAudio,
Mp4Sink::final_report() describes the FILE rather than the plan, and for the
boxed dyn Stream path a defaulted Stream::undelivered_streams() carries the
information out to MuxOutcome::undelivered_streams with a driver-side warn.
**MP4 track ids could collide.** ids were assigned before the retain that drops
sample-less tracks, while next_id came from the post-retain count, so [1,3]
yielded next_id 3. Now max(track_id) + 1, saturating.
**Stream selection silently skipped its codec_privates prune** when the lists were
not the same length — but codec_privates is consumed POSITIONALLY and trailing
extras are documented as benign, so the length-equality guard was itself the bug.
The prune now runs unconditionally by index.
**The m2ts_mux scaffolding armed params_written on both the absent and the
unparseable codec_private arms** — the same defect already fixed in tsmux.rs.
Split into params_attempted (a latch, since retrying identical bytes cannot help)
and params_emitted, with a warn on each failure arm and an accessor so the
eventual wiring and its test can observe it.
Each fix verified red by mutating back to the prior behaviour: errors() 0 vs 1,
E6008 vs E9051, final_report [0,1] vs [0], next_track_id 3 vs [1,3], and the
selection prune resolving index 1 to the wrong track's record.
API surface deliberately widened: MuxOutcome gains a public field and Mp4Sink
becomes public. Nothing in-repo breaks. Note a behaviour change on the mkv://
input path — a 3D re-mux now reports non-zero loss, so a consumer treating
errors > 0 as disc damage will trip on it. That is intended: the outcome IS
degraded.
**dec3 declared a 0 kbit/s AC-3 substream.** parse_dolby routes bsid < 11 to
parse_ac3, which leaves data_rate_kbps = 0 and keeps the AC-3 bsid, yet
dolby_sample_entry wrapped that config in ec-3/dec3 for any Codec::Ac3Plus track.
ETSI TS 102 366 Annex F.4 assigns ac-3/dac3 to an AC-3 bitstream and F.6 assigns
ec-3/dec3 to an Enhanced AC-3 one, so the entry now follows the SYNCFRAME that was
actually parsed, not the playlist's codec label. Computing an AC-3 data rate and
keeping ec-3 was rejected: it fixes one field while bit_stream_identification and
num_dep_sub keep misdescribing the stream. The bsid threshold is hoisted into one
constant so parser and entry-chooser cannot drift. Adjacent defect fixed in the
same box: data_rate is 13 bits from a u16 source, so push's mask WRAPPED anything
above 8191 (9000 became 808); it now saturates.
**colr tagged HLG as PQ, and PAL as BT.601.** video_colr carried a second,
drifted copy of the ColorSpace-to-CICP map: transfer 16 (PQ) for every BT.2020
stream with no HdrFormat override, and 6 (BT.601) for Bt470bg. Per ITU-T H.273
Table 3, HLG is 18 and BT.470-6 System B/G is 5 — and mkv::cicp_for_video already
got both right. video_colr now delegates to that shared resolver, so the
duplicated table is gone and cannot drift again. It keeps only its own decision
about WHETHER to emit the box, since an absent colr and an all-unspecified colr
mean the same thing per ISO/IEC 14496-12.
**Exact 24.000 / 30.000 / 60.000 fps was declared 23.976 / 29.97 / 59.94.**
detect_rate took the FIRST STD_RATES entry within 0.5 fps, and every 1000/1001
entry precedes its integer twin 0.024 fps away — a 0.1% error across the whole
track's mdhd and stts. Fixed as nearest-wins rather than by reordering the table:
reordering fixes today's table and re-breaks the moment someone appends a rate,
while nearest-wins is order-independent. Verified red here independently by
reverting to first-match, which fails exactly the two timing tests.
**ddts MultiAssetFlag was set from has_extension.** In the DTSSpecificBox
(ETSI TS 102 114) that flag signals more than one audio ASSET. A DTS-HD MA/HRA
track is one asset whose extension substream carries the XLL/XBR component, so
setting it from "an EXSS sync follows the core" sent a parser looking for a second
asset descriptor while StreamConstruction simultaneously said there was no
extension — the box contradicting itself. This module parses the core header only
and never reads the EXSS asset table, so 0 is the only honest declaration. A
StreamConstruction index for core+EXSS was deliberately NOT invented: that field
is a table lookup that could not be confirmed against the standard, and a wrong
index is worse than an under-declaration.
DTS_AMODE_LAYOUT's masks were independently re-derived against ETSI TS 102 114
§5.3.1 and all 16 are CORRECT — only three adjacent comments were wrong (the
AMODE 2/3/4 annotations were rotated by one, and AMODE 9's said "5.1 with LFE"
when 0x0007 is the 5.0 mask and LFE is OR'd in separately). Comments corrected.
Every one of the eight new tests decodes the field back OUT of the emitted bytes —
data_rate from the dec3 body's leading 13 bits, MultiAssetFlag from bit 48 of the
ddts tail, colr from the nclx payload inside a real stsd, and the frame rate from
mdhd.timescale plus stts.sample_delta of a fully muxed MP4 — rather than
restating arithmetic. This audit has already caught one of my own tests doing the
latter.
NOT fixed, root-caused and recorded at the site instead: an A_PCM/INT/BIG track
ships with no BitDepth, which the Matroska Codec Specifications make a MUST. The
width exists on disc (BD LPCM signals it in the ES header byte 3, DVD in the IFO
audio attribute byte 1) but neither source reaches MkvTrack::audio, and the fix
needs a new AudioStream member plus a deferred setter in files another agent held
this round. Guessing 16 was rejected — it would confidently misdecode every
24-bit disc — as was refusing the track, which would regress the 16-bit majority
that currently plays by accident.
resolve_content_key_map loops every title into resolve_mux_key_map, which called
resolve_fmts_key_map FIRST — before the CpsUnitCache — and on an FMTS disc
returned immediately. So every playlist re-derived facts that belong to the DISC:
a full UDF walk plus /AACS/IndividualSegment.tbl, and on an FMTS disc the anchor
probe, the 32-index phase probe, and a fetch.fmts_indexes round trip.
On a 60-playlist disc, measured on a synthetic fixture: 840 -> 14 metadata reads,
2,400 -> 40 probe reads, and 60 -> 1 key-service calls. Worst case before was up
to 256 probe reads and 32 key-service calls per title. The 60 redundant
key-service round trips are a strong candidate for the keyserver storm seen in
the field.
Two memos behind a pub(crate) DiscKeyCache. The table memo (UDF walk + tbl parse)
is disc-invariant outright — nothing in that path mentions the title — and runs on
EVERY disc, so a plain BD benefits too. Only the deterministic negatives are
memoised as "not FMTS"; a DiscRead fault propagates uncached so a later title
retries.
A blind once-per-disc hoist of the PROBES was rejected as unsafe, and this is the
load-bearing reasoning: the title enters through clip_byte_to_lba, which decides
which segments are addressable and which LBA every probed clip byte reads from, so
two titles with different extent lists probe different physical bytes. A hoist
would serve title B an answer derived from title A's media and could silently turn
a per-title FmtsKeyMissing into a success. The extent list is the ONLY per-title
input, so keying on it is exactly sufficient — matching the ForcedProbeCache
precedent.
Result-identity was proved, not assumed: NEITHER probe reads the key pool.
Verified here independently — probe_fmts_index_keys takes no keys parameter at
all; index keys come from `fetch`, and the anchor's reply feeds the phase probe.
So the pool's growth across titles, the one thing that does change between calls,
cannot move a memoised value, and the result is order-independent. A test resolves
three titles through a shared memo and through fresh memos and asserts both the
per-title ranges and the final key pool (keys, slots, order) are identical.
Not memoised, deliberately: fail-loud FmtsKeyMissing, and any run where an index
hit a read fault — that is a property of a transient drive fault, not of the
extents, and caching it would spread one bad read across 59 playlists.
A fully-memoised title now does zero I/O, which made the old in-loop halt polls
unreachable for it, so a check_halt on entry was added with a test that cancels
after warming the memos.
Also corrects my own overstatement from last round: the CpsUnitCache doc now says
plainly that on an FMTS disc it removes NO reads, because this function returns
before the extent loop ever runs.
Five mutations, all verified red. Pre-existing bug flagged but not fixed:
filter_addressable_segments only checks that a segment's START byte maps to some
LBA in the title, so a play-all playlist can pass the filter while mapping segment
bytes into the wrong clip, whose anchor then returns empty and aborts the sweep.
The AC-3 parser's own module doc stated the assumption: "AC3 frames are
self-contained and always start with syncword 0x0B77". True for legacy AC-3,
false for E-AC-3 above 5.1. Per ETSI TS 102 366 (A/52) Annex E, byte 2 of an
E-AC-3 syncframe is strmtyp(2) | substreamid(3) | frmsiz[10:8], and an access
unit is one INDEPENDENT substream plus every DEPENDENT substream that follows it
until the next independent one. The parser emitted one PES frame per syncframe,
so a decoder saw each dependent substream as a standalone frame with no parent —
including the AC-3-core + E-AC-3-dependent form Blu-ray uses for Dolby Digital
Plus. The extra channels were lost and the timeline ran at 2x.
The bit position is cross-checked against code already in the tree: the existing
frmsiz parse takes byte2 & 0x07 as its high bits, which is only consistent with
strmtyp occupying byte2's top two bits. Legacy AC-3 is excluded by bsid < 11,
where byte 2 is crc1 and reading strmtyp there would be nonsense. Reserved
strmtyp 3 is treated as INDEPENDENT so an unknown type starts a fresh AU rather
than merging into an unrelated one.
The AU carries the INDEPENDENT substream's PTS, and only the independent
substream advances the clock — dependents cover the same time period and add zero
duration. That is what removes the doubled timeline.
A trailing AU that can still grow is HELD across the PES boundary, because the
boundary is unknowable until the next independent sync; the whole AU is re-scanned
next call, so there is no shift and no double-count in the loss tally. Plain AC-3
is never held, which keeps DVD/AC-3 latency and behaviour unchanged.
A latent pre-existing bug surfaced while testing this: a new PES's PTS was
re-stamping an AU that began in an earlier PES, a constant one-frame shift. Fixed
with a PtsAnchor so a PES timestamp applies to the first AU that STARTS in that
PES's own bytes, while a genuine PTS jump is still adopted.
Nine tests. Verified red against five mutations, each killing a specific set:
reverting to the pre-fix behaviour kills 8 while
plain_ac3_frames_are_not_grouped_or_delayed SURVIVES as the no-regression guard —
reproduced independently here. Stamping the dependent's PTS kills 6; not holding
across PES kills 4; holding plain AC-3 too kills 15; neutering the PTS anchor
kills exactly the 2 split-across-PES timing tests.
Three sibling defects found and deliberately NOT fixed, all in mp4/audio.rs:
dec3 hardcodes num_dep_sub = 0 (and a nonzero value changes the box LAYOUT, not
just a field, per Annex F/G); parse_eac3 ignores strmtyp/substreamid entirely;
and a 7.1 DD+ track is still labelled 5.1 because the channel count comes from
the independent substream while the extra channels are described by the
dependent's chanmap, which nothing parses.
Not verified: no real E-AC-3-with-dependents sample exists here, so all evidence
is synthetic frames plus the spec layout. The multi-independent-substream case
(num_ind_sub > 1, main + associated audio in one PID) is deliberately treated as
one AU per independent substream and is untested.
Round 3 audited the round-1/2 fix commits rather than trusting them, and found
three defects in that new code. This is why the pin moves each round.
1. LICENCE REGRESSION, and it was mine. Reverting the "distinguish a failed key
source" commit also restored a verbatim reference-decoder table citation in
src/mux/codec/dts.rs, because both changes were in that one commit. The MIT
licence cleanup was silently undone at HEAD and nothing caught it.
The citation is replaced with ETSI TS 102 114 §5.3.1 again, and — more
importantly — the rule now lives in the leak gate instead of in my memory.
scan-secrets.sh gains LICENCE_RE, which flags ff_dca*, dcadec, l-smash,
libav*, and bare ffmpeg/FFmpeg as REF-IMPL-CITATION. `no ffmpeg` is
explicitly allowed via negative lookbehind: stating what this project does
NOT depend on carries no risk and is a genuine selling point. Verified by
re-introducing the citation (gate fails) and removing it (gate clean).
2. TsMuxer armed params_written even when the avcC/hvcC parser returned None, so
a track whose codec_private exists but will not parse was muxed to BD-TS with
no VPS/SPS/PPS ever emitted — undecodable video, reported as success, with no
log line. Last round's fix corrected WHICH parser is used and left this half
untouched. Arming the flag is still right (retrying identical bytes cannot
succeed) but it is no longer silent: it now warns with the track, codec and
codec_private length.
3. test_aes_cbc_roundtrip defined a LOCAL fn aes_cbc_encrypt that SHADOWED the
production primitive, so it round-tripped a copy of the algorithm against
itself and never touched crypto::aes_cbc_encrypt — the function this cycle
added. Any mutation to the shipped code passed it. The shadow is deleted and
the test now calls the real primitive; verified by mutating
crypto::aes_cbc_encrypt, which now fails it and previously would not have.
The Windows UI needs winsafe, whose current release requires rustc 1.87. The
alternative was pinning winsafe back to an older release, which would bake a
stale API surface into a brand-new UI permanently to dodge one minor version.
The pin's purpose is to sit BELOW the Mac default so clippy drift is caught
locally before CI, not to stay on 1.86 specifically, so 1.87 preserves the
discipline exactly.
Verified before moving anything, not after: `cargo +1.87 clippy -- -D warnings`
and `cargo +1.87 fmt --check` are clean across all eight repos, and the full
precommit gate (fmt + clippy + tests) passes on libfreemkv, autorip, bdemu,
freemkv-engine and freemkv-keysources. Zero new lints, zero formatting drift.
resolve_content_key_map calls resolve_mux_key_map once per title, and on a
multi-CPS disc that path issues 8 random single-unit reads per extent. A disc's
playlists overwhelmingly reference the same few clips — main feature, play-all,
per-chapter and seamless-branch variants — so the same physical extents were
re-sampled from the drive once per playlist. On a 60-playlist / 15-clip disc
that is ~2,400 non-sequential 6144-byte reads, roughly 8 minutes of pure seeking
at 200 ms per seek, before the mux starts. Now ~600 reads.
Keyed per EXTENT — (format, start_lba, sector_count) — rather than per title's
whole extent list, which is finer-grained than the forced-subtitle probe's cache
and strictly better here: a play-all playlist sharing 4 of 5 extents with the
main feature still hits on those 4.
Why a cached pool index is provably identical to a recomputed one, verified
rather than assumed:
* `pick` iterates the pool IN ORDER and returns the FIRST index whose key
decrypts a sample to clean.
* The pool is APPEND-ONLY. Checked across the whole crate: only `push`, with no
insert/remove/clear/retain/sort/dedup/truncate/drain/swap/reverse anywhere.
So appended keys can only land AFTER a matched index, and the first match for
the same samples cannot shift.
* The samples are a pure function of the three values in the key, read from
read-only optical media.
Two outcomes are deliberately NOT cached, which is what makes this safe rather
than merely faster:
* the inherited index (`None if samples.is_empty() => last_idx`) is per-TITLE
state, not a property of the extent — caching it would let one title's
carry-in index leak into another title's clear extent, i.e. a WRONG key;
* the fail-loud DecryptFailed verdict, so a retry after a key source banks the
missing key re-samples instead of inheriting a stale answer.
Halt is still polled before the cache lookup, so cancellation is unchanged.
resolve_mux_key_map keeps its exact signature and delegates with a fresh cache,
so there is no public API change. ContentFormat gains Eq + Hash (additive).
Four tests, and the two mutants that matter both verified red: disabling the
cache short-circuit fails the hit and recompute-equivalence tests, and wrongly
caching the inherited index fails multi_cps_inherited_index_is_not_cached.
Every BlockGroup frame back-patched its element size via ebml::end_master, which
does two stream_position() calls and two real seeks. BufWriter does not override
Seek::stream_position, so each position query is seek(Current(0)) = flush_buf +
lseek — and each of those flushed the 4 MiB BufWriter while every position-moving
seek reset WritebackPipeline::last_flush_pos. The buffer never got to do its job.
An MPEG-2 title takes this path for EVERY frame (the parser stamps a per-frame
duration, so I, P and B all become BlockGroups) — roughly 350,000 per feature.
A BlockGroup's size is knowable before writing, so there is no need to back-patch
at all. New seek-free twins start_master_buf / end_master_buf patch a placeholder
by buffer INDEX instead of file offset, and build_block_group assembles the whole
element into a persistent buffer that write_block_group and its MVC sibling take,
fill, write once, and hand back — including on the error path — so the allocation
is made once rather than per frame.
Measured with a counting writer that, like BufWriter, does not override
stream_position, over 200 frames:
seek calls position-moving
plain before 912 451
plain after 112 51
MVC before 2516 1253
MVC after 116 53
Per-frame cost is now zero; the residual is the header, the per-cluster
back-patch and Cues. For 350k BlockGroups that is 1.4M seek calls removed on the
plain path, 4.2M on an MVC title.
end_master is deliberately NOT changed for its other callers. The Cluster master
genuinely streams — frames are appended to an open cluster over time, so its body
cannot be buffered without holding a whole cluster in memory — and the rest
(EBML header, Tracks, Info, Chapters, Cues) run once, not per frame.
Byte-identity is the safety property, and it is structural: end_master always
patches a FIXED-width 8-byte VINT, and the buffered pair writes and patches
exactly that same placeholder, so the encodings cannot differ. Verified by
capture-then-compare over 200 frames (17 keyframes / 183 non-keyframes, multiple
clusters, BlockDuration present and absent, both reference branches) — output
byte-for-byte identical.
Three tests now pin it permanently: buffered vs seeking output byte-for-byte for
empty/tiny/multi-byte bodies, the same for NESTED masters (the MVC path nests
BlockAdditions > BlockMore, where an index-arithmetic slip would surface), and
end_master_buf erroring rather than panicking on a position outside the buffer.
All three verified red against a deliberately divergent placeholder width.
A note on verifying this kind of change: comparing emitted MKV across two
worktrees at different commits shows a spurious 14-byte difference, because
MuxingApp/WritingApp embed the build's git SHA twice. Compare at the same base.
probe_and_set_forced broke out of its read loop on a read error but still
applied whatever partial observation it had accumulated as an authoritative
verdict, overwriting the vendor-label-derived forced flag. A disc that faults
early could have a correct flag replaced by a guess from a fraction of the data.
The file already got the zero-observation case right — it deliberately leaves
the vendor flag alone rather than "assert not-forced from having seen nothing".
The defect was that a PARTIAL observation cut short by a fault was treated as
complete.
The fix rests on the two verdicts not being symmetric evidence.
settled_not_forced() is POSITIVE evidence — a non-forced display set was
actually seen, and no unread data can retract it. is_forced() is an ABSENCE
claim — display sets were seen and none was non-forced — which is only sound if
the read got far enough for the absence to mean something.
So every loop exit now yields a named StopReason, and absence claims are
asserted only for a designed stop:
* Exhausted / Budget → conclusive. The budget is deliberately conclusive: the
natural exit is "every track settled not-forced", which a genuinely forced
track never satisfies, so the budget is the ONLY path by which a real forced
verdict is ever reached. Treating it as inconclusive would disable forced
detection entirely.
* Halted / ReadFailed → inconclusive. A cancelled probe's cut-off point is as
arbitrary as a faulted one, so its absence claim is worth no more.
Evaluated per track, matching the existing observed() gate: a track that already
saw a non-forced set keeps its sound verdict even on a truncated run, while a
forced-so-far sibling keeps the vendor flag.
An inconclusive run is also NOT memoised. The cache key is the extent list and a
disc's playlists share clips, so caching a truncated run would replay one read
fault onto every playlist referencing those extents and deny any later title the
chance to re-read them.
Four tests; three of them verified red by forcing absence_is_conclusive() back
to always-true (the old semantics), while the budget test correctly stays green
either way — confirming the guard was not over-corrected.
Also moves the function doc comment back onto probe_and_set_forced; the earlier
probe commit left it attached to the ForcedProbeCache type alias.
The `Ok(_) | Err(_)` arm in resolve_and_apply_traced conflates "this source
had no entry for the disc" with "this source failed", and reports both as
KeyNode::NoEntry. An operator whose key server is returning 502s is therefore
told their disc is not in the database.
The conflation is real but LATENT, and fixing it here would change nothing an
operator can see, because no shipped KeySource ever returns Err:
KeydbSource::get_unit_keys maps a load/parse failure to Ok(Vec::new()),
OnlineSource::get_unit_keys is Ok(self.query(ctx)) where query returns empty on
transport error, HTTP status, oversize body and bad JSON alike, and MultiSource
discards inner Errs. Only test doubles return Err. FetchOutcome::errored in
drive_unit_keys / drive_fmts_indexes is dead for the same reason — the right
contract, honoured by no source.
autorip already works around the missing signal by re-probing the service over
HTTP (probe_online_reachability / key_service_transient_status), and its own
comment names the incident: "the online keysource swallows every failure
(transport error, 502, timeout)".
So the fix belongs at the source boundary in freemkv-keysources, with
Disc::aacs_error as the channel the operator actually reads — not in this
trace. Documented here so the next reader does not assume the arm works, and
does not "fix" a dead path as I nearly did twice.
This crate is MIT licensed. Comments citing another decoder's internal symbols
and reproducing its tables verbatim create licence risk that no engineering
benefit justifies, so every such citation is replaced with the primary source:
ETSI TS 102 114 §5.3.1.
Eleven sites across src/mux/codec/dts.rs and src/mux/mp4/audio.rs. The technical
substance is unchanged in every case — the deficit-sample-count semantics, the
reserved-field skips, the invalid LFF value, and the 16 legal AMODE codes are all
spec facts and are now attributed as such. Four CHANGELOG entries that named a
validator are reworded; the "no a reference decoder" dependency claim stays, since stating
what this project does NOT depend on carries no risk.
I initially argued this was a false positive on the grounds that the project's
hygiene rules name internal infrastructure and reverse-engineering material, not
open-source citations, and that a channel-count table from a standard is fact
rather than expression. That reasoning missed the point: the exposure is MIT
distributing text derived from GPL/LGPL sources, and that is the maintainer's
risk to weigh, not mine. Reversed in full.
resolve_and_apply_traced collapsed `Ok(_) | Err(_)` into a single
KeyNode::NoEntry step, so a key source that FAILED — server unreachable, keydb
unreadable, malformed entry — was recorded identically to one that simply had no
entry for this disc. The front-end renders that trace, so it told the operator
their disc is not in the database when the real cause was a fixable
infrastructure problem. drive_unit_keys and drive_fmts_indexes were refactored
this cycle to preserve exactly this distinction; this path had not been.
KeyNode gains a SourceFailed variant and the two arms are split. freemkv's
trace renderer matches KeyNode exhaustively with no catch-all, so its arm is
added in the same change — otherwise the consumer would not build.
Also made ETSI TS 102 114 the primary authority for DTS_AMODE_COUNT's comment
rather than a reference decoder internal symbol, and pointed it at this
crate's own cross-checked DTS_AMODE_LAYOUT / DTS_AMODE_CH tables.
A round-2 finding asked for every a reference decoder and a reference decoder citation in the DTS parser
to be stripped as a public-repo hygiene violation. Rejected: the project's rules
(scan-secrets.sh, CLAUDE.md) prohibit internal infrastructure references and
reverse-engineering material, and a reference-decoder citation is neither. The
AMODE channel-count table is a factual table from the standard, not expression
copied from an implementation. Citing the spec plus a corroborating
implementation is how a decodability gate should be justified.
The ReferenceBlock offset was computed for ANY video track, but the keyframe tick
it measures against was recorded in a single global slot gated to the PRIMARY
video track. On a title with two video tracks — an MVC base plus secondary view,
or a multi-angle disc — a secondary track's non-keyframe therefore referenced a
keyframe on a different track, or 0 (a self-reference) when the primary had not
produced one yet. The tick is now recorded per track, so a non-keyframe can only
reference a keyframe on its own track.
The faststart moov-hole estimate modelled every audio track as (E-)AC-3 at 1536
samples per frame. 1.6.0 added DTS to the writer's carried set, and a DTS core AU
is commonly 512 samples — a third of that — so a DTS track's sample table was
under-reserved threefold and the mux fell back to moov-at-end, losing faststart
on exactly the files 1.6.0 newly supports.
mvc_frame_emits_blockgroup_additional_and_reference asserted only that the
non-keyframe's ReferenceBlock was Some(_). Its non-MVC sibling, added in the same
commit, pins the exact offset; this one now does too, so a mutant emitting a
constant or wrong-signed offset no longer passes.
The comment above the mp4 sample budget claimed file_len stops a crafted file
inflating allocations "past the file's own size". Each indexed sample costs ~52
bytes, so the real ceiling is ~52x file_len (still capped by MAX_SAMPLE_COUNT).
The bound is real; the comment overstated how tight it is.
StreamSelection::apply validated a listed PID by scanning ALL streams, so a PID
named in the wrong class's filter passed validation — an audio filter listing a
subtitle PID, say. `keeps` then matched it against the audio streams only, so
the requested track was silently absent from the output. That is precisely the
outcome this validation documents itself as preventing: "fail loud rather than
silently emit an MKV missing a requested track".
Each filter is now checked against its own stream class. The `listed_pids`
helper existed only for the cross-class scan and is removed rather than left
behind as dead code.
Test covers both directions plus the sanity case, and asserts a rejected
selection leaves the title unpruned.
The NAL path called length_prefixed_to_annex_b, which allocates a whole-frame
Vec of its own, then copied the result into a second whole-frame Vec — two
full-frame allocations and two full-frame copies per video frame. The crate
already has append_length_prefixed_as_annex_b, which writes the conversion
straight into a destination buffer; it is the same code path with the
intermediate removed.
The destination is also sized once up front instead of starting from Vec::new(),
which re-grew from zero capacity inside every conversion.
On a UHD HEVC title muxed to m2ts:// — ~200k video frames averaging ~310 KB of
ES at 60 Mb/s — that removes roughly 62 GB of allocation and 62 GB of memcpy.
Behaviour is unchanged: the existing tsmux conversion tests, including the
non-NAL passthrough and Annex-B default pair added last round, all still pass.
A first attempt reused a persistent scratch buffer across frames, which does not
work: the buffer is handed out as Cow::Owned and so can never be returned. A
right-sized single allocation gets most of the win without restructuring the
function around the borrow.
ddts CoreSize wrapped to zero on a maximum-size core. core_size is FSIZE + 1 and
FSIZE is itself 14 bits, so the maximum is 16384 — one past what the 14-bit
CoreSize field holds — and push()'s mask turned that into 0, declaring an empty
core frame. Clamped to 16383 instead: one byte short beats telling a decoder
there is no core. Proven red first (the field read back as 0).
ddts avg/max bitrate under-declared every non-integral frame rate. It computed
sample_rate / frame_samples first, so a 512-sample core at 48 kHz truncated
93.75 frames/s to 93. Multiplying before dividing, with round-to-nearest, keeps
the precision.
set_decrypt_threads skipped the pool swap on a poisoned lock while
DECRYPT_THREADS had already been updated, so the new thread count was reported
as taking effect while the stale pool kept serving. decrypt_pool() deliberately
recovers from poisoning for exactly this reason; the setter now does the same.
The pool Arc is immutable once stored, so a prior panic cannot have left it
half-written.
The CoreSize test decodes the value back out of the emitted box rather than
restating the clamp — a first draft asserted the clamp arithmetic against
itself, which would have passed against the unfixed writer.
Two defects in the encrypt_unit promoted to public API last round, both found by
round 2 auditing that new code.
It returned silently without encrypting when the slice was shorter than
ALIGNED_UNIT_LEN. Its own contract requires the caller to set the container's
encrypted flag BEFORE calling — the header is the key seed — so a silent no-op
leaves a unit advertised as encrypted while still carrying plaintext, with
nothing for an authoring caller to check. It now returns bool and is
#[must_use], so ignoring the refusal is a compile-time warning; every call site
was updated to assert on it.
bool rather than Result deliberately: a wrong buffer length is a programming
error at a library boundary, not a disc condition, and a new Error variant would
mean a new numeric code plus its rendering in another repo.
It also drove CBC from the single-block aes_ecb_encrypt, rebuilding the AES key
schedule for each of the 383 blocks in a unit — an order of magnitude slower
than its inverse, which expands the key once via aes_cbc_decrypt. The missing
counterpart aes_cbc_encrypt now exists alongside it, and encrypt_unit calls it,
so the two directions are symmetric in structure as well as in result. For an
authoring caller encrypting a 90 GB image that removes ~5.6 billion redundant
key expansions.
New test pins the boundary: ALIGNED_UNIT_LEN - 1 returns false and leaves the
buffer byte-identical, ALIGNED_UNIT_LEN succeeds. The existing round-trip and
padding-asymmetry tests still pass, so the CBC rewrite is provably the same
transform.
All six describe code that does something different from what they say, which
is the class of defect that gets a maintainer to write a bug on purpose.
docs/clpi.md presented the CLPI stream-PID entry as byte-aligned 2/2/2/4/4-byte
fields with a 32-bit fine-entry count. It is one 80-bit packed block —
reserved(10) + EP_stream_type(4) + num_EP_coarse(16) + num_EP_fine(18) +
EP_map_start_address(32) — and num_EP_fine is 18 bits. Anyone parsing to the
doc's offsets would read garbage. Replaced with the real bit layout.
docs/udf.md said read_directory()'s recursion cap is 3; MAX_DIR_DEPTH is 8.
TROUBLESHOOTING.md called Pass 1 `recovery::copy`. The engine's `sweep` is
documented as "Pass 1 of a multipass rip"; `copy` is the dispatch verb that
chooses between sweep and patch. This inconsistency was mine, introduced in the
1.6.0 doc rewrite. docs/drive-access.md already said `sweep` and was right — a
round-2 finding claimed the opposite on the grounds that `recovery::sweep`
appears nowhere else in this crate, which it cannot, being in another crate.
io/pipeline.rs cited `disc::patch` as WRITE_THROUGH_DEPTH's caller; that moved
to freemkv-engine in 1.6.0 and no `patch` exists here.
truehd.rs's doc on mlp_major_sync_crc_ok said the trailer is compared
big-endian while the body compares u16::from_le_bytes — and a big-endian
compare was the bug the function was fixed for, so the comment described the
defect rather than the code.
sector/decrypting.rs claimed the decorator owns "the only mutable state (its
call-count cap and spent flag)". DecryptingSectorSource has no such fields and
no KeyFetch field at all in this revision.
Three defects around MuxOptions in the mux driver.
MuxInput::Live never applied MuxOptions.selection, so a caller's audio/subtitle
selection was silently ignored on the live-drive path while the field's own
documentation said it was applied before the demux pipeline is built. The Iso
and Session arms both apply it; Live now does the same, in the same place —
before resolve_inline_base_map, which is keyed on extents and so unaffected by
pruning the stream list.
The header gate returned Error::MkvInvalid whenever headers had not resolved.
On the prefetch-highway path a halt landing while the pump is blocked in a read
can end the stream as Ok(None) rather than Err(Halted), so the loop breaks with
headers unresolved through no fault of the data. Reporting that as MkvInvalid
tells the consumer its disc is malformed and skips the stop-preserves-staging
path that a clean completed=false triggers. The gate now re-checks halt first.
MuxOptions.selection's doc claimed it was applied without naming the one input
it is not applied to. That exception lived only in an internal comment at the
Url match arm, where a caller reading the public field docs would never see it.
It is now on the field, pointing at InputOptions::selection instead.
parse_audio_attr mapped DVD audio_coding_mode 2 to Codec::Mpeg1 — the MPEG-1
VIDEO variant. Codec::kind() reports Video for it, so a DVD MPEG-audio stream
was classified and handled as video everywhere downstream. Modes 2 and 3 are
both MPEG audio Layer II (3 adds the MPEG-2 multichannel extension), so both
map to Codec::Mp2. A test now walks every coding mode and asserts each result's
kind() is Audio, so no mode can map to a non-audio codec again.
docs/aacs.md documented an entire keydb-resolving API that does not exist:
ScanOptions::with_keydb, Disc::open_title, reader.read_unit(). None of those
symbols appear anywhere in the crate, and ScanOptions has no keydb field — its
own doc comment says "libfreemkv is lookup-free — it resolves no keys". A
reader following that page would conclude the library reads keydb.cfg, which
inverts the actual design: the caller resolves keys out-of-band through a
KeySource and applies them with Disc::decrypt_with.
The section is rewritten against the real API, and the AacsState table's
`key_source` type corrected from KeySource to KeyOrigin.
Worth recording: the first replacement example I wrote was itself wrong. It
used `input("disc://...")`, which resolve.rs explicitly rejects with
Error::DiscUrlNotDirect — live disc must go through Drive::open + Disc::scan +
DiscStream::new. Every symbol and signature in the committed example was
checked against the source rather than assumed.
Mp4Sink::write returned Ok(()) without recording the sample whenever an audio
track's frame would not parse into a sample entry. Two consequences, both
silent: leading audio frames were lost until one frame parsed, and a track
whose frames never parsed disappeared from the output entirely — finish()'s
retain() removed the sample-less trak and the run reported success. That
contradicts this crate's stated policy that a skipped track is never silently
dropped.
The drop was never necessary. audio_entry is read in exactly one place,
build_trak, reached only from build_moov inside finish() — nothing on the write
path consumes it. So write() now records every sample and derives the entry
opportunistically from whichever frame parses first.
That makes build_trak's `audio_entry.unwrap_or_default()` reachable, which
would emit an stsd declaring entry_count=1 around an EMPTY sample entry: a
structurally invalid mp4 returned as success. finish() therefore drops any
audio track it cannot describe, with a tracing::warn! naming the codec and
sample count.
Dropping rather than erroring is deliberate. It matches finish()'s existing
treatment of sample-less tracks, keeps an export whose video is fine from
failing outright, and needs no new error code — a new code would mean a new
i18n key across 29 locale files in another repo, which is not this change's
scope. The track's bytes stay unreferenced in mdat: wasted space in a valid
file, which is the cheaper failure.
Test pins both halves — moov describes only the video track, and the
unparseable audio bytes still reach mdat rather than being discarded at write
time.
Three defects in the mp4:// read path, all of the same family: a damaged
source was remuxed minus a track, or with fabricated data, and the run
reported success.
Silent drops. Eight paths dropped a whole track on malformed input with no
report of any kind, so an mp4:// source missing its audio looked like a clean
run. Each now emits a tracing::warn! naming the track and the missing or
inconsistent table (tracing English is permitted in this crate; the numeric
error codes are unchanged). The non-A/V handler case is debug!, since skipping
a timecode or hint track is normal.
Fabricated offsets. sample_offsets ended with a `while offsets.len() <
sizes.len()` loop that packed unplaced samples after the last known offset.
Those samples have no known location, so the invented offsets made the reader
pull frame data from arbitrary file bytes — the exact "emit garbage" outcome
the stco/stsc presence guards refuse. It now returns the short list and the
caller drops the track.
Short stts. `durations.get(i).unwrap_or(0)` gave every sample past the end of a
short stts a duration of 0, collapsing the whole tail onto one timestamp. That
is the same degenerate timing the `durations.is_empty()` guard was written to
refuse, so the guard now refuses both cases.
Two shared test fixtures were internally inconsistent and only passed because
the reader was lenient: stsz declared 3 samples while stsc placed 1, and the
hostile-stsz fixture's stsc/stts covered a single sample. Both are now
consistent. The hostile fixture keeps its lying stsz count — that lie is what
it tests — but its stsc and stts now cover whatever count survives the
file_len bound, so it exercises the allocation bound rather than the
inconsistency guards.
Two new tests pin the new refusals by mutating the consistent fixture: an stsc
that places 1 of 3 samples, and an stts that covers 1 of 3.
The probe's only natural exit was "every PGS track has shown a non-forced
display set". A genuinely FORCED track never satisfies that, so on the common
authoring — a forced-narrative track for foreign dialogue — the loop read the
title's entire extent set at 2 MiB per call with no byte cap, no time cap and
no halt check. It was also invoked once per title rather than once per distinct
clip, and a disc's playlists overwhelmingly reference the same few clips (main
feature, play-all, seamless-branch variants), so the same physical extents were
re-read 30-150 times. The two defects multiplied: tens of GB, times the
playlist count, off an optical drive.
Reached via ScanOptions::probe_forced_subtitles, whose only consumer is
`freemkv info -v` (freemkv/src/disc_info.rs). The rip path leaves it off. So
the symptom is `info -v` never returning on an ordinary UHD, not a corrupt rip.
Three changes:
* PROBE_BUDGET_SECTORS caps a probe at 256 MiB. A forced track's display sets
appear throughout the title, so a bounded prefix classifies it; the budget
only decides how long we keep looking for a non-forced set before accepting
the forced verdict.
* ScanOptions::halt is now plumbed in and checked per chunk, so `info -v` is
cancellable. The probe previously took no halt at all.
* A ForcedProbeCache memoises verdicts against the title's exact extent list.
Keying on byte-identical input means a hit cannot change any result — it
only removes the re-read.
Verdict application is factored into apply_verdicts so the cached and freshly
probed paths cannot diverge.
Three tests pin the behaviour, each against a reader that counts sectors and
never ends: the budget stops at exactly PROBE_BUDGET_SECTORS, a cancelled halt
reads zero sectors, and a second title with identical extents costs no further
reads while a different extent list still misses the cache.
Also worth recording: the CLI acceptance harness never exercised `info -v`
against an optical drive — it reads ISOs from local SSD, where a full-extent
read is fast enough to hide both defects.
MkvTrack::audio's catch-all was `_ => CODEC_AC3`, and ebml.rs defined no
A_AAC, A_MPEG/L2, A_MPEG/L3, A_FLAC or A_OPUS constant at all. A CodecID
names the payload, so any of those codecs was written into the MKV declaring
AC-3 while carrying something else — a player either refuses the track or
decodes noise.
Reachable through two ordinary paths, both verified: ifo.rs:577 maps DVD
audio_coding_mode 3 to Codec::Mp2, so a DVD with MPEG audio muxed to mkv://
produced a track declaring A_AC3 over MP2 bytes; and mp4/read.rs:479 maps the
`mp4a` sample entry to Codec::Aac for an mp4:// source. codec/mod.rs already
has working parsers for MP2, MP3, AAC, FLAC and Opus, so the pipeline carried
these codecs end to end and only the container label was wrong.
MkvTrack::video had the same shape: `_ => CODEC_MPEG2` announced Codec::Mpeg1
and Codec::Av1 as MPEG-2 video. V_MPEG1 and V_AV1 added.
Both catch-alls stay, because MkvTrack::audio/video return Self and have no
error channel, but they are now reachable only by a non-audio/non-video or
Unknown codec routed there in error. Two tests enumerate every real codec of
each kind and cross-check each against Codec::kind(), so a codec added to the
enum later cannot silently inherit another codec's ID.
Both proven red first: Aac declared A_AC3 before the fix.
TsMuxer::write_frame handed every NAL video track's codec_private to
hvcc_to_annex_b. An H.264 track carries an avcC record, whose box layout is
different, so the parser returned None and no parameter sets were emitted —
and params_written was set unconditionally, so it never retried. H.264
muxed to m2ts:// reached the player with no SPS/PPS and was undecodable,
silently: frame_count still advanced and the mux reported success. AVC is
the dominant Blu-ray video codec, so this was not an edge case.
The correct dispatch already existed and was already used by
demux_sink::annexb_param_sets. tsmux simply never got the codec: it knew
only PIDs and a NAL-or-not bool, so it could not tell hvcC from avcC.
Rather than add a second setter, set_nal_video(track, bool) becomes
set_video_codec(track, Codec). One fact decides both the ES framing and the
parameter-set parser, so the two can no longer disagree — and that
disagreement is precisely this defect. The default stays Codec::Hevc, which
is the behaviour the bool's `true` default encoded, so a caller that never
calls it is unaffected.
Proven red first, end-to-end through M2tsStream::create with a real avcC
record: before the fix neither the SPS nor the PPS reached the transport
stream.
Disc::sweep, Disc::patch, Disc::copy, SweepOptions and PatchOptions have
zero occurrences in src/ — recovery moved to freemkv-engine — but they were
still documented in 30 places across README.md, TROUBLESHOOTING.md, six
files under docs/, seven src/ doc comments and a Cargo.toml comment.
README.md is the crate's GitHub front page and carried a full multi-pass
code example that cannot compile.
Two of the src/ references were intra-doc LINKS to deleted items
([`disc::Disc::copy`], [`disc::Disc::patch`] in scsi/mod.rs). They produced
no warning on a normal `cargo doc` only because they sit on pub(crate)
items; `--document-private-items` reports both, and they are gone now.
The README example is deleted rather than rewritten against the engine's
API: libfreemkv documenting a downstream crate's API on its own front page
is the drift that produced this, and it cannot even depend on it. The src/
references become plain code spans naming freemkv_engine::recovery::* —
deliberately not links, for the same reason.
docs/rip-recovery.md was 202 lines about relocated code. It now documents
only what this crate owns — Drive::read, SenseFamily, DiscStream's adaptive
batch halving — plus the read-path design constraints, which belong with the
code that enforces them, and points at freemkv-engine/src/recovery/ for the
strategy. api-design.md's module tree is regenerated from the real src/disc/
and src/drive/ layouts instead of hand-patched; it had listed sweep.rs,
patch.rs, mapfile.rs and read_error.rs, none of which exist.
Three stale facts surfaced while rewriting and are corrected: the read
timeouts are 10 s / 60 s, not the documented 1.5 s / 30 s; Drive::reset and
SgIoTransport::reset no longer exist at all, so "no SCSI reset from any read
path" is now stated as the stronger fact it has become; and verify_title,
listed as a progress-emitting operation, was removed entirely.
CHANGELOG.md keeps its references — those are the historical record of the
releases that shipped the API.
Two tests checked only find_id(&data, ebml::REFERENCE_BLOCK).is_some().
ReferenceBlock's EBML ID is the single byte 0xFB, so that asks whether one
exists ANYWHERE in the output — not which blocks carry one. Inside a
BlockGroup, keyframe-ness is signalled by the ABSENCE of ReferenceBlock, so
"which" is the entire question.
Measured rather than assumed. A mutant emitting a ReferenceBlock on every
BlockGroup, keyframes included — which reintroduces the exact 1.6.0 defect
these tests were added for, every frame reading as a non-keyframe — passed
mvc_frame_emits_blockgroup_additional_and_reference untouched. The
duration-bearing roundtrip test did catch it, via its MkvStream read-back
rather than via the presence check. A mutant corrupting the offset VALUE
was invisible to both.
all_block_groups() parses every BlockGroup in emission order with its
Block flags, relative timestamp, duration and decoded signed ReferenceBlock.
first_block_group() now delegates to it, so there is one walker rather than
two. Both tests assert per block: the keyframe carries NO ReferenceBlock,
each non-keyframe carries one, the offset equals the distance back to the
keyframe, and the SimpleBlock-only 0x80 flag is clear throughout.
Both tests previously took muxer.writer.into_inner() WITHOUT calling
finish(), leaving cluster sizes unwritten — an unparseable file, which is
why they could only byte-scan in the first place. They now write through
SharedWriter and finish(), so the assertions run against a well-formed
Matroska file.
Verified: with the fixes in place, the every-block mutant and the
off-by-one-offset mutant both fail.
set_nal_video had exactly one production caller and zero test callers. The
branch it gates decides whether a video track's ES goes through
length_prefixed_to_annex_b, and MPEG-2 and VC-1 must not: they are already
start-code ES. Getting it wrong is silent — frame_count still increments,
so the mux reports success while emitting a video-less file.
Five tests, each verified against a real mutant:
* non-NAL ES passes through byte-for-byte, and the default path still
converts. Both use a deliberately length-prefix-SHAPED payload so a
wrongly-applied conversion rewrites the leading four bytes into a start
code — a payload the converter happened to leave alone would let a
mutant pass.
* a non-NAL keyframe arms params_written, so following non-keyframes are
not dropped by the pre-keyframe guard.
* set_nal_video rejects an out-of-range track instead of panicking.
* M2tsStream::create wires a VC-1 track to the non-NAL path.
That last one matters more than it looks. The four TsMuxer-level tests set
the flag themselves, so deleting the set_nal_video loop from
M2tsStream::create left all 2366 tests passing — the exact mutant the
finding named went undetected until a test drove the real wiring. It now
also catches the subtler mutant of widening the matches! arm to include
Vc1.
encrypt_unit becomes public library API rather than a #[cfg(test)] helper.
Authoring an encrypted disc image is a legitimate use of this crate, and
the capability was already written four times over: a pub(crate) test-only
copy in aacs/content.rs plus three hand-rolled duplicates in decrypt.rs,
sector/decrypting.rs and disc/extract.rs. All four now call one function,
removing ~110 lines of duplicated cipher code that could drift from
decrypt_unit independently.
It mirrors decrypt_unit's purity contract: crypto only, no encrypted-flag
handling, because where that flag lives is container-specific (CPI bits in
byte 0 for BD-TS, elsewhere for HD-DVD-PS). Callers set the flag BEFORE
encrypting — bytes 0..16 are the key seed left in plaintext, so touching a
header byte afterwards changes the key a decryptor derives. That footgun is
documented at the function and at every call site.
Two tests pin it: an exact round trip through both directions, and the one
place the pair is deliberately asymmetric — decrypt_unit restores
all-zero-on-disc packets to zero, and the test proves an all-zero plaintext
packet enciphers to non-zero bytes so it is never mistaken for padding.
That asymmetry was previously only prose.
Two decrypt tests were also weaker than their own names:
* aacs_clear_trailing_partial_passes_through asserted only is_ok(), so a
mutant corrupting the clear partial while returning Ok passed. It now
snapshots the buffer and asserts byte equality, matching the
none_keys_is_noop pattern already in the file.
* aacs_decorator_decrypts_encrypted_unit_via_map checked only that 0x47
reappeared at the 192-byte stride, leaving corruption in the other 6112
bytes undetected. The plaintext is fully known, so it now asserts
byte-exact recovery against it.
Both were verified red first by mutating the production path.
The ddts box declares both a channel count and a 16-bit speaker mask, and
a decoder may trust either. dts_channel_layout ended in a `_ => 0x0007`
catch-all describing five speakers (C + L/R + Ls/Rs), so for AMODE 6, 7,
and 10 through 15 the box contradicted the count DTS_AMODE_CH declared
alongside it — provoking a downmix or an outright decode error.
AMODE 6 (L + R + S) is the reachable case: its S is a single
centre-surround, not the Ls/Rs pair, so it is three channels and 0x0012,
not four and 0x0006. AMODE 13/14/15 do not occur on retail media, which
ships a 5.1 AMODE-9 core plus an extension substream.
Replace the catch-all with the full 16-entry ETSI TS 102 114 mask table.
Every entry is cross-checked against DTS_AMODE_CH by counting the
speakers its bits imply: sixteen independent constraints, all satisfied,
and that table is itself pinned to the per-AMODE channel counts in ETSI TS 102 114.
AMODE is a 6-bit field, so the reserved 16..=63 are reachable from a
malformed stream. The old `unwrap_or(6)` invented a channel count for
them that no mask could match; parse_dts now refuses those frames rather
than guessing a layout it cannot name.
One existing fixture placed the ext-sync pattern at f[4..8], which
incidentally set f[7]=0x25 → AMODE 20. That test is about where the
pattern sits, not about AMODE, so the frame is now spec-legal (AMODE 9,
48 kHz) with the pattern moved into the payload proper.
The bisect ReadAction regression tests moved to freemkv-engine with the
recovery strategy, but their explanatory block stayed behind — seventeen lines
describing a `let _ = handle_read_error(..)` bug and asserting "the tests below
prove the required ReadAction values are produced". There are no tests below it;
handle_read_error is not even resolvable here any more. Anyone auditing whether
that bug is still guarded would read this and conclude yes. The block moved to
the engine alongside the tests it describes.
The enforcement contact was an address on a domain used for internal
infrastructure, which does not belong in a public repository. GitHub needs no
mailbox to exist and is reachable for anyone who can read this file.
Inside a Matroska BlockGroup the SimpleBlock 0x80 keyframe bit is
reserved and is always written as 0; keyframe-ness is carried only by
the presence or absence of a ReferenceBlock child. Both halves of the
round-trip got this wrong:
- the reader skipped past ReferenceBlock and read the reserved bit,
so every BlockGroup frame came back as a non-keyframe;
- write_block_group discarded its `keyframe` argument and never
emitted a ReferenceBlock, on the assumption that only intra frames
(PGS subtitles) reached that path.
The MPEG-2 parser stamps a per-frame duration on I, P and B pictures
alike, so all MPEG-2 video is written as a BlockGroup. That makes the
assumption false and left no video frame looking like a keyframe on
read-back. Downstream, mkv:// -> m2ts:// dropped every video frame (the
TS muxer discards non-key video until the first keyframe) while still
reporting success, and mkv:// -> mkv:// and the stdio round-trip failed
E6008, because the MKV muxer opens a cluster only on a track-0 video
keyframe and so wrote nothing at all. HEVC was unaffected: it carries no
per-frame duration, so it takes the SimpleBlock path where the flag bit
is authoritative.
Verified on a real CSS DVD: 841 keyframes out of 11440 video packets
survive a re-mux, matching the I-picture count in the source bitstream.
Also stop running non-NAL video through the Annex-B converter. MPEG-2
and VC-1 elementary streams are already start-code framed, so
length-prefix conversion corrupts them. TsMuxer takes a per-track
nal_video flag, defaulting to the previous behaviour, which M2tsStream
sets from each video stream's codec.
The Url mux path prunes streams inside input() via InputOptions.selection;
MuxOptions.selection only applies on the File/Session arms. A Url-source caller
must set InputOptions.selection — noting it so a future caller does not put the
selection on MuxOptions and silently keep every track (the bug the GUI hit).
The sweep/patch recovery strategy, mapfile, retry-decision state machine,
section-recover, and damage classification move out of libfreemkv into the
new freemkv-engine crate. libfreemkv keeps the raw single-shot read and
SCSI-fact translation (SenseFamily stays in scsi).
- Delete disc/{sweep,patch,mapfile,read_error,section_recover}.rs, the
Disc::copy/sweep/patch methods, the Copy/Sweep/Patch option+result types,
classify_damage/DamageSeverity, progress_snapshot_from_mapfile, and the
three recovery integration tests.
- Trim public surface the recovery deletion orphaned: delete the dead
READ_PIPELINE_DEPTH const, the write-side SectorSink/FileSectorSink (no
consumer), and the DriveSpeed enum (its one live use — set max drive
speed — becomes Drive::SPEED_MAX_KBPS). Make mapfile_path_for,
decrypt_sectors_mapped pub(crate); gate NoopEvents to test.
- Version 1.6.0.
A title declaring two audio PIDs, pruned to one via StreamSelection::apply
before build_iso_pipeline, must never surface a frame from the excluded PID.
Proves the declaration-driven seam end-to-end through the full highway (read ->
demux -> codec parse): the demuxer is built from the pruned title.streams, so
the excluded PID is untracked and skipped, and both track headers and frames
follow the pruned list. Builds on the existing synthetic-TS harness.
Typed iterators over the audio/subtitle/video streams, cleaner than matching on
the Stream enum for the common iterate-the-tracks case (stream selection, the
desktop UI info panel, disc-info). Additive; all lib tests pass on 1.86.
The demux pipeline is declaration-driven off DiscTitle.streams (build_demux_state,
DiscStream::new, and the MKV writer all key off that list), so 'which streams to
keep' is already a pipeline capability with no public knob. This adds the knob:
- mux/select.rs: StreamSelection { audio, subtitle: PidFilter::All | Only(Vec<u16>) }
+ apply(&mut DiscTitle): keep Video always, keep Audio/Subtitle whose PID the
filter lists, prune the rest (and the parallel codec_privates in lockstep);
error SelectionPidUnknown on a listed PID absent from the title (fail loud, not
a silently-missing track). Pure; 6 unit tests. Re-exported at crate root.
- Error::SelectionPidUnknown (E6014).
- MuxOptions gains (+ derives Default now) applied in mux_stream's
Iso/Session arms before the highway/DiscStream builds demux state (and before
probe_and_remap's DVD AC-3 PID rewrite). InputOptions gains applied
in input()'s iso arm right after the title-index bounds check.
PIDs not languages -- language->PID is caller/engine policy. Default All/All is a
no-op (apply gated on !is_all()), so the no-selection path is byte-identical:
nothing below the title-finalization line changes (ts/ps/demux_thread/
pipelined_stream/mkv/disc untouched). All 2488 lib tests pass on 1.86.
Distinguishes a WHOLE-DISC key failure (E_NO_DISC_KEY / E_KEYDB_LOAD /
E_AACS_NO_KEYS -- every title fails identically) from a per-title skippable
stub. The engine's multi-title loop uses it to fail-fast on the first no-key
title instead of iterating all N. Additive.
The engine's multi-title rip loop classifies per-title mux failures (halt vs
skippable stub vs hard) using these typed classifiers instead of E-code string
matching. Additive; no behavior change.
Small pure error->(status,sense) introspection helper the relocated
sweep/patch will need externally. Same category as the prior WritebackFile/
resolve_content_key_map promotions -- infra, not policy. No behavior change.
Bounded-cache buffered File replacement used across mux/extract/sweep/patch
-- general I/O infrastructure, not recovery policy. freemkv-engine's
relocated sweep/patch need to construct it directly once those methods
leave this crate. No behavior change.
Moved SenseFamily::from_sense_key + is_wedge_family from disc/read_error.rs
into scsi/mod.rs (with its own tests) and re-exported at the crate root.
This is pure SCSI sense-code classification -- objective hardware fact, zero
recovery-policy opinion -- so it belongs in the library primitives, unlike
the retry-DECISION state machine (ReadCtx/PassSummary/ReadAction/
handle_read_error) built on top of it, which is freemkv's specific recovery
strategy and is moving to freemkv-engine next.
disc/read_error.rs and disc/section_recover.rs now import SenseFamily from
crate::scsi instead of defining/re-exporting their own copy. No behavior
change. Precommit green on Rust 1.86 (fmt+clippy+test).
The upcoming freemkv-engine crate needs both to build the multipass
sweep/patch recovery strategy externally over Disc's public API. Everything
else sweep/patch touch on Disc was already pub; these were the only two
gaps. No behavior change -- visibility only.
DTS core decodability gate (core_header_drop_reason) — full ETSI TS 102 114
spec-conformance sweep against a reference decoder the spec core-header rules and
a reference decoder parse_frame_header:
- deficit_samples: only require ==32 for NORMAL frames (FTYPE==1). A
TERMINATION frame (FTYPE==0, the last frame of a stream) legitimately
carries fewer and is fully decodable; the old unconditional check dropped
it on every stream that ends on one — a guaranteed per-track silence gap.
Matches a reference decoder (normal_frame && deficit != DCA_PCMBLOCK_SAMPLES) and
a reference decoder (branches on normal_frame).
- reserved bit (after RATE): both reference decoders SKIP it (a reference decoder
skip_bits1, a reference decoder bits_skip1 "Reserved field") and never reject on it.
Rejecting was a false-drop that silenced any real stream whose encoder
set the bit. Relaxed to read-and-discard; DropReason::ReservedBit removed.
Swept and confirmed spec-correct as-is (no change): npcmblocks multiple-of-8,
frame_size>=96, audio_mode>=16 (a reference decoder-permissive), sample-rate validity
table (matches avpriv_dca_sample_rates incl 96k/192k at 14/15), LFE flag==3
invalid, PCMR bits table (matches a reference decoder sample_res {16,16,20,20,0,24,24,0}).
Bit-read order verified field-by-field against a reference decoder. bit_rate is left
unvalidated (lenient, never-false-drop direction) as before.
Tests: termination frame with small deficit is kept; normal frame with bad
deficit is dropped; reserved-bit-set frame is kept. make_bad_dts_core now
uses an invalid LFE flag (duration-neutral) instead of the relaxed reserved
bit.
TrueHD: add coverage for the EXTENDED major-sync header CRC path (ms[25]&1,
mshdr=28+2+2n) — previously zero-tested, the exact path a shipped endianness
bug once used to silently drop whole 7.1/Atmos tracks. Trailer is an
independently-computed oracle (separate CRC-16/0x2D, anchored to the 0x4FF7
catalogue value, NOT crc16_mlp), stored little-endian; test asserts accept,
body-corruption reject, and big-endian-trailer reject.
mux driver: extract the finish completion mapping into pure mux_run_completed
so the finalize_failed -> completed=false branch (reachable only via real
write-thread wedge timing) is unit-tested; add an out-of-range
MuxInput::Session title_index test asserting a clean Error::MuxTrackRange
(E9011) instead of a panic.
- resolve_fmts_key_map: distinguish a genuinely-not-FMTS disc from a
transient live-drive read fault. read_filesystem now returns the new
Error::UdfNotFilesystem for a deterministic tag/format mismatch (no AVDP,
no partition descriptor, no FSD); resolve maps only UdfNotFilesystem (fs)
and UdfNotFound (.tbl absent) to Ok(None), and PROPAGATES DiscRead / other
I/O faults so a marginal AACS 2.1 disc fails loud instead of silently
dropping forensic content under a base-Unit-Key-only map.
- DTS_AMODE_CH (mp4/audio.rs): extend 10→16 entries
{1,2,2,2,2,3,3,4,4,5,6,6,6,7,8,8} (the spec per-AMODE channel table / ETSI TS 102 114) so
the spec-legal high AMODEs that now pass the decodability gate declare
their true channelcount (AMODE 13→7, 14/15→8) instead of a truncated 6.
- session.rs resolve_keys "called before scan" guard is now testable:
from_parts_for_test takes Option<Disc>; added a test that a disc-less
session returns a clean DeviceNotReady Err rather than panicking.
- mp4/read.rs: a track with samples but a missing/malformed stts (mandatory
per ISO/IEC 14496-12) is dropped rather than emitting all-zero timestamps,
matching the existing stco/stsc guards; all-tracks-dropped → Mp4Invalid.
- Remove the inert MuxInput::Iso.key_map field (the Iso path re-derives its
map inside build_iso_pipeline); the live path keeps Live.key_map.
All four fixes are mutation-verified.
Fix 1 (correctness): resolve_fmts_key_map's per-index phase probe read a
single representative segment via read_unit (whose read_sectors(...).ok()?
swallows read errors into None) with no fault fallback. A transient live-drive
read fault (e.g. NOT READY 2/04/3E on the BU40N) made every probe read return
None, giving even==0 && odd==0 — indistinguishable from a genuine wrong key —
so resolve_tie_phase returned FmtsKeyMissing and aborted the entire multi-title
rip even though the forensic index keys were valid and in hand.
Extract the probe into probe_index_phase, which returns Phase / WrongKey /
ReadFault. It mirrors the anchor loop's tolerance: it tries multiple same-index
segments and only concludes WrongKey once a read actually succeeded and decrypted
to no clean parity. If every read of every same-index segment faults it returns
ReadFault; the caller then leaves the phase unresolved so the range-builder
defaults to Phase::All (decrypt both parities, demux drops the garbled alternate
half) instead of aborting. A wrong key can never be masked as a read fault:
ReadFault requires that not a single read succeeded, so zero decrypt evidence.
Fix 2 (test coverage): resolve_mux_key_map's multi-CPS branch was only exercised
with an all-zero source, so pick(), the KeyFetch cold path, and the fail-loud
DecryptFailed guard never ran. Add tests over real AACS ciphertext (via
aacs_encrypt_unit_for_test) covering pick() selecting the correct pool index,
the DecryptFailed guard firing when a clean sample matches no held/fetched key,
and the fetch cold path recovering a missing unit key. Mutation-verified.
Fix 3 (doc): move the reader_event_fn EventKind->MuxEvents mapping doc off
session_mux_keys onto reader_event_fn.
Round-2 follow-ups to 6d6e60f (inline base-map resolve on the live
single-pass Session/Live mux arms).
Fix 1 (halt threading) — the inline resolve chain sampled ciphertext off
the LIVE drive with no cancel token, so an operator /api/stop during key
resolution was not honored (the FMTS probe can issue hundreds of reads,
each able to stall to the 60s SCSI recovery timeout — violating the
"don't hammer a struggling live drive" rule). Add an optional
`halt: Option<&Halt>` to `resolve_mux_key_map`, `resolve_fmts_key_map`,
`resolve_inline_base_map`, and `Disc::resolve_content_key_map`, and poll
it at each loop boundary (FMTS anchor + per-index probe loops, multi-CPS
extent loop) — returning Err(Halted) promptly. Live/Session arms pass the
driver's halt; sweep/patch pass their own token (via Halt::from_arc);
file-backed probe/ISO callers pass None. Tested with a pre-cancelled halt
(Err Halted, no extent sampling) and a None-halt no-abort case;
mutation-verified (dropping the extent-loop check → Ok, not Err).
Fix 2 (Session-arm coverage) — the MuxInput::Session arm ran the same
resolve→install→decrypt sequence as Live but had NO end-to-end test
(DiscSession only exposed open(), which needs live hardware). Add a
#[cfg(test)] DiscSession::from_parts_for_test (injected reader + scanned
disc, no Drive), an end-to-end AACS decrypt test through the Session arm
(mutation-verified: dropping with_key_map → mux aborts), and a
missing-reader clean-error (not panic) test.
Fix 3 (cleanups) — io_error_code: remove the unreachable typed-Error
downcast branch (From<Error> for io::Error stringifies; no path builds an
io::Error holding a typed Error), keeping the stringify parse is_halt /
is_skippable_title_stub rely on. Add a resolve_keys_for test covering the
largest-title sampling branch. Document the patch wedge-exit coverage gap
(TODO) in passn_handler_ab.rs.
Under the map-only decrypt model an AACS DecryptingSectorSource decrypts
nothing until a key map is installed; with no map the AACS arm fails loud
with DecryptFailed on the first content unit. The two inline live-mux arms
in mux_stream did not install one:
- MuxInput::Session (freemkv `rip disc://…mkv`) installed NO map at all.
- MuxInput::Live (autorip non-FMTS single-pass) installed only a
caller-supplied forensic FMTS map, which is None for a plain AACS disc.
So EVERY plain AACS Blu-ray/UHD ripped via the live single-pass path failed
DecryptFailed on the first content read. This predates the mux_stream
refactor: the bug was introduced with the map-only decrypt model, and the
pre-refactor CLI likewise built DiscStream::new without with_key_map.
Fix: add resolve_inline_base_map, the inline counterpart to what
build_iso_pipeline does for the file highway. Both arms now resolve the
AACS map off the reader (borrow to sample, then move into DiscStream) and
install it via with_key_map before any read. DVD/CSS keeps DecryptKeys::None
(DiscStream's per-title CSS crack owns it); clear/raw resolve to no map.
Session passes session.key_fetch() so a multi-CPS/orphan unit can still be
recovered; a caller-supplied FMTS map (autorip) is used verbatim, never
re-resolved.
Tests: an end-to-end MuxInput::Live mux over a genuinely-AACS-encrypted
synthetic unit now decrypts and finalises (mutation-verified: dropping the
resolve/install makes the mux abort). Adds a pub(crate) test-only AACS
encrypt helper so the mux test can build a real encrypted fixture, and a
gating test for resolve_inline_base_map (AACS→map, CSS/clear/raw→none).
- dts: accept all 16 legal AMODE channel-arrangement codes (0-15), not just
0-9. Per ETSI TS 102 114 the 6-bit AMODE field has 16 defined arrangements;
only 16-63 are reserved. a reference decoder the spec per-AMODE channel table confirms 10-15 are
decodable 6/7/8-channel layouts. The old bound of 10 dropped spec-legal
multichannel core frames as undecodable, silencing recoverable audio. Add a
regression test (literal 0..16 range) that fails if the bound reverts to 10.
- keysource: only memoize a NEGATIVE (empty) key-fetch result when every source
genuinely ran and none held the key — never when a source Err'd (network down,
unreachable). A transient outage was being cached as a permanent "no key" for
the fingerprint, permanently dropping a unit that could be recovered once the
source came back. Thread an `errored` flag out of the drivers and gate the
cache insert on it. Tests cover both the recover-after-outage case and that a
genuine absence is still memoized.
- pgs_forced_probe: add happy-path coverage feeding real synthetic BD-TS PGS
display sets through the full demux -> parse -> observe -> apply path, both a
forced verdict landing and a non-forced verdict clearing a vendor flag.
- mp4: correct fit_report doc (audio carried is AC-3/E-AC-3 AND DTS/DTS-HD).
- scan_iso test: add independent fixture expectations (volume id) so the parity
test is no longer purely tautological against a re-run of the same composition.
The finish stage and the highway path already treat Error::Halted as a clean
operator stop (completed=false), but the header pump (`stream.read()?`) and the
frame pump (`Err(e) => return Err(e)`) propagated it as a hard error. Since slow
recovery reads dominate wall-clock, a Stop almost always lands mid-read, so it
surfaced as a mux failure instead of a resumable incomplete outcome — breaking the
driver's own "a clean operator stop is not an error" contract and diverging from
the ISO/Url highway (which returns Ok(None) on halt).
Both read arms now route Error::Halted to the completed=false path. Adds
error::is_halt(&io::Error) (typed, mirrors is_skippable_title_stub) as the check.
Two regression tests cover a halt landing mid-header-read and mid-frame-read.
The 60s per-frame send deadline the mux driver applies in drive_mux was
hardcoded. That is correct for autorip (its hard watchdog + container-restart
model wants a wedged sink surfaced as a bounded per-frame timeout), but wrong
for the CLI's interactive stdout:// / network sinks: the old inline
output.write() had no deadline, so a slow-but-alive downstream (paused pager,
backpressured pipe, slow peer) was spuriously reported as an interrupted mux
after 60s of backpressure.
Add MuxOptions.send_deadline: Option<Duration>. Some(d) keeps a hard d timeout;
None means no backpressure timeout (block on a live-but-slow sink). None is
resolved to an effectively-unbounded-but-Instant-safe NO_SEND_DEADLINE at the
effective_send_deadline seam. Ctrl-C / halt stays responsive under None because
send_with_halt slices its wait at POLL_INTERVAL and re-checks the halt token
every slice regardless of the deadline value.
Unit-test the routing seam (Some(d) -> d, None -> unbounded, never collapses to
60s) and that MuxOptions carries the knob.
The mux drive loop hoisted out of autorip/CLI into libfreemkv's
drive_mux dropped three guarantees the hand-rolled loops enforced:
A. Header-buffer OOM cap. The header pump buffered every PES frame
until codec_private resolved with no byte accounting, so a title
whose video header never resolves (damaged/undecryptable HEVC on
marginal UHD) buffered the whole 30-90 GB title into RAM until
OOM-kill. Restore the 512 MiB HEADER_BUFFER_CAP_BYTES cap autorip's
run_mux enforced; once exceeded, fail fast with the same
Error::MkvInvalid the headers-never-resolved gate already returns.
B. Watchdog feed during the header-frame drain. After headers resolve
the buffered header frames are flushed to the sink, but
on_write_progress (the sole feed for autorip's hard watchdog) fired
only in the steady-state loop, leaving the watchdog unfed during a
long/slow drain and able to false-escalate. Feed on_write_progress
per buffered frame during the drain, matching the steady-state path.
C. Per-title CSS key on MuxInput::Session. The Session arm set keys via
disc.decrypt_keys() unconditionally, which returns the largest
title's VTS CSS key; muxing a bonus title in a different VTS
descrambled with the wrong key (DiscStream's per-title crack only
fires on DecryptKeys::None). Special-case DVD to None so the pipeline
cracks the correct per-title key, matching resolve.rs::input(). AACS
and clear discs still resolve from decrypt_keys().
Adds mutation-verified synthetic tests for each (no live drive).
STEP 4c-ii (layering refactor): give `mux_stream` a live-drive source so
autorip's live single-pass mux stops hand-rolling `DiscStream::new`.
`MuxInput::Live` is the live analogue of `MuxInput::Iso`: it wraps the
raw `Box<dyn SectorSource>` in the INLINE `DiscStream` (the same
constructor the `Session` arm uses — NOT `build_iso_pipeline`, the
prefetch highway), so a consumer's adaptive batch-retry in
`DiscStream::fill_extents` still fires on a bad live-drive sector. Unlike
`Session`, `Live` carries an optional pre-resolved forensic `AacsKeyMap`
that is applied via `DiscStream::with_key_map` before any read — required
for single-pass FMTS correctness (read only our-phase units; decrypt the
forensic segment with the mapped key). The consumer resolves keys as its
own policy and hands the banked material in; the driver re-resolves
nothing.
Unit test drives `MuxInput::Live` with a recording `SectorSource` and a
phased forensic key map, asserting the alternate-phase (odd) units are
never fetched — proof the arm builds the inline `DiscStream` and applies
the key map. Mutation-verified: dropping the `with_key_map` call reads the
dropped odd-phase LBAs and fails the assertion.
Split the ambiguous MuxEvents::on_progress(bytes, total) into two
callbacks — on_read_progress(bytes_read, total) and
on_write_progress(bytes_written, total) — so a consumer no longer has
to guess which side of the pipeline a progress figure came from. The
CLI drives its bar from the write side; autorip from the read side.
The reader EventFn's BytesRead now maps to on_read_progress; the
per-frame emit in drive_mux to on_write_progress. Both keep empty
defaults; NoopEvents and the driver tests are updated to match.
Add DiscSession::stage_drive_as_reader so the live single-pass path can
run through MuxInput::Session: the owned Drive (itself a SectorSource)
moves into the reader slot for mux_stream to take. The drive is now
held as Option<Drive> with the device path cached up front, so the
mux driver can still name the device after the drive has been staged;
the driver's Session arm uses the new device_path() accessor.
mux_stream took events as &dyn MuxEvents and passed None for the
constructors' EventFn, because that hook is Box<dyn Fn(Event)+Send+
'static> and a borrowed handle cannot satisfy 'static. The reader-side
events (BytesRead progress, SectorSkipped, BatchSizeChanged, ReadError)
therefore never fired.
Take events as Arc<dyn MuxEvents> (MuxEvents: Send+Sync+'static) and
clone it into a 'static EventFn (reader_event_fn) that translates each
real EventKind into the matching MuxEvents call. Wire it into
build_iso_pipeline (ISO path) and DiscStream::on_event (live path).
The write-side on_progress in drive_mux is unchanged.
Add a translation unit test (all four variants) and an end-to-end
mux_stream ISO test that asserts the read-side BytesRead reaches the
Arc; passing None regresses the latter.
Add the shared decrypt+mux driver both consumers hand-roll today, as a
library-only API. `mux_stream(input, dest_url, opts, halt, events)` runs
the construct -> headers gate -> open sink -> pump -> finish pipeline:
- MuxInput::{Session, Iso, Url} selects the source. The file/ISO path
calls the untouched build_iso_pipeline highway (zero added copies —
the driver reads frames exactly where consumers call stream.read());
the live path builds a DiscStream; a URL source goes through input().
- chapters:// / json:// metadata sinks short-circuit BEFORE the header
pump/gate. They write their whole file from the scanned title and need
no codec headers; the CLI placed this after the gate, so a metadata
export on a title whose video headers never resolved failed with
MkvInvalid. Fixed by construction.
- The header gate refuses a stream with no resolved codec_private
(MkvInvalid); the zero-output gate refuses an empty/undecryptable drain
(NoStreams); a halt mid-pump yields completed=false, never a success.
- Frames are written through a WRITE_PIPELINE_DEPTH consumer pipeline so
the latency-bound sink write overlaps the next read.
Add error::is_skippable_title_stub(&io::Error) so consumers can drop the
E7023/E6008 string-match, and DiscSession::take_reader for the live arm.
Driver body unit-tested in isolation via a synthetic Stream against
null:// / chapters:// / json:// sinks (short-circuit, header gate,
zero-output gate, halt, happy path); each gate mutation-verified.
Consumers are NOT migrated yet (steps 4b/4c).
Hoist the AACS base-unit-key resolution glue the CLI and autorip hand-rolled
around the existing primitives into one place: sample the largest title's
ciphertext, run the ordered key sources first-valid-wins, bank the winning
unit keys onto the disc, and build the read-time KeyFetch.
- resolve_keys_for(reader, disc, sources): the free-function core, over any
SectorSource (live drive or a scan_iso file reader). Returns the structured
ResolutionTrace plus the KeyFetch; a non-AACS disc is a no-op (empty trace,
no fetch).
- DiscSession::resolve_keys(sources): samples through the session's reader
(staged file reader if present, else the live drive), banks onto the
scanned disc, and retains the KeyFetch on the session (key_fetch()
accessor) for a later mux.
- KeySourceFactory: the Arc source factory the consumer supplies (libfreemkv
builds no key sources itself).
Sampling is skipped when the factory yields no sources (resolution is a miss
regardless) — no wasted disc read. Tests cover banking, the no-key path
still building a fetch, and the CSS/None non-AACS no-op.
Introduce libfreemkv::scan_iso(path, opts) -> (Disc, Box<dyn SectorSource>),
the file-backed counterpart to DiscSession::scan. It is the single place
that opens a FileSectorSource, reads its capacity, and runs Disc::scan_image,
returning the scanned Disc plus a reusable reader over the same image so
consumers stop hand-rolling that triple.
Add an integration test that materialises a minimal synthetic UDF image to a
real file, asserts scan_iso matches the manual open+scan_image composition,
and confirms the returned reader is still usable (capacity + sector read).
Also covers open-failure and scan-failure error propagation.
New src/session.rs providing DiscSession, DeviceTarget, and KeySpec,
re-exported from lib.rs. DiscSession::open runs the drive open +
advisory wait_ready/init/probe_disc bring-up (owning the Drive by
value); identify()/scan() are split so consumers keep their own
UI/TMDB sequencing. KeySpec carries consumer-built credentials +
key_sources which scan() forwards into ScanOptions without clobbering
caller-set fields (the library derives no certs and reads no keydb).
Unit tests cover the KeySpec->ScanOptions forwarding.
Multi-round audit of the decrypt/AACS/mux-codec refactor. Fixes, in
descending severity:
- mux/mp4/read.rs: bound untrusted-input allocations. `sample_budget`
now also capped by file_len (a fixed-size stsz claiming count=u32::MAX
can't inflate the Vec<SampleRef> past the file's own size); trak scan
capped at MAX_TRACKS matches; find_box() takes only the first match
(cap=1) instead of materializing every match. Removes dead find_boxes
wrapper.
- disc/mod.rs: merge_content_key_ranges now UNIONS same-key overlapping
ranges (coverage-preserving) instead of dropping the non-overlapping
tail, which silently left encrypted LBAs uncovered -> ciphertext
passthrough in the whole-disc sweep/patch map. Different-key overlap
(malformed) still dropped to keep the set disjoint.
- sector/decrypting.rs: remove dead unit_key_idx field + with_unit_key_idx
setter (vestigial from the pre-keymap trial-decrypt design; AACS is
map-only now). Fix stale docs.
- decrypt.rs / resolve.rs / error.rs / extract.rs: doc/comment drift from
the refactor (AacsKeyMap positive-map semantics, resolve_mux_key_map doc
reattachment, decrypt_sectors_in_content legacy-alias, E_MP4_INVALID
meaning, multi-CPS orphan by-design note).
Test coverage (all mutation-verified real):
- DTS NeedMore force-flush buffer bound; FLAC/MPEG-audio PTS carry-forward;
mp4 mdhd timescale=0 divide-by-zero guard, MAX_TRACKS cap, sample-count
file_len bound, MAX_ALLOC_BYTES cap under inflated file_len.
- resolve_fmts_key_map: extracted filter_addressable_segments,
resolve_tie_phase, fill_base_key_gaps as pure behavior-preserving
helpers, each unit-tested (segment filter, phase-tie arms, gap-fill
gaplessness over every extent).
- AacsKeyMap now derives its distinct key-index set once at construction
(from_ranges_phased) instead of re-allocating/sorting it on every
decrypt batch; key_indices() returns the cached slice.
- Extract the whole-disc content-map range merge out of
resolve_content_key_map into merge_content_key_ranges and cover it:
sort/disjoint, shared-clip dedup, overlap drop, adjacent-kept.
- Repoint three tautological AACS tests (which passed only via the
unconditional-Err AACS arm of decrypt_sectors) at the real shipping
path decrypt_sectors_mapped: out-of-range mapped key index, empty pool
vs a non-empty map, and a scrambled encrypted trailing partial (the new
guard) — plus a clear-trailing-partial pass-through and an explicit
fail-loud safety-net test for AACS reaching the unmapped wrapper.
- Add the missing end-to-end AACS round-trip through the decorator +
key map (aacs_decorator_decrypts_encrypted_unit_via_map) and a
mapless-AACS-fails-loud decorator test — the exact class of bug the
TrueHD probe shipped. Fills the // TODO: AACS round-trip test gap.
- Delete the now-dead recovery test fixtures (encrypt_aacs_unit_bad,
AnyLbaUnit) left after the reactive key-fetch path was removed; keep
encrypt_aacs_unit + FixedUnit, now used by the round-trip test.
- resolve_fmts_key_map: filter segments to those addressable within THIS
title's extents; a title with no forensic content (menu/extras playlist,
or a different clip) returns Ok(None) and takes the base Unit-Key/CPS
path instead of hard-failing FmtsKeyMissing. Previously the first
non-forensic title aborted the entire whole-disc sweep
(resolve_content_key_map iterates every title) and blocked muxing any
non-main title.
- FMTS phase probe: an even/odd is_clean tie now only fails loud when
BOTH halves are 0 (no clean decrypt). A both-clean tie is source-zero
padding (is_clean is true for any key on all-zero content) — the key is
valid, default Even, never abort the rip on a padding-heavy sample.
- extract_tree: multi-CPS discs now build the exact per-CPS content map
(resolve_content_key_map) instead of a blanket key-0 map that silently
mis-decrypted every secondary-CPS file into garbage. Single-CPS keeps
the blanket key-0 map (one key opens every unit, incl. orphan clips).
- decrypt_sectors_mapped: a trailing partial unit that is inside a mapped
range AND flagged encrypted in its clear seed now fails loud (a CBC
fragment split across a boundary can't be decrypted) instead of being
emitted as clear. New aacs_unit_seed_encrypted reads the flag on a
partial.
- Correct the stale decrypt_sectors doc (AACS arm now always errors;
AACS decrypts only via decrypt_sectors_mapped).
The mux's TrueHD 7.1/Atmos channel-correction probe built its
DecryptingSectorSource with no key map. Since AACS now decrypts only via
the resolved map, the mapless probe read failed loud on the first unit
and the correction was silently skipped on every AACS disc — 7.1/Atmos
stayed understated as the MPLS-declared 5.1. The probe now resolves the
same up-front key map the mux read installs (non-fatal on failure).
Delete the reactive per-read key-fetch recovery path
(DecryptingSectorSource::with_key_fetch, the recovery field/cipher
scratch, and sector/recovery.rs). It was unreachable: AACS resolves its
full key map up front, so a mapless AACS read is a bug, not a
recover-mid-read case. KeyFetch itself stays — it is the up-front
resolver fetch used by resolve_mux_key_map.
Every AACS decrypt now goes through the resolved key map (decrypt_sectors_
mapped): the map keys each content unit up front and a missing key fails at
resolve time. The old trial-decrypt path — try each held key per unit, keep
the first-tried plaintext on a miss — is gone; decrypt_sectors_impl's AACS
arm now fails loud (reaching it means a reader was built without its map,
which would silently apply a wrong key). CSS (self-descramble) and the clear
no-op path are unchanged.
Disc::sweep and Disc::patch resolve a whole-disc key map up front for a
decrypting pass (the fetch secures any missing CPS-unit key, fail-loud) and
decrypt via the map — clear nav/filesystem sectors are in no range and pass
through, so the separate content-range gate and the reactive per-unit
key-fetch recovery are no longer needed. extract_tree keys every unit with
the base Unit Key through the map (its encrypted-flag gate skips clear
files). Multipass sweeps stay --raw.
Removes the obsolete non-mapped-AACS trial/gate/recovery tests (the mapped
path and resolve fail-loud are tested directly).
The key map is now purely "these sectors use this key": entry_for returns
Option and an LBA in no range is left untouched (no default-decrypt-
everything fallback). resolve_mux_key_map builds explicit content ranges
for every case — single-CPS keys each content extent, multi-CPS keys each
extent with the key that opens it, and FMTS fills the non-segment content
with the base Unit Key so a whole-disc read decrypts content and passes
nav/filesystem through. Combined with the fail-loud resolve, a map can
never silently apply a wrong key, and clear sectors are never scrambled.
decrypt_sectors_mapped skips a unit with no map entry; read_plan keeps an
unmapped unit (pass-through content) and drops only alternate-phase
forensic units. Tests updated to the Option semantics.
mp4 demuxer (untrusted input): bound every allocation sized from a box
field (stsz/stco/stsc counts, stts/ctts run-lengths, per-sample and moov
sizes, plus an absolute cap so a sparse file can't inflate file_len);
guard the parse_stsd slice and a zero mdhd timescale; cap track count so
the per-track PID can't overflow; rewrite read_moov to handle size==0 /
size<8 / 64-bit largesize; parse esds/AudioSpecificConfig for AAC; write
tkhd duration in the movie timescale.
decrypt: resolve_mux_key_map now fails loud on an extent no key can
classify instead of inheriting the previous extent's key, so a keymap
never silently carries a wrong key; the sweep/patch key-fetch recovery
fails loud when a unit is still unresolved after the retry.
AACS: reject inverted forensic segments in both range builders; compare
the forensic index in u16 space so an out-of-range value can't truncate
onto a valid u8 index. RECOVERED_ERROR no longer latches the damage zone,
preserving the 30s wedge cooldown for a following hard error.
audio: AAC/MP2/MP3/FLAC carry the last PTS across a PES with no timestamp;
the DTS-HD extension-sync search is bounded to after the core; the MP4
16.16 sample-rate field saturates. demux_sink records the video reference
before the kind filter so audio:// / sub:// keep multi-clip PTS continuity
and the DELAY tag.
Remove a dead error variant and the AACS-unsupported-video code; codec
comments cite the primary format specs; assorted doc/naming fixes and
regression tests throughout.
Every DVD read path — the file-backed mux highway (build_iso_pipeline) and
the live-drive single-pass DiscStream — now resolves the per-VTS CSS title
key through one shared step, css::resolve_dvd_title_key, cracked keylessly in
playback order from the title's own extents. Removes the earlier design that
reused a single scan-time key (meaningless for a per-VTS scheme) and muxed a
detection-miss disc's scrambled sectors as garbage.
- Disc::scan no longer cracks a key up front; it does only the CSS bus-auth
read-unlock, hoisted before the UDF prefetch so scrambled small/menu VOBs
no longer cost a rejected read each (CSS-DVD scan ~25s -> ~6s).
- An uncrackable title hard-fails (E7023) instead of passing ciphertext as
plaintext; --raw skips the crack entirely; a Stop mid-crack surfaces Halted.
- DiscStream::new is now fallible and threads raw + halt.
- Fix a stale codec-parser doc claim (TrueHD/FLAC/MP2/AAC do gate via DropTally).
The prior wording described the round-1 'self-cracking per-sector' approach
that the audit replaced; rewrite it to match what ships (per-title crack in
playback order + scan-key reuse + loud hard-fail on an uncrackable title).
The last release left this pinned to the git tag v1.5.1; release.sh expects
a committed path dep on the branch tip (it swaps path -> git tag only in the
tagged commit, matching bdemu). Restores that state so a local full-workspace
build links the sibling freemkv-unlock, and release.sh's path->tag swap works.
A CSS DVD whose main title was mis-detected as unencrypted (the up-front
crack scanned the largest cell first and starved its budget in that cell's
clear prefix) muxed scrambled sectors as plaintext at exit 0. CSS leaves
the pack/PES header clear, so an un-descrambled sector muxes as a
structurally-valid but corrupt PES packet with zero loss reported.
decrypt_keys_for_title resolves a DVD title's CSS key two ways:
- Fast path: reuse the scan's cracked key when its crack_span covers this
title's VTS (no re-read; on a live drive no second bus-auth).
- Crack: on a detection miss or a different VTS, crack from the title's OWN
extents in a SINGLE scan in natural PLAYBACK ORDER (never largest-first).
One scan = one CSS-locked early-bail, so a locked title is not re-hammered
per cell against a live drive (hard rule #2); the 50k-sector budget is the
same accepted bound the disc-wide scan uses. Cracked -> key; Unencrypted
-> clear; ScrambledUncracked -> hard-fail.
ensure_title_decryptable hard-fails an uncrackable DVD title even when
detection missed, and passes a title that resolved its OWN valid key
regardless of the disc-wide css_error. descramble_region is unchanged from
v1.5.1 (validated-key seed).
Also rename the unlocker report's DVD entry CSS -> DVD. Bump 1.5.2.
Regression since the previous release, which added an MLP major-sync checksum
gate to drop genuinely-undecodable audio frames. The checksum itself was computed
with mismatched byte order: `crc16_mlp` is the correct crc_2D table (poly 0x2D,
MSB-first) but returns its two bytes in the OPPOSITE order to libavutil's
`av_crc`, and `mlp_major_sync_crc_ok` then folded in the pre-trailer word
little-endian while comparing the trailer big-endian. The net result never
matched a real major sync, so EVERY major sync was judged corrupt. That armed the
drop-forward on the first AU and, since no major sync ever validated to clear it,
collateral-dropped every following AU forever — the entire TrueHD track was
silently dropped. Its AUs then flushed only at mux end, so the track's blocks
landed physically after all the video: a decoder reading video+TrueHD had to
buffer the whole title to reach the first audio block and spiralled into an
unbounded memory runaway ("decoder ran out of memory"). Every TrueHD title
produced after the gate landed was affected; a title from the release before it
is clean. (The header-size parse — a frequent suspect for extended 7.1/Atmos
headers — is NOT the bug; it already matches ffmpeg's `mlp_get_major_sync_size`
byte-for-byte.)
Fix: compute the checksum exactly as ffmpeg's `ff_mlp_checksum16` —
`crc16_mlp(body).swap_bytes() ^ AV_RL16(word) == AV_RL16(trailer)`. Cross-verified
byte-exact against two real discs (a 7.1/Atmos title and a 5.1 title, independent
32-byte headers both validate). With the checksum correct, major syncs validate
and the drop-forward corruption protection works as intended.
Defence in depth: a major-sync checksum that STILL can't be validated (a genuinely
corrupt or as-yet-unparsed header) no longer arms the drop-forward until we hold a
validated baseline (`num_substreams` from a prior clean major sync) — so a single
bad header can never again silently drop an entire track.
Tests: the `finalize_major_sync` fixture now builds the checksum the corrected way;
a synthetic checksum-failed head major sync is kept, not dropped; the existing
baseline-then-corrupt drop-forward tests still pass. Verified end to end against a
real disc: the TrueHD track demuxes to a full, cleanly-decodable 48 kHz 8-channel
stream, interleaved with the video, instead of 0 bytes.
The HD DVD Title Key File (VTKF*.AACS) stores 64 title-key entries of 36 bytes
each — 1-byte BIFO + 3 reserved + 16-byte encrypted key + 16-byte binding MAC —
per AACS "HD DVD and DVD Pre-recorded Book" Table 3-8, confirmed byte-exact
against real discs (Freedom VTKF090, Dukes VTKF000: every 36-byte slot has
BIFO=0x80, a clean key, and a 0xFF binding MAC).
The parser used a 32-byte stride (a 12-byte pad in place of the 16-byte binding
MAC) with flag-based termination. That aligns entry #1 (key at offset 132, where
both strides agree) but drifts +4 bytes per entry after it and never terminates
(the previous entry's 0xFF MAC reads as a set present-flag), so it recovered a
correct key only for single-CPS-unit discs and garbage for CPS unit >=2. Every
multi-title HD DVD (Freedom, Harry Potter) was affected.
Fix: 36-byte stride, iterate the fixed 64 slots, take slots whose BIFO AV_FLG
(bit 7) is set, key at offset 4, slot index = CPS unit (skip empty slots rather
than terminate so a gap can't renumber later keys), and never read the trailing
16-byte TKF MAC as a key. Tests rebuilt on the real layout, including a full
64-entry file.
Also correct the VTKF-selection TODO in mod.rs: the AACS HD DVD Book gives the
selector explicitly (match the TKF's PLAYLIST_NAME field to the active
playlist), not the "validate against an encrypted unit" placeholder.
Reconciled against the new HD DVD reference (freemkv.org/docs/hddvd/); the spec
source is archived in private-repo/spec/.
The HD DVD AACS directory name and title-key filename are chosen by the
authoring house, but the resolver hardcoded a single spelling
(/ANY!/VTKF000.AACS, /ANY!/MKBROM.AACS, /ANY!/CONTENT_CERT.AACS). Real discs
diverge: Freedom (Memory-Tech) names its AACS dir AAC! and ships VTKF090.AACS
+ VTKF100.AACS; Harry Potter carries VTKF000/001/002/099. On such a disc the
hardcoded path finds nothing, so no MKB/title-key/cert is read and decryption
silently can't engage.
Replace the fixed HD DVD path constants with structural discovery:
- find_hddvd_aacs_dir() locates the AACS dir as the root child dir ending in
'!' that contains MKBROM.AACS (so the ..._BAK mirror is skipped; the dozens
of decoy advanced-content '!' dirs are excluded by the MKBROM.AACS guard).
- role_paths(udf, role) builds the ordered candidate list per role: the static
BD/UHD /AACS/ paths first, then the discovered HD DVD files — MKBROM.AACS,
CONTENT_CERT.AACS, and every VTKF*.AACS (sorted), not just VTKF000.
- read_first() is now generic over &str / String so it takes the Vec<String>.
BD/UHD unaffected (no '!' dir → discovery returns None, list is the /AACS/
constants exactly as before). Verified on real Freedom (AAC!/VTKF090+100) and
Dukes (ANY!/VTKF000) ISOs; unit tests cover both shapes.
Open item (TODO(hddvd-encrypted)): when a disc has multiple VTKF variants the
correct one must be chosen by validating its VUK-derived key against a real
encrypted unit rather than first-that-reads. Blocked on obtaining a genuinely
encrypted HD DVD image — all HD DVD ISOs on hand are already-decrypted rips.
A read was only ever judged "good" by the drive's SCSI GOOD status — with no
integrity check on the bytes. On dirty/marginal media a drive can silently
return best-effort ECC data (occasionally mis-corrected) with GOOD status, so a
rip could "pass clean" yet decode with errors, pushing corruption downstream to
the mux instead of catching it at the read.
Enable recovered-error REPORTING at drive-prep: MODE SELECT the Read-Write
Error Recovery page with PER=1 (TB on / DTE off so the data still comes back),
preserving the drive's own retry count. Marginal reads now surface as
CHECK CONDITION / RECOVERED ERROR. Best-effort — a drive that doesn't honor it
keeps its defaults (no regression), and on a clean disc nothing changes.
A recovered error is distrusted: it marks just that ECC block NonTrimmed for a
Pass N re-read (a clean re-read wins; a persistent marginal becomes an honest
concealed gap), and is counted in the pass summary. Crucially it takes a
per-block SkipBlock, NOT the damage-jump path — a single marginal sector must
not trigger a 64 MB skip that would discard good data on a lightly-smudged disc.
Give `info` the same forced-subtitle verdict the muxer derives during a
rip, so the two agree. A shared classifier (mux::codec::pgs::ForcedTracker)
folds a PGS track's display sets — forced iff every one carries the
forced_on_flag — and is used by BOTH the MKV writer and a new scan-time
probe that reads the title's PGS streams (reusing the TS demuxer and PGS
parser). The probe only overrides a track it actually observed content
for, so an undecrypted/unread stream keeps its vendor-derived flag. Gated
behind ScanOptions::probe_forced_subtitles (off for the rip path, which
detects forced while muxing without a second read).
Free-format MPEG-audio (bitrate_index 0) is a legal, decodable mode — the
decoder derives the frame size from the sync spacing. Dropping it was a
false positive on a clean stream, so it now passes the gate.
Three TrueHD state-machine fixes from an adversarial audit:
- A corrupt access unit drops forward to the next major sync, but only
the individually-verified corruption now feeds the whole-track poison
verdict; the resync run is collateral. A couple of transient errors can
no longer poison and discard an otherwise-good multi-hour track. The
shared drop tally gains a verified/collateral split for this.
- The per-AU PTS rate is refined only from a CRC-validated major sync, so
a corrupt major sync whose rate nibble decodes to another rate family
can no longer shift the resumed audio — a drop stays a silence gap.
- The resync clears only on a CRC-validated major sync, never on a runt
too short to hold and validate its header.
Flag a PGS subtitle track FlagForced when it displays subtitles and every
one carries the HDMV forced_on_flag (a dedicated forced/narrative track),
independent of the disc's vendor label metadata. The track header
reserves a FlagForced byte up front and it is promoted at finish() from
the accumulated display-set state. Only ever promotes — a track already
forced from the playlist metadata is never demoted.
A damaged audio access unit is now dropped rather than muxed as a
decoder-choking glitch. Sync is preserved — a drop becomes a silence
gap, never a shift — and every drop is logged. Detection is per-codec,
each mirroring the format's authoritative integrity check:
DTS core-header validity gates
AC-3/E-AC-3 native frame CRC-16 + bitstream-id range
FLAC whole-frame CRC-16 residue
MP2/MP3 header sanity + free-format reject
AAC-ADTS header sanity (raw AAC passes through untouched)
TrueHD/MLP major-sync CRC-16 + AU parity; corrupt AUs drop forward
to the next major sync, since decode state carries
across access units
LPCM and video are excluded by design (no in-frame integrity data;
inter-frame prediction). A shared DropTally handles counting, logging,
and a whole-track fallback for a mostly-undecodable track.
DTS-HD MA access units are a lossy core frame followed by trailing
extension substreams up to the next core sync. On source-damaged discs
(observed on the Bourne UHDs) the bytes where the XLL extension belongs
are neither a core sync nor an extension sync -- pure garbage -- which
desyncs ffmpeg's XLL decoder and cascades into 'Read past end of XLL
band data' / 'DSYNC check failed' across the whole track.
next_core_boundary now distinguishes three boundary states via a new
ext_clean flag on NextCore::Found:
- precise/recognized extension sync -> ext_clean=true (keep full AU)
- garbage at the boundary byte -> ext_clean=false (drop the ext)
When ext_clean is false we emit the DTS core alone (drop the smallest
junk piece, keep the frame and its PTS) and drain past the garbage to
the next core. Recognized-but-unsizeable extensions still ride the
heuristic scan and are kept intact, so lossless tracks are unaffected --
only genuinely corrupt extension bytes are dropped.
Bourne s1.dts: 1606 damaged frames / 691620 (0.232%), 93% isolated
single frames, worst run 3 in a row (~32ms lossy blip).
Parse the DTS core header (SFREQ/AMODE/RATE/LFF/NBLKS/FSIZE) → a ddts box
(sample rate, channel layout mask, core size, computed bitrate, LFE);
whole access units (core + DTS-HD extension substreams) pass through, so
an HD decoder finds the extension. dtsh when an extension sync follows the
core, else dtsc. Fit oracle now carries DTS / DTS-HD MA / DTS-HD HR.
Validated on 300 (real DTS-HD MA 7.1): freemkv's mp4 DTS track is
byte-identical to ffmpeg -c copy under ffprobe (dts / 48000 / 8ch / 7.1)
and decodes clean (exit 0). Channel layout correct.
moov now precedes mdat. At create() reserve a moov-sized hole (a free
box) between ftyp and mdat: reserve = round_up_4MB(16 B/sample ×
est_samples) + 4MB buffer, floored at 8MB. Because the hole precedes
mdat, sample offsets are fixed from the start — no rewrite, no offset
patch. finish() writes moov into the hole and pads the slack with a free
box; if the estimate is blown it falls back to moov-at-end. Streams over
HTTP without a pre-fetch. Reserve-math + box-order tests added.
Read side of mp4://: parse moov/trak/stbl (stsd codecs+hvcC/avcC/channel
info, stsz/stco/co64+stsc → per-sample offsets, stts+ctts → decode/
composition timing, stss → sync), rebuild a DiscTitle, and emit samples
as PesFrames in global decode order. Video NALs are length-prefixed in
MP4 — the exact framing the MKV muxer wants — so no reframing. Wired into
input() so mp4:// flows to every sink (mkv://, audio://, json://, …).
In-memory write→read round-trip test proves symmetry (streams, hvcC,
sample sizes survive). Progressive MP4 only; fragmented (moof) is future.
Make Mp4Sink generic over a seekable writer; the CLI output() arm wraps
the file in WritebackFile (as mkv:// does) so a UHD-scale mux to slow /
NFS staging avoids the dirty-page burst. The mdat backpatch is an
ordinary seek WritebackFile already handles (seek_then_patch_roundtrip).
Tests switched to in-memory Cursor writers.
Generalize the sink to N tracks: one video + every MP4-mappable audio
track. Audio sample entries built from the first frame's bitstream —
AC-3 (ac-3/dac3) and E-AC-3 (ec-3/dec3), parsing the (E-)AC-3 BSI for
fscod/bsid/bsmod/acmod/lfeon and the data rate. Per-sample audio
durations from PTS deltas (no reorder → no ctts, all sync). Fit oracle
(mp4_fit_report, exported): video HEVC/H264, audio AC-3/E-AC-3;
TrueHD/DTS/LPCM and bitmap subs are excluded with a typed reason so the
CLI can report exclusions — never a silent drop.
Verified vs ffmpeg -c copy on real discs: AVC+AC3 and HEVC(HDR10)+AC3
both frame-exact on every track (video 2650/4270, audio 5567/3454),
duration and colour identical, clean decode. On a pathological 4-clip
title ffmpeg's OWN output emits the same DTS-monotonicity warnings (more
of them) — source-inherent, not a muxer defect.
New mux/mp4: writes ftyp+mdat+moov (moov-at-end), streaming samples into
a 64-bit mdat and building the sample tables in memory, patched at
finish(). Video track (HEVC/AVC): passthrough length-prefixed NALs
(already MP4 framing), full stts/stsz/stsc/co64/stss and signed ctts,
CFR-derived decode timeline (pipeline carries presentation PTS only), and
a colr box for HDR10 colour signalling. hvc1/avc1 sample entry from the
hvcC/avcC codec_private. Fail-loud on codecs with no MP4 mapping.
Verified against ffmpeg -c copy on real discs: AVC-SDR (300) and
HEVC-HDR10 UHD (Dune) both frame-exact (8159 / 8160 frames), colour-exact
(bt2020/smpte2084/bt2020nc), and clean-decoding. Audio + fit oracle land
in M2.
Completes the per-track-class trio with audio:// / sub://. video:// is
the demux path with a TrackKind::Video kind filter — each video track to
its own native elementary stream (.hevc/.h264/.vc1/.m2v/.obu), no audio
or subtitles. Reuses the existing kind_filter machinery and extension
map; scheme/parse/output/write-only-guard arms added symmetrically.
The json:// sink previously emitted only codec + language + pid per
stream. Fill it out to the full DiscTitle: video carries resolution
(+ pixel dims, interlaced), frame rate (+ fraction), HDR, colour space,
display aspect, and measured CICP; audio carries channels (+ count),
sample rate (+ Hz), and editorial purpose; subtitles carry the
qualifier. Add the clip list and chapter names. This is everything the
scan resolved, so json:// is a lossless machine-readable view of a
title rather than a summary.
Two write-only file sinks that ignore the PES stream and emit the title
metadata at construction (fvi-style, wired through output()):
- chapters:// — chapter markers in the format the extension picks: .xml
(Matroska, default), .txt/.ogm (OGM simple), .vtt (WebVTT). Reuses the
existing chapters_xml/chapters_ogm writers; adds a WebVTT writer.
- json:// — one title's model (playlist, duration, size, format, streams,
chapters) as pretty JSON via serde_json.
New mux/meta_sink.rs; StreamUrl gains Chapters/Json; codec_label +
chapters_xml/ogm promoted to pub(crate) for reuse.
Tests: chapters_format_selected_by_extension, title_json_carries_streams_and_chapters.
Two extraction sinks built on the existing demux machinery: audio:// keeps
only audio tracks (native containers .thd/.dts/.ac3/.eac3/.pcm...), sub://
only subtitle tracks (PGS .sup, VobSub .idx+.sub, text .srt) — one file per
track, no video, no chapters sidecar. DemuxOptions gains a kind_filter; the
DemuxSink writers (already complete) are reused verbatim. StreamUrl gains
Audio/Sub variants; output() builds the filtered DemuxSink.
Test: kind_filter_keeps_only_the_selected_class.
The read_plan fix landed on the file-backed highway (build_iso_pipeline)
but the inline DiscStream (live single-pass) still passed the alternate
device-group half to the demux — a known FMTS bug on rip_mode="single".
DiscStream::with_key_map installs the proactive map and rewrites the
extent walk to the read plan, so the live single-pass path reads ONLY
our-phase units, exactly like the highway. DecryptingSectorSource gains
set_key_map for the already-constructed decorator. Non-forensic maps
return the extents unchanged, so every non-FMTS disc is byte-identical.
Test: with_key_map_reads_only_our_phase_units — a forensic Even segment
drops exactly its alternate units from the extent walk. Precommit green
on 1.86.
An FMTS forensic segment interleaves our device group's variant with a
foreign group's at the aligned-unit level. The mux was decrypting only
our phase but leaving the alternate (foreign) units in the buffer as
ciphertext, trusting the demux to 'drop untouched ciphertext cleanly'.
It doesn't: random 0x47 bytes at the 192-byte stride hit tracked PIDs,
mis-parse, and trip the demux's concealed-gap keyframe-resync — which
drops GOOD frames of ours around every segment (349 resyncs / ~6391
packets on Stand by Me, visible as playback flaws).
The map already knows which unit each LBA is and, for a forensic
segment, which phase is ours. AacsKeyMap::read_plan turns that into the
title's read plan: every default/CPS unit, plus inside a segment ONLY
our-phase units. The alternate units are never fetched, decrypted, or
handed to the demux — the demux sees one gapless our-variant stream.
- read_plan is general (single-CPS, multi-CPS, FMTS): a map with no
Even/Odd range returns the extents unchanged, so DVD/CSS, single-CPS
UHD and multi-CPS Blu-ray read byte-for-byte as before.
- Wired into build_iso_pipeline (the file-backed highway that muxes
resumed ISOs). Producer re-anchors unit_base per extent, so per-unit
segment reads stay unit-aligned and decrypt correctly.
- Extent gains PartialEq/Eq for the read_plan tests.
Tests: read_plan non-forensic unchanged; forensic omits exactly the
alternate units, kept units match the decrypt gate unit-for-unit. All
2314 lib tests pass; precommit green on 1.86.
Pending: end-to-end ISO re-mux validation (concealed gaps 349 -> ~0).
Verified zero callers across all consumer crates AND libfreemkv integration tests:
MUX_APP, Disc::{aacs_disc_hash,encrypted_content_ranges,inject_unit_keys},
locate_ranges, mapfile::{MapEntry,entries}, diag::dump_mkv_track,
DiscStream::{errors,lost_bytes} (read via accessors). Removed the unused mux
DemuxSink/FviSink crate-root re-exports (constructed internally by output() via
the direct module path). Staged demux option variants marked allow(dead_code).
Removed the test-only unchecked wrapper and renamed decrypt_keys_for_title_checked
-> decrypt_keys_for_title (one method returning (keys, title_is_clear)). Production
already used the checked form; the _checked suffix had no counterpart. Callers +
docs updated; two tests take .0.
crack_key_halt had no caller except the crack_key wrapper — a needless middle
layer. crack_key now calls crack_key_scan directly; crack_key (Option) and
crack_key_outcome (full CrackOutcome + halt) remain as the two real entry points.
create_with_source (full SourceInfo) was only ever called by the create wrapper
with a partial SourceInfo — the extra capability was unused (YAGNI). Inlined the
constructor body into create and deleted the _with_source variant; callers
unchanged.
One method per action (CLAUDE.md), not a foo_with_X pair — create_at was the real
constructor and create was a None wrapper. Callers pass Option<&Path> directly.
Adds parse_hex_u16/u32/u8 and exposes strip_hex_prefix so callers stop hand-rolling
from_str_radix(trim_start_matches("0x")) — a case-sensitive strip that this module
exists to prevent. disc::aacs_disc_hash now uses strip_hex_prefix.
AacsState (public via Disc.aacs) and Key (the decrypt_with key-transport enum)
are crate-root re-exported and carried VUK/unit/read-data keys + volume id on
#[derive(Debug)]; HandshakeResult carried the VID + AACS 2.0 bus key. Manual
Debug impls print shape only, guarded by red->green tests.
These carry raw unit-key / VUK / processing-key bytes on their Debug; manual impls
print shape only (unit_keys_len, redacted markers). Each has a red→green test.
CssState is reachable via the public Disc.css field; #[derive(Debug)] leaked the
raw CSS title key on any {:?} of a Disc. Manual Debug prints crack_span only.
DeviceKey/HostCert/Vid/MediaKey/Vuk/ProcessingKey/UnitKey/DiscEntry carried key
material on #[derive(Debug)] — a stray {:?} would leak device/host-private/media/
volume/unit keys. Manual Debug impls print shape only. redaction_tests asserts no
key bytes appear and a redaction marker is present (fails if a derive returns).
FMTS (AACS 2.1) now decodes per (LBA, phase): Phase enum + AacsKeyMap::
from_ranges_phased, decrypt only the variant's parity half. resolve_fmts_key_map
does a 2-phase index-1 anchor then per-index phase probe, and sizes the forensic
set to whatever the source returns (no hardcoded 32). KeyFetch is now two explicit
operations (unit_keys / fmts_indexes) and KeySource splits get_uk into
get_unit_keys + get_fmts_indexes. BYPASS_FMTS_KEY gate removed (first-class format).
Teed up for 1.4.5. Local WIP baseline.
The keyserver protocol now returns all 32 index keys as an array for a
forensic content sample (and a single-element array for plain content).
resolve_fmts_key_map sends one forensic batch and maps array element i to
segment index i+1, replacing the per-index blind-probe collection loop
that repeatedly hit the key service. Segment/index parsing and the
aligned-unit content classification are reworked to support this:
- rename variant_select -> index_select (per-index, not per-variant)
- content classification moves to is_clean(buf, ContentFormat) so the
unit selector emits only units the key service accepts
- segment.rs: parse IndividualSegment.tbl index tags + SPN ranges,
build contiguous LBA key ranges from the resolved 32-key array
- decrypt/decorator plumbing for the resolved per-index keys
Fail loud (FmtsKeyMissing) when the forensic query returns < 32 keys or
any segment index stays unresolved.
decrypt:
- decrypt_sectors is now a pure decrypt (apply key, leave plaintext, report
unverified bytes); TS-structure is a separate primitive (is_clean_ts/ps) used
only for key selection and read-verify. The mux passes decrypted bytes through
(the demuxer drops non-conforming packets), ending the NULL-TS conceal loop and
the per-unit key-server refetch storm. Key-proof floor replaces the 75%
supermajority.
recovery:
- Removed the post-read decrypt-verify gate (verify.rs) that mis-aligned the
disc-absolute unit grid against clip-anchored AACS units and false-failed good
clips (e.g. Dunkirk's orphan-CPS clip). Bad sectors are marked by physical read
result; decryptability is proven at scan + mux time.
HD DVD (first-class AACS):
- Role-based candidate-list file sourcing so an HD DVD's /ANY!/ files
(MKBROM.AACS, VTKF000.AACS, CONTENT_CERT.AACS) are found with no disc-type
branch. parse_vtkf parses VTKF000.AACS into the same UnitKeyFile as a BD
Unit_Key_RO.inf, so the shared VUK unwrap applies unchanged. set_unit_base
clip-anchoring. Two decrypt-axis assumptions remain UNVERIFIED-HDDVD-DECRYPT
(no encrypted disc to test).
mux:
- MVC (Blu-ray 3D) track signals unified into one MVCDecoderConfigurationRecord;
release-safe track_vint (3-byte VINT) and pid_index (i32) guards.
hardening:
- Container-aware is_clean / encryption detection; bytes_bad_in_title fail-safe
on a corrupt mapfile; CSS crack gated on DiscFormat::Dvd (HD DVD excluded);
non-vacuous CSS tests; patch NOT_READY/HARDWARE/ILLEGAL_REQUEST/ABORTED
sense-path tests.
decrypt_sectors is now a pure decrypt — apply the CPS unit key, leave the
plaintext, report how many bytes did not reach clean TS ("unverified"). It
never restores ciphertext, nulls, or re-fetches. "Did a key produce clean TS?"
is a key-selection / read-verify signal, not the verdict "did we decrypt?": a
correct key can decrypt a bad-encoded region, and broken TS is a muxer concern
(the demuxer drops the packet and resyncs).
Callers own the policy:
- mux (read > decrypt > mux): pass the decrypted bytes to the muxer, whatever
they are; fail loud only on a genuine can't-decrypt (no key / misaligned).
- sweep/patch (reading from a disc): an unverified unit is a bad read — recover
a fresh key and retry, or fail loud so disc-recovery re-reads it.
Removes three duplicated decisions — the decrypt-time ciphertext restore, the
mux NULL-TS conceal loop, and the per-unit key-server refetch — plus the dead
aacs_unit_still_ciphertext predicate. Key-fetch recovery now samples the on-disc
ciphertext explicitly (a pure decrypt leaves the buffer plaintext) and lives
only on the rip/verify path, never the mux.
Fixes the 30-90s/region mux stalls and key-server storm on bad-encoded UHD runs
that 1.4.1 left behind (it relaxed the gate but not the surrounding machinery).
AACS content decryption rejected a whole 6144-byte aligned unit unless
EVERY content packet was conformant MPEG-TS. One authored-bad packet (a
pressing/encoding defect or an AACS 2.1 forensic-variant frame) made the
mux conceal the entire unit as NULL TS — destroying up to 31/32 good
packets and tallying them as loss, surfacing as false "corruption" on
otherwise-clean discs (observed across two UHD titles).
decrypt_unit now asks only "did a key OPEN this unit?" — a padding-aware
>=75% supermajority of content packets restoring their 0x47 sync, a gate
no wrong key can reach (uniform-AES noise floor) yet one that tolerates a
minority of authored-bad packets. Opened units pass through VERBATIM; a
non-conforming packet is left for the demuxer to drop on sync-loss and
resync past. TS-sync conformance is a muxer concern, never a decryption
verdict. The post-read verify/sweep gate now shares the same primitive so
it can never disagree with the mux decrypt.
Also unify the MVC (Blu-ray 3D) track signals: the mvcC CodecPrivate
extension, the BlockAdditionMapping, and each per-frame BlockAdditional
all derive from one MVCDecoderConfigurationRecord built once per track, so
a malformed dependent-view parameter set can no longer orphan a BlockAddID.
Second audit round converged (severity collapsed 6 HIGH -> 1; the one
HIGH was a bounded 32-element scan, not a defect; the sole spec MEDIUM
was the same false-positive re-raised — 0xBF matches ISO/IEC 14496-15
§7.6.2 verbatim). One genuine robustness fix plus coverage:
- extract_mvc_params: skip a zero-length NAL instead of abandoning the
scan, so a stray length prefix before the subset SPS/PPS no longer
silently drops 3D signalling. Test proves params after a zero-length
NAL are still found.
- Tests: parser_for_mvc_dependent routes H.264 to a passthrough parser;
passthrough with an IDR does not re-assert param sets (the keyframe &&
!mvc branch).
- Document the per-playlist (not per-clip) is_3d latching as a known
limitation (real main-feature playlists are uniformly 3D).
Triage of a 10-lens code audit of the 3D branch. Fixes for real defects;
rejected three spec false-positives that matched the ISO/IEC 14496-15
§7.6.2 record verbatim.
Robustness / correctness:
- Never panic when a title's only video is the MVC dependent view: the
base is now the first NON-dependent video, so a dependent-only title
sets up no merge (muxed as an ordinary track) instead of hitting an
`expect` on the skipped track slot.
- Drop a per-frame BlockAdditional (BlockAddID=2) when the track declared
no mvcC mapping (dependent params not captured before the header) — a
plain block keeps the file conforming instead of an orphaned add.
- A non-keyframe MVC base frame always carries a ReferenceBlock (fall back
to a 0 offset in the pre-first-keyframe corner) so it is never mistaken
for a seek point.
- Reference the last keyframe on the PRIMARY video track only, so a
secondary video track's keyframe can't become a cross-track reference.
- dep_by_pts overflow: bound BEFORE inserting so the just-arrived
dependent survives the drift-clear; count a displaced duplicate-PTS
dependent as an orphan instead of losing it silently.
API / docs:
- Fold write_frame_with_additional into write_frame(..., Option<&[u8]>)
per the "no foo_with_X" convention.
- Fix mvc_params doc (StereoMode is intentionally not emitted); remove a
stale PAT/PMT comment describing an approach that was never taken.
Tests: MVCDecoderConfigurationRecord over-length guards; write_int minimal
two's-complement widths; BlockGroup/BlockAdditions/BlockAdditional +
ReferenceBlock emission; additional dropped without a mapping; h264 MVC
passthrough keeps param sets in-band; extract_mvc_params no-panic on
truncated/empty input; pairing window + dep-overflow edges; no-panic on a
dependent-only title.
Fold the MVC dependent (right-eye) view into the base H.264 track as a
per-frame BlockAdditional under an mvcC BlockAdditionMapping, so a 3D
title produces one MVC video track instead of two independent H.264
tracks.
- h264: MVC-passthrough parser mode keeps the dependent view's subset
SPS/PPS in-band, so each emitted frame is a self-contained dependent
access unit for a BlockAdditional
- resolve: route the dependent stream through the passthrough parser
- mkvstream: detect the dependent view, pair it to the base frame by
PTS (bounded FIFO), attach it as a BlockAdditional (BlockAddID=2),
and skip building its own track; build the mvcC
MVCDecoderConfigurationRecord from the captured subset SPS/PPS and set
it on the base track at activation
- mkv: emit the mvcC BlockAdditionMapping and BlockGroup/BlockAdditions,
with a ReferenceBlock on non-keyframe base frames
- ebml: add BlockAdditions/BlockMore/BlockAdditional/BlockAddID/
BlockAddIDValue/ReferenceBlock elements and a signed-int writer
Verified against a Blu-ray 3D ISO: ffprobe shows a single MVC track,
the mvcC mapping is present, ~144k BlockAdditionals carry the dependent
view (8.7 GB), and the base view decodes cleanly with no regression.
MVCDecoderConfigurationRecord follows ISO/IEC 14496-15 7.6.2; StereoMode
is intentionally omitted (no enum value describes MVC-in-BlockAdditional;
the mvcC mapping is the primary 3D signal per RFC 9559).
Blu-ray 3D as a mux-path variant:
- Detect 3D (a clip has STREAM/SSIF/<clip>.ssif — note the SSIF/ subdir, which
the old .ssif fallback path got wrong) and use the SSIF extents for the mux:
one transport stream carrying both the base (left) and MVC dependent (right)
eyes on distinct PIDs, so muxing it captures the full 3D.
- Add the dependent-view video stream. The base STN table lists only the left
eye; the on-disc PAT/PMT are AACS-encrypted (unreadable pre-key) and the base
STN omits the dependent view (it lives in the MPLS STN_table_SS), so use the
BD-3D PID convention: dependent = base video PID + 1 (0x1011 -> 0x1012),
coding 0x20 -> H.264. The demux routes its packets from the SSIF.
info now reports two video streams for a 3D title; the mux writes both eyes.
from_coding_type maps 0x20 (MVC dependent view) to Codec::H264, so the existing
PAT/PMT scan surfaces the SSIF right-eye substream as a second video stream on
its own PID instead of dropping it as Unknown. No new parser: the dependent eye
rides the same demux path as any other TS video stream. Removes the throwaway
3D-structure probes (findings captured in prior commit messages).
First step of Blu-ray 3D as a mux-path format variant: name stream_type 0x20
(H.264/MVC dependent view, the SSIF right-eye substream). Recognition in the
scan/demux + dependent-stream enumeration follow.
Empirical (300, clip 00042): base .m2ts (1567 extents, 25.85 GB) is entirely
inside the SSIF LBA span; dependent = SSIF − base = 1567 ranges, 9.59 GB
(exact vs 35.43−25.85). 200/200 sampled dependent-view units decrypt under the
base UK. => de-interleave needs no SSIF parsing (complement of the base extents
we already resolve), and the dependent eye needs no extra key.
Dev tool for the 3D build: uses the library UDF reader (7z cannot parse these
ISOs) to dump base .m2ts vs SSIF sizes and the MPLS ExtensionData entries
(STN_table_SS + SS sub-path). Not product code.
read_icb_extents parsed a continuation block's allocation descriptors from
offset 0, but a continuation block begins with a 24-byte Allocation Extent
Descriptor (ECMA-167 4/14.5): 16-byte descriptor tag + prev-loc(4) +
length_of_allocation_descriptors(4); the real ADs start at offset 24. Reading
from 0 mis-parsed the AED tag as a descriptor -> one garbage extent, then an
unknown-type break, silently truncating any file whose extents spill into a
continuation block.
Few-extent files store their ADs inline and never hit this path, so it stayed
hidden; a heavily fragmented file (~1600 fragments) truncated to 113 extents +
a bogus non-unit-aligned fragment that then aborted the mux
(ExtentNotUnitAligned). Skip the 24-byte AED header, read l_ad bytes of
descriptors from offset 24. Adds a regression test with a non-zero AED tag.
Add UnitKey.variant_number (0 = ordinary, 1..32 = forensic variant) with new/variant constructors, and aacs::variant_select — resolve a disc's single variant and classify each aligned unit (default / variant / drop foreign / conceal keyless). Correct IndividualSegment.tbl: the per-record field is the variant (cycles 1..32 on a retail disc), not a segment number — Segment.number -> Segment.variant.
Relicensed from AGPL-3.0 to MIT, effective 1.3.1 (<=1.3.0 remain AGPL). The CSS
content cipher and Stevenson title-key attack are attributed to their published
cryptanalysis (not libdvdcss); all libaacs/libbluray/libdvdread/libdvdnav name
references were dropped from comments while keeping the standard format/spec
descriptions. Also bumps to 1.3.1.
Parse ADV_OBJ/VPLST000.XPL (the real HD-DVD player playlist) with roxmltree
into one DiscTitle per <Title>: its PrimaryAudioVideoClip clips in order (EVO
via the .MAP sidecar), the titleDuration, the displayName, and the ChapterList.
A layer-break split (FEATURE_1+FEATURE_2, feature/feature_Divide) composes into
ONE title with the two parts as clips, each carrying its title-time in/out
points (45kHz ticks) for seamless-join splicing. Falls back to the clip-name
heuristic when no playlist is present.
Fixes unknown runtimes (real durations), poor names (FEATURE -> 'Main Movie'),
and gives authoritative composition + chapters. Validated on ANCHORMAN (97m),
SHAUN (99m), HARRY_POTTER (152m).
libfreemkv is publish=false (it git-deps the firmware crate freemkv-unlock,
never on crates.io), so the Release workflow's 'cargo publish' step failed
hard on every tag. Consumers git-tag-pin libfreemkv; the git tag is the
release artifact. Removed the publish job.
Round-11 findings from the 10-phase release audit (no real HIGH):
- When a no-declared-duration source (HD-DVD) muxes a degenerate single
frame at tick 0 with no per-frame duration, max_block_ticks stays 0 and
the reserved DURATION placeholder was left as a literal 0.0 (players read
that as a zero-length file). Void the element instead, so the Segment
omits DURATION as an unknown-duration source did before the back-patch.
- Add a regression test for the sniff_video_codec overlap fix (a
picture_start_code whose payload begins 00 00 followed by a real start
code) so the i+=4 marker skip can't silently regress to i+=3.
Round-10 findings from the 10-phase release audit:
- A finder claimed the DTS SFREQ→rate table was wrong at 11/12; verified
it against ffmpeg's avpriv_dca_sample_rates (12k/24k/48k/96k/192k at
11-15) — the table is CORRECT. Added a test that locks the full table so
it can't be mis-"fixed".
- sniff_video_codec advanced 3 bytes after a matched start code, re-reading
the code byte as an overlapping start code; skip the full 4-byte marker.
- Guard the HD-DVD next_id title counter with saturating_add so a crafted
disc with >65536 clips can't overflow (panic in debug).
- Add a test that an undecryptable unit (DecryptFailed) is zero-filled and
counted as loss through ExtractResult (complete=false, bytes_lost>0) —
the recovery-seam consolidation folded that bucket into bytes_unreadable.
Round-9 findings from the 10-phase release audit (no HIGH):
- Remove the MAX_PTS_MARKS backstop and its tautological test: an empty
DTS PES returns before recording a mark, and a non-empty run is already
bounded by the MAX_AU_BYTES buffer clear (which clears pts_marks) — so
the deque cannot grow unbounded and the cap was dead code.
- AuAssembler::for_codec no longer reserves 256 KiB for a Passthrough
stream (audio/subtitle, and every TS/BD stream) whose buf is never
written; only the reassembling modes reserve.
- Correct the scan comment that claimed region is computed (it is a
Region-free stub until region detection lands) and drop a public-repo
reference to internal "private refactor notes" in the mkb module doc.
Round-8 findings from the 10-phase release audit:
- detect_disc_format's BDMV fallback passed detect_format's result
through unchanged, so an SD bonus/menu title could tag a BD-tree disc as
DVD (mis-sizing the ECC sweep) — violating its own "never below Blu-ray"
invariant. Clamp anything but UHD up to Blu-ray.
- Track the MPEG-2 GOP byte total incrementally instead of re-summing the
whole gop_buf on every pushed picture (was O(pictures²) on any MPEG-2
disc, not just adversarial input).
- Back the DTS pts_marks deque with a VecDeque so the over-cap prune is an
O(1) pop_front, not an O(n) Vec::remove(0).
- Add a test exercising parse_stream_id_extension's PTS/DTS skip branches
(the real AU-opening 0xFD video PES path) — previously untested.
Round-7 findings from the 10-phase release audit (no HIGH; convergence):
- Cap DtsParser.pts_marks (MAX_PTS_MARKS): a run of zero-length timed PES
packets grew no buffer bytes, so the drain_front mark-prune never ran —
the deque could accumulate without bound on hostile PS input.
- detect_disc_format tested HVDVD_TS before BDMV while the title-scan
dispatch tests BDMV first, so a disc with both trees would be classified
HD-DVD but enumerated as Blu-ray. Align both to BDMV → HVDVD_TS →
VIDEO_TS.
- Document why the DTS new-PES re-base can emit a locally-decreasing PTS
(the muxer's block_ts applies the strictly-monotonic audio nudge, tested
in mkv.rs) — this is by design, not a mux defect.
- Fix stale aacs/keys.rs comment references (functions moved to
aacs/inf.rs / aacs::resolve/derive in the module split).
Round-6 findings from the 10-phase release audit:
- Wire the documented MAX_PENDING_BYTES byte cap into the MPEG-2 GOP
buffer (it was dead code) and add an equivalent MAX_GOP_BYTES cap to the
sparse-PTS reorder, so a crafted stream of few-but-huge access units
cannot over-allocate — both were bounded only by frame count before.
- probe_evo_streams defaulted an unsniffable HD-DVD video stream to H.264,
which mis-parses a VC-1 (or still-encrypted) clip into a corrupt track.
Emit the video stream only when the codec is actually identified — the
honest outcome, matching the audio path (a real clear clip always carries
its sequence header at the head).
- Resume the AU-opener search from a cursor (like the boundary search), so
a long unsynced junk run is O(bytes), not O(buffer) per push.
- Mark mpeg2's now-dead MAX_AU_BUFFER test-only; restore #[doc(hidden)] on
the aacs probe harness module.
- Add regression tests: the 0xFD video-routing guard, the FMTS-is-UHD key
state, and the GOP byte caps.
Round-5 findings from the 10-phase release audit:
- collect_es routed EVERY extended-stream-id (0xFD) PES into the video ES
buffer, so a 0xFD HD-audio sub-stream (MLP/TrueHD) could pollute the
video sample and — if it preceded the video PES — stamp the video track
with the audio PID, losing the video. Only the VC-1 extension (0x55) is
now treated as video; routing 0xFD audio to its own track is deferred to
the HD-DVD program-chain follow-up.
- The sparse-PTS reorder now carries its calibrated per-frame duration onto
each frame, so the muxer emits a BlockDuration and the back-patched
Segment Duration covers the final frame instead of understating it.
- Add regression tests for the MAX_MARKS and MAX_VTI_HITS caps (promote
MAX_VTI_HITS to module scope); make the differential-test factory array a
named type; drop an identity-op in a reorder test.
Round-4 findings from the 10-phase release audit (the first fully clean
round; it dug into the new #22/#18 refactor code):
- AuAssembler closed each AU from only the FRONT mark's fields, so when
one PES fragment carried the source and a later fragment of the same AU
carried the PTS, the second field was dropped — a regression vs the old
separate pts/source mark deques. Now merge the first Some of each field
across all in-range marks.
- parse_vti_clip_order picked the largest residue bucket with
HashMap::into_values().max_by_key(), nondeterministic on a size tie
(randomized HashMap iteration) — could select a different clip table
run-to-run. Break ties by smallest offset.
- Bound the marks/disc_marks deques (MAX_MARKS): the buf-size cap prunes
marks only when bytes accumulate, so a run of zero-length timed
fragments could grow them without bound on hostile input.
- Add push_owned so the PS path moves the PES payload into a passthrough
AU with no copy (MPEG-2 video + all audio), removing a per-PES
malloc+memcpy the refactor had introduced on the DVD path.
- Back-patch the MKV duration from the block END (start + its own
duration) so it covers the final frame instead of understating by one.
- Add direct tests for the MKB record-framing walker; drop a stale
drain_complete_aus doc comment left on process_au.
AuAssembler::drain rescanned the whole buffered access unit from a fixed
offset on every push, so reassembling one AU split across N program-stream
PES fragments cost O(bytes²/fragment) — amplified on the HD-DVD PS path
where H.264/HEVC/VC-1 frames are large and now flow through this shared
assembler (unlike the TS path, which delivers one AU per PES).
Carry a scan_pos cursor (and, for the stateful VC-1/MPEG-2 rules, a
seen_unit flag) so each push resumes the boundary search where the last
one stopped instead of restarting. Total scan work for one AU is now
O(AU bytes). The from-scratch scanners are retained as a #[cfg(test)]
oracle; a new differential test asserts the resumable path yields
byte-identical AUs at every fragment granularity for all three modes.
Round-2 findings from the 10-phase release audit:
- parse_vti_clip_order bucketed hits by residue with an O(stride*hits)
rescan and no hit cap, so a crafted HD-DVD VTI packed with millions of
`.EVO` tokens (up to the 64 MiB UDF read cap) could burn seconds of CPU
on a routine scan. Bucket in a single O(hits) pass and cap collected
hits at MAX_VTI_HITS (a real table holds a few dozen).
- Fix the stale `super::keys::…` intra-doc links left by the aacs module
rename: the referenced fns live in `super::derive`.
Round-1 findings from the 10-phase release audit:
- SparsePtsReorder buffered its current GOP with no bound, draining only on
a keyframe — an open-GOP or crafted program stream that never signals one
could hold the whole title in RAM. Force-complete the GOP at
MAX_GOP_FRAMES, matching the MPEG-2 parser's backstop.
- inject_unit_keys labelled a 2.1 FMTS disc as AACS 1.0 / bus-encryption
off; FMTS is UHD-family, so synthesize the UHD version + bus encryption.
- The compiled Key Correction Data was a non-zero 16-byte constant fed into
the Media Key derivation. Per the no-compiled-keys rule it is now all-zero;
the chain still cannot complete on a real disc (documented), so this is
behaviour-neutral — all variant tests pass unchanged.
- Fix stale doc references (broken `super::variants` intra-doc links, and
`aacs::keys` comments) left by the module rename.
A title whose scan yields no duration (HD-DVD — its `.MAP` timemaps are
not parsed, so DiscTitle.duration_secs is 0) produced an MKV with no
Segment Duration element, so players/MediaInfo reported an unknown
runtime.
Reserve a DURATION placeholder when the source declares none, track the
highest block timestamp written, and back-patch the placeholder at
finish() with the real muxed runtime (also enabling the per-track BPS
tags for these titles). Gated on duration_secs == 0, so BD/UHD/DVD —
which carry a real mpls/IFO duration — write it up-front exactly as
before, unchanged.
HD-DVD Standard Content splits the main feature across clips at the
layer break (FEATURE_1/FEATURE_2, or feature/feature_Divide). The scanner
enumerated one title per .evo, so main-title selection picked only part 1
(e.g. Shaun's 11 GB FEATURE_1, missing the 6.8 GB FEATURE_2).
Parse the HVA*.VTI navigation file's clip table — a fixed-stride record
list naming every clip in authored order, isolated by residue-mod-stride
rather than the imprecise header pointer — and concatenate the feature
clips (matched by the feature* naming convention) into one title whose
extents run in authored order. Every other clip stays its own title.
Falls back to one-title-per-clip when the VTI is absent or unparseable,
so nothing regresses on a disc with no readable navigation.
Validated on real discs: Shaun 17.8 GB / Anchorman 20.1 GB / Harry Potter
24.4 GB now enumerate as one 2-clip FEATURE title (largest = the movie).
The MPEG-2 parser hand-rolled its own PES reassembly — a byte buffer plus
parallel PTS / source / discontinuity mark queues keyed by absolute
offset — duplicating what AuAssembler already does for H.264/HEVC/VC-1.
Add a Mode::Mpeg2 to AuAssembler (picture 0x00 with preceding sequence
0xB3 / GOP 0xB8 headers — the same headers-precede-picture shape as the
VC-1 mode) and have the MPEG-2 parser own one via AuAssembler::mpeg2().
parse() now feeds fragments to the assembler and processes each complete
access unit; the buffer, base offset, and three mark queues are gone. The
GOP-buffered temporal_reference reorder and PTS origin-locking are
unchanged. The parser's external contract is unchanged, so all existing
MPEG-2 parser tests pass as-is; new AuAssembler tests cover the MPEG-2
boundary rule directly.
HD-DVD EVO (and DVD VOB) program streams timestamp video at GOP
granularity: only one access unit per GOP carries a PES PTS. The H.264 /
HEVC / VC-1 parsers collapsed a missing PTS to 0, so on such a source
every non-anchor frame landed on the same block timestamp and a decoder
reported "non monotonically increasing dts".
Add a shared SparsePtsReorder that rebuilds a display-order PTS per frame
from the coded picture type (I/P/B) plus the sparse anchor PTS, with a
per-frame duration self-calibrated from the spacing between consecutive
GOP anchors (no external frame-rate needed). Display order is derived via
the classic single-anchor-delay rule (an anchor displays only after the
previously-held anchor; a B displays immediately), exact for the
non-hierarchical GOP structures HD-DVD H.264/VC-1 use. It mirrors the
MPEG-2 parser's GOP-buffered origin-locking.
Gated to the program-stream path only: the three parsers enable it via
with_ps_reorder(is_dvd_ps), so the BD/UHD transport path (per-frame PTS)
is byte-identical and untouched.
Correct the variant Media Key chain against two real variant MKBs
(Zombieland v70, Stand By Me v70): C for Kmp is the per-slot block of
the 0x0c cvalue table indexed by the matched subset-difference, not the
0x2d head. The 0x2d record is the VARIANTS table (leading body-16 bytes,
one big-endian u16 per subset-difference) followed by a trailing 16-byte
Nonce, with no leading header. VKD stays at 0x2f.
Route record-type selection through the named REC_* consts in mkb rather
than bare hex, and document that Key Correction Data is per-licensee: no
universal constant exists and none is compiled in, so on a real disc the
chain yields a wrong Media Key that the Verify-Media-Key gate rejects
rather than emitting a bad key. This is a key-acquisition gap, not a
code gap.
VC-1 HD-DVDs (e.g. Shaun of the Dead) carry video on MPEG-PS extended
stream id 0xFD, with the real stream selector in stream_id_extension
inside the PES extension. Parse that field so the video routes to a
distinct track (pid 0xFD00|ext) instead of being dropped.
Reframe VC-1 access units in AuAssembler with a dedicated Mode::Vc1:
an AU is delimited by the next frame BDU (0x0D) once a frame has already
been seen, so the sequence (0x0F) and entry-point (0x0E) headers that
precede an I-frame stay attached to the frame they describe. The old
single-start-code split stranded those headers on the prior AU, which
the decoder reported as bits-overconsumption and hard decode failures.
hddvd probe now tracks the video pid it detects and emits VC-1 on 0xFD.
Bump to 1.3.0 and add the 1.3.0 changelog entry (FMTS/HD-DVD formats, AACS 2.1 variant chain, recovery seam, aacs module split, main-title-by-size, and the fixes since v1.2.2). Not tagged or pushed.
- aacs/resolve: a media-keys-only provider missing the VID classifies as
VidUnavailable, not NoMaterial (an MK derives the VUK once the VID
arrives).
- disc/bluray: mark a clip seen only after its .clpi parses, so a
transient parse failure on the first PlayItem cannot suppress the
clip's extents for a later PlayItem referencing it that succeeds.
- disc/patch: log rather than swallow mapfile record/flush failures on a
reverify downgrade, so a failed persist cannot silently mismark a bad
unit good on resume.
- mux/ts: flag a discontinuity when a partial PES is dropped, matching
the other partial-drop paths.
- mux/demux_thread: the no-demuxer branch forwards an empty batch for
early consumer-disconnect detection instead of reading the whole disc.
- io/pipeline: correct the send-timing log (as_secs_f64, not as_micros
printed as ms).
- aacs/derive, aacs/variant, disc/read_error, keysource: comment/doc
accuracy. sector/prefetched, udf: remove dead fields/functions.
- mux/disc: assert unit-aligned read counts in the test.
Replace the AACS-specific inline key-fetch in the decrypt decorator with
a scheme-neutral recovery seam: the input stream (L3) installs a Recover
closure (none / AACS key-fetch) and the decorator (L2) runs it at the
single decrypt-miss point. FMTS (AACS 2.1) forensic-segment units that no
key opens are just undecryptable units, concealed and counted as ordinary
decrypt loss with no FMTS-specific branch ("a loss is a loss"), so the
separate bytes_undecryptable bucket collapses into one loss count.
- sector/recovery.rs: the seam (MissOutcome, none/key_fetch factories),
naming no encryption scheme in its type.
- FMTS: segment routing primitives + BYPASS_FMTS_KEY, and an upfront
ensure_forensic_segments_decryptable gate (Error::FmtsKeyMissing) in
the mux input path, parallel to the unit-key gate.
- CSS descramble/rekey moves from decrypt_sectors into
css::descramble_region: CSS self-recovers from the data itself, so it
stays OFF the seam (which is only for external inputs).
- disc/mod.rs also: main-title selection aligned to largest physical
size; is_regular read from the open file handle, not metadata(path),
fixing a swallowed sync_all on a fresh-rip ISO. decrypt_threads()
resolved once via OnceLock off the per-buffer hot path.
- DetectFn now takes a SectorSource so a parser can inspect a jar's
central directory in detect() instead of firing on "any BD-J jar".
dbp/deluxe do the real com/<vendor>/ prefix check up front, so each
claims only its own discs (foundational for scaling the registry).
- criterion: treat a stream-map value of 0 as unmapped and synthesize a
real 1-based number, so a 0 can't shadow or collide with a genuine
stream 1 (with regression tests).
- png_filenames: new Low-confidence, last-resort parser reading menu
language from {title}_UHD01_{LANG}_Composite artwork; sits below the
MPLS floor so a real framework parser always wins.
- vocab: add menu_lang() for 639-2/B to 639-2/T menu-token normalization.
New aacs::segment_key: parses the AACS 2.1 SegmentKeyNNNNN.tbl container, confirmed against a retail disc as an 8-byte header + 65536 records of 536 bytes, indexed by the 16-bit variant selector the Media Key Variant chain produces. This is the confirmed link between the two 2.1 variant layers (selector picks the device's per-segment variant). The per-record 528-byte payload layout is not yet reversed.
New aacs::segment: parses the AACS 2.1 IndividualSegment.tbl into the source-packet ranges of the forensic variant segments (validated against a retail disc: 792 segments, 2560 packets each). First piece of the FMTS variant decoder — the segments' variants are encrypted under segment keys, not the unit key, so a unit-key rip corrupts them (broken HEVC refs).
canonical_title_order keyed on clip-count ascending before duration, so a short 1-clip bonus reel outranked the real feature — which is often chaptered into many clips (one per chapter). Make physical size the primary key: the main feature is the largest non-oversize title; decoy 'play-all' playlists run long but tiny (reused clips) and self-eliminate.
Validated across 23 UHD/BD discs: fixes t1 on F9, Fast Five, Fast & Furious 6, and Furious 7 (feature was ranked #13-36); no regressions on the 19 already correct.
Add DiscFormat::Fmts (AACS 2.1) and DiscFormat::HdDvd as first-class peers. Format derives from the AACS MKB generation (mkb_type().generation(): V10=BD, V20=UHD, V21=FMTS), reusing existing AACS code, and from the on-disc tree for HD-DVD/DVD. One detector (detect_disc_format) shared by the coarse DiscId probe and the full scan — no more 'default BluRay, defer to full scan'.
FMTS is a BD-tree stream variant: parse_playlist resolves the clip stream via CLIP_STREAM_EXTS (.m2ts -> .fmts -> .ssif), so the .fmts main feature yields real extents (previously silently empty). HD-DVD is a tree-level peer with its own enumerator (disc/hddvd.rs): HVDVD_TS/*.evo -> MpegPs titles with real extents (playlist/stream parsing honestly stubbed).
Sample selection for key resolution now uses the authoritative AACS CPI flag (aacs_unit_encrypted, byte-0 & 0xC0) not the ts_sync_destroyed heuristic — container-agnostic (M2TS/FMTS/EVO; TS-sync is meaningless on HD-DVD program streams) and stops the key-service '0 encrypted units' rejection.
Tests live with each format (bluray/hddvd/mod); generic UDF fixture builders extracted to a shared udf::fixture module.
Adopt freemkv-unlock's split Unlocker trait: run_features drives the drive-prep capability, run_bus the content bus removal, each iterating unlockers until one doesn't decline (NotApplicable = try next; Ok or a real error stops).
unlocker_matrix now reports which drive-prep unlocker actually ran — LibreDrive removes the bus at the drive; Renesas unlocks features but leaves the bus to the cert. Wire product_id through to fu::DriveId.
Bump to 1.2.3.
aacs::boil was a thin newtype veneer over aacs::derive — a duplicate layer.
Delete it: move the shared key newtypes (Vid, MediaKey, Vuk, ProcessingKey,
UnitKey) into aacs::types, and expose resolve_candidate + KeyCandidate from
aacs::derive directly. Downstream (keysource, disc::encrypt) now import from
aacs::{derive,types}. Pure API consolidation, no behaviour change; full test
suite green on Rust 1.86.
The module structure is the public API, but the typed key primitives
(MediaKey/UnitKey/Vid/Vuk, mk_from_dk/mk_from_pk/uk_from_vuk/vuk_from_mk),
derive_vuk, the aligned-unit decrypt entry points, and DeviceKey/HostCert
are load-bearing names that downstream key-source crates import through the
flat `aacs::` path. Re-export them here so those crates can track the
module refactor without a lockstep re-pin.
Redesign derive_media_key_variant to the minimal derivation surface:
derive_media_key_variant(mkb_records, pk) -> Km
- PK-input only. Deriving Kp from device keys (DK -> PK) is a separate
concern (walk_processing_key); a leaked 2.1 key is a PK, and the chain
starts at Kp. A bare PK arrives without its subset-difference slot, so
the primitive tries it against every slot and returns the Km for the
slot that passes the MKB's Verify-Media-Key record -- mirroring the
classical bare-PK derive_media_key_from_pk, gated by the chain's own
verify so an unverified key is never returned.
- VID-free: the Media Key is MKB-scoped. VUK stays the separate
derive_vuk(Km, VID) step.
- KCD is a fixed algorithm constant compiled in, not a caller parameter
(removes the kcd argument, the placeholder const, and KcdNotProvided).
- Soft-correction / online-challenge slots are treated as non-covering;
surfaced over the generic miss so a disc needing those modes is
distinguishable from a non-covering key.
resolve_keys_v21 updated to walk DK -> PK first, then call the primitive
and derive the VUK from Km + VID. Module + helper docs refreshed to the
pinned record layout; tests reworked for the PK-only signature.
variants_for_uv previously returned None (placeholder), dead-stopping the
Media Key Variant chain at VariantsTableUnavailable on any real disc. Layout
now pinned against two real 2.1 variant MKBs (Zombieland v70, Stand By Me v70):
the 0x2d Encrypted-Media-Key-Variant-Data body is sd_count u16 VARIANTS entries
(1:1 with 0x0c cvalues / 0x04 subset-diffs) followed by the 16-byte per-disc
Nonce at the tail. variants_for_uv reads the sd_slot_index-th u16.
With this the chain runs fully: Kmp -> Kpnew(=Kmp^KCD, extracted CyberLink
constant) -> VKD(0x2f) -> Km -> Kvu. The only remaining input is a covering
2.1 Processing Key to validate against a known answer; until then the final
Verify-Media-Key (0x86) gate rejects any wrong layout pick, so a bad key can
never be emitted. Tests updated to reflect the wired lookup; 1.86 precommit
green (fmt+clippy+tests).
The facade removal (5cf4252) left tests/ calling aacs::disc_hash,
aacs::decrypt_unit, aacs::AacsVersion, etc. at the old flat paths,
so the branch did not compile its integration tests. Repoint each
to its real module (inf/derive/content/mkb). No logic change.
The volume_key module was only 34 lines and is just the tail of the same
DK/PK -> MK -> VUK -> UK derivation ladder as media_key. Fold both into one
derive module so every aacs module is a substantial, distinct responsibility
(crypto/mkb/derive/inf/content/variant/resolve). Relocation only; logic hash
identical (95fb9924); 2210 tests green.
Break the 2800-line keys.rs into four responsibility-scoped modules:
- media_key.rs: DK/PK -> Media Key subset-difference walk (+ probe harness)
- volume_key.rs: VUK derivation, unit-key unwrap
- inf.rs: Unit_Key_RO.inf parsing, disc_hash, content cert, in-drive MKB read
- resolve.rs: the resolve_keys_* orchestration (keys.rs renamed)
Relocation only; the (white-box) test suite stays in resolve.rs and pulls
the moved items via glob imports. Proven byte-identical to the pre-refactor
state via the logic hash (95fb9924); 2210 tests green.
Relocate the shared MKB machinery into a single mkb module: the record
framing walker + MkbRecord view (from variant), the MkbType/AacsVersion
classification, the MKB-file utilities, and the record-body finders (from
keys). Fixes the inversion where the MKB parser lived in the 2.1-only
variant module. variant.rs keeps its local MkbRecord-based mkb_find_mk_dv
(name collision with the raw one; unified in the dedup follow-up).
Relocation only. Proven byte-identical to the pre-refactor state via the
function-body logic hash (imports normalized out); 2210 tests green.
Relocate the shared low-level primitives into a single crypto module:
aes_ecb_encrypt/decrypt, aes_cbc_decrypt, aes_g (from content/variant) and
aesg3 + AESG3_SEED (from keys), plus AACS_IV. Fixes the scatter where AES-G
lived in the 2.1 file and AES-G3 in keys. Relocation only — no rename, no
logic change (logic-hash identical to baseline; 277 items; 2210 tests green).
Pure file+module-path rename. 'content' names the AACS unit-decrypt layer
(distinct from the top-level sector-decrypt driver crate::decrypt), and
'variant' (singular, spec term 'Media Key Variant') names the 2.1 chain.
Logic-hash identical to baseline; 277 items intact; tests green.
Add [C]/[PR]/[BD]/[libaacs] §x.y provenance markers across the AACS
crypto so each primitive links to the spec section it implements, with a
source-tag legend in mod.rs. Doc-comments only — no logic, constant, or
signature changes.
Also: correct two stale record-type comments in variants.rs (0x82/0x83 →
the real 0x2d/0x2f) and document the Variant Number width (spec lsb_10 vs
the 2.1 chain's lsb_16, driven by the 65,535-entry VKD table).
@@ -36,7 +36,7 @@ This Code of Conduct applies within all community spaces, and also applies when
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at owner@example.invalid. All complaints will be reviewed and investigated promptly and fairly.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported privately to the project maintainer via GitHub at https://github.com/MattJackson. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Drive-level unlocking is handled internally; consumers work with disc access and decryption only.
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, and content reading in one crate. Drive-level unlocking is handled internally; consumers work with disc access and decryption only.
DVDs (CSS) decrypt out of the box. Blu-ray and UHD (AACS) require a `keydb.cfg` (default `~/.config/freemkv/keydb.cfg`) supplying disc-specific volume unique keys; no AACS key material is compiled in.
DVDs (CSS) decrypt out of the box. Blu-ray and UHD (AACS) require disc-specific volume unique keys, supplied by the consumer (e.g. via [freemkv-keysources](https://github.com/freemkv/freemkv-keysources), which owns `keydb.cfg` lookup/download); libfreemkv itself never reads `keydb.cfg` or downloads keys, and no AACS key material is compiled in.
**12+ MB/s** sustained read speeds on BD. Drive prep (`init()`) handles unlocking internally via the `freemkv-unlock` crate — clients never see it; when no drive unlock applies, the library rips via the host-certificate AACS handshake.
Multi-lingual by design — the library outputs structured data and numeric error codes, never English text. Build any UI or localization on top.
Streams implement a single unified `pes::Stream` trait (re-exported as `PesStream`) exposing `read()` and `write()` on one type. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Inputs implement `PesSource` (`read()`) and outputs `PesSink` (`write()`); a format that is both implements both. `open_source()` / `input()` open every input URL (`disc://` included) and `output()` every output URL. All URLs use the `scheme://path` format — bare paths are rejected.
### Keys
DVDs (CSS) decrypt out of the box, with no external key file needed.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg`(or passed via `ScanOptions`). No AACS key material is compiled into the binary.
Blu-rays and UHD (AACS) require Unit Keys passed in via `ScanOptions`/`KeySpec`. libfreemkv does not read `keydb.cfg` or fetch keys itself — that's the consumer's job (see [freemkv-keysources](https://github.com/freemkv/freemkv-keysources)). No AACS key material is compiled into the binary.
├── Demux — TS / PS → PES (one demux for every sector source)
├── PesSource/PesSink — read()/write() PES frames
├── MkvStream — MKV mux/demux
├── M2tsStream — BD transport stream
├── NetworkStream — TCP with FMKV metadata header
@@ -158,7 +133,8 @@ Streams — unified PES pipeline
└── NullStream — discard sink
```
See [docs/](docs/) for detailed technical documentation on each module.
Build API documentation with `cargo doc --no-deps --open`. The public
[FVI format specification](https://freemkv.org/docs/fvi-format/) describes exported video indexes.
## Error Codes
@@ -188,6 +164,53 @@ All errors are structured with numeric codes. No user-facing English text — ap
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to capture your drive's identity for contribution. Drive-unlock profiles are maintained in the [freemkv-unlock](https://github.com/freemkv/freemkv-unlock) repository.
## What this is NOT
To keep expectations honest:
- **No bundled AACS or CSS keys.** No AACS key material is compiled into the
crate. Blu-ray/UHD unit keys are supplied by the consumer at runtime (e.g. via
@@ -139,10 +139,10 @@ If your machine has a free SATA port, use it.
### How freemkv Handles This
freemkv uses a three-layer recovery model. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for full details.
freemkv uses a three-layer recovery model.
- **Pass 1 (Disc::copy):** Fast sweep with 64 KB reads. On failure, zero-fills the block and skips forward. Writes a ddrescue-format mapfile for later retry.
- **Pass 2+ (Disc::patch):** Targeted re-reads of bad ranges with a long 30-second timeout per CDB. The drive firmware performs its own ECC and laser power retries within that window.
- **Pass 1 (`freemkv_engine::recovery::sweep`):** Fast sweep with 64 KB reads. On failure, zero-fills the block and skips forward. Writes a ddrescue-format mapfile for later retry.
- **Pass 2+ (`freemkv_engine::recovery::patch`):** Targeted re-reads of bad ranges with a long 60-second timeout per CDB. The drive firmware performs its own ECC and laser power retries within that window.
- **In-stream (DiscStream):** Adaptive batch halving -- reduces request size on failure to isolate bad sectors within a larger block.
This means a disc with some bad sectors will still produce a usable ISO. The damaged areas are zero-filled in pass 1 and retried in subsequent passes. Structure-protected sectors (deliberate unreadable regions from copy protection) will never yield, which is expected.
@@ -164,7 +164,7 @@ Workarounds:
- **Accept the partial ISO.** freemkv's skip-forward recovery will zero-fill the unreadable blocks and continue. The resulting ISO may be playable with minor glitches in the affected areas.
- **Physical replug between retry passes.** If running multi-pass patch, replug the drive between passes to clear the taint state.
freemkv deliberately does not attempt inline SCSI resets or eject cycles to recover from this state, because those operations were found to make the problem worse on affected hardware (see the design rationale in [`docs/rip-recovery.md`](docs/rip-recovery.md)).
freemkv deliberately does not attempt inline SCSI resets or eject cycles to recover from this state, because those operations were found to make the problem worse on affected hardware.
---
@@ -302,7 +302,7 @@ Many external drive enclosures (Vantec NexStar, Sabrent, OWC, etc.) do not adver
When something goes wrong during a rip, gather this information before filing an issue:
1. **freemkv version:** `freemkv --version` or the crate version in `Cargo.toml`.
2. **Drive model:** from the drive label, or from `freemkv info`.
2. **Drive model:** from the drive label, or from `freemkv info disc://`.
3. **Connection type:** USB (with bridge chipset if known) or direct SATA.
4. **Operating system and kernel:** `uname -a`.
5. **Kernel messages during the failure:** `dmesg | tail -50` immediately after the crash.
| `src` | object | MUST | Provenance: `{ "file": int?, "sector": uint, "byte": uint }` — the offset of this AU's **first byte** in the source (§9). MUST be carried from demux, never reconstructed. |
| `type` | string | MUST | Coding type:<br>`"I"`<br>`"P"`<br>`"B"`<br>_ISO/IEC 13818-2 §6.3.9; H.264/H.265 slice types collapsed to frame type._ |
| `key` | boolean | MUST | `true` iff this picture is an intra (I) picture / parser-flagged decode-restart point (IDR / IRAP / I-picture).<br>_MPEG-2 open-GOP clean-RAP precision (`closed_gop`) is not currently distinguished — see note below._ |
| `gop` | boolean | SHOULD | `true` iff this picture begins a GOP / coded video sequence.<br>_Omitted when the implementation does not carry a distinct GOP-boundary signal._ |
| `pts` | integer\|null | SHOULD | Presentation timestamp in `timescale` ticks; `null` if unknown. |
| `dts` | integer\|null | MAY | Decode timestamp in `timescale` ticks. |
| `size` | integer | MAY | AU length in bytes; enables byte-range extraction with `src`. |
| `recovered` | boolean | MAY | `true` iff any byte of this AU came from a retried/marginal read (§9.1).<br>_Default `false`._ |
| codec ext | object | MAY | Codec-specific members under the codec's namespace (§8). |
The `type` and `key` members are **codec-agnostic** and MUST be populated for
every codec. `type` is the I/P/B coding type the parser decoded (collapsing
H.264/H.265 slice types to a frame type); where no per-picture coding is carried
(audio / synthetic frames), `type` is `"I"` for a key picture else `"P"`. `key`
is the picture's random-access flag as the codec parser sets it (IDR / IRAP /
I-picture). A writer MUST NOT emit a degraded record (`type:"?"` or `src:null`)
merely because a codec lacks per-picture coding info — those fallbacks are
reserved for a field that is genuinely unavailable (e.g. provenance absent on a
synthetic source).
> **Limitation (honest random-access).** `key` is set from the picture's
> intra / decode-restart flag. The per-picture coding model this index carries
> does **not** distinguish MPEG-2 open-GOP clean random-access points
> (`closed_gop`) from any other I-picture, so `key` is the parser-flagged
> decode-restart point, not a verified clean-RAP claim. A future revision MAY
> tighten `key` for codecs/profiles that carry that signal; readers MUST NOT
Codec-agnostic interlace/pulldown attributes, derived through the indexer's
per-picture coding accessors (MPEG-2: ISO/IEC 13818-2 §6.3.10). Emitted as
top-level members of the record, and ONLY when the codec actually measured the
signal — an OPTIONAL member that is omitted (not defaulted) when unknown:
| Member | JSON type | Req | Semantics / reference |
|---|---|---|---|
| `field_order` | string | MAY | Display field order:<br>`"tff"` — top field first<br>`"bff"` — bottom field first<br>`"progressive"` — no field order applies<br>_Omitted when the codec did not signal it._ |
| `progressive` | boolean | MAY | `true` iff the picture is progressive.<br>_Omitted when the codec did not signal it._ |
| `nb_fields` | integer | MAY | Number of displayed field periods this picture occupies (the soft-telecine / 2:3 pulldown basis):<br>`1` for a single field picture<br>`2` for a normal frame<br>`3`/`4`/`6` for `repeat_first_field` pulldown per §6.3.10 |
Codecs that carry only a coding type (e.g. H.264 / HEVC / VC-1 through this
pipeline) omit `field_order` and `progressive` rather than guessing a default.
## 8. Codec model and extensibility
Core record members (§7) are codec-agnostic and present for every codec.
Codec-specific data is either (a) promoted to top-level members for a small,
registered set per codec profile (e.g. MPEG-2 §7.1), or (b) placed under an
`ext` object keyed by codec id for richer/optional data:
```json
{
"n":42,
"type":"P",
"key":false,
"src":{
"sector":17,
"byte":924
},
"ext":{
"hevc":{
"temporal_id":0,
"nal_type":1
}
}
}
```
New codecs and members are added through Appendix B (codec registry) without a
breaking version bump, provided readers continue to ignore unknown members (§11).
## 9. Provenance and recovery semantics
`src` is **byte-exact** to the source as read. `src.sector` counts in
`source.sector_size`-byte units; `src.byte` is the offset within that sector of
the AU's first byte. For multi-file sources, `src.file` indexes a writer-declared
file list. Provenance MUST be the value observed at demux time; an implementation
MUST NOT recompute `src` by re-parsing — the point of FVI is to *carry* the truth.
### 9.1 Recovery
Because FVI is provenance-native, it can record reliability. A record with
`"recovered":true` indicates the AU's source bytes required retry/marginal-read
recovery. This lets downstream tools surface or quarantine pictures whose bytes
are not byte-identical to a clean read — a capability legacy index formats lack.
## 10. Time model
All `pts`/`dts` are integers in units of `1/timescale` seconds. `pts` is
presentation (display) time; `dts` is decode time. Records are in **coded**
(decode) order, so `pts` is not necessarily monotonic across records (B-pictures
reorder); `dts` is non-decreasing. Readers needing display order sort by `pts`.
## 11. Versioning and forward compatibility
-`fvi_version` is the document version; this spec defines `1`.
- **Additive** changes (new OPTIONAL members, new registered codecs) do NOT bump
`fvi_version`. Readers MUST ignore members they do not recognize.
- A change that alters the meaning of an existing member or makes a new member
REQUIRED bumps `fvi_version`.
- A reader encountering a higher `fvi_version` than it implements SHOULD process
the members it understands and MUST NOT reject the document solely for the
version being higher, unless a member it relies on is absent.
## 12. Conformance requirements (summary)
A conformant **writer** MUST: emit a Header first; emit records in coded order
with contiguous `n`; populate `src` from demux; use named/registered codec ids;
encode one JSON value per UTF-8 LF-terminated line.
A conformant **reader** MUST: accept any §4–§10 document; ignore unknown members;
not assume `picture_count`, `pts`, or `size` are present unless required above.
CLPI (Clip Information) files describe the structure of individual M2TS transport streams on a Blu-ray disc. Each `.clpi` file in `BDMV/CLIPINF/` corresponds to one `.m2ts` file in `BDMV/STREAM/` with the same numeric name (e.g. `00001.clpi` describes `00001.m2ts`).
The most important data in a CLPI file is the **EP (Entry Point) map**, which provides timestamp-to-sector mapping. This is essential for seeking and for extracting the specific sector ranges that correspond to a playlist's in/out time window.
## File Structure
CLPI files use big-endian byte order. The header:
```
Offset Size Field
------ ---- -----
0 4 Magic: "HDMV"
4 4 Version: "0200" (BD) or "0300" (UHD BD)
8 4 SequenceInfo start offset
12 4 ProgramInfo start offset
16 4 CPI (Characteristic Point Information) start offset
20 4 ClipMark start offset
24 4 ExtensionData start offset
```
### ClipInfo Section (offset 40)
Contains basic clip metadata. The source packet count (total number of 192-byte source packets in the M2TS file) is at offset 56:
```
Offset Size Field
------ ---- -----
40 4 ClipInfo length
44 2 Reserved
46 1 Stream type
47 1 Application type
48 4 Reserved
52 4 TS recording rate
56 4 Source packet count
```
The source packet count multiplied by 192 gives the total byte size of the M2TS file.
## EP Map
The EP map lives inside the CPI section and provides random access into the transport stream. It maps PTS timestamps to Source Packet Numbers (SPN), enabling precise seeking without scanning the stream.
### CPI Section Layout
```
Offset Size Field
------ ---- -----
0 4 CPI length
4 2 Reserved / CPI type
6 ... EP map
```
### EP Map Header
```
Offset Size Field
------ ---- -----
0 1 Reserved
1 1 Number of stream PID entries
2 ... Stream PID entry headers (one per stream)
```
Each stream PID entry header (14 bytes):
```
Offset Size Field
------ ---- -----
0 2 Stream PID
2 2 Reserved + EP stream type
4 2 Number of coarse entries
6 4 Number of fine entries (note: 32-bit, can be large)
10 4 EP map start offset (relative to EP map start)
```
libfreemkv parses only the first stream (primary video), which is sufficient for sector-level seeking.
### Two-Level Index
The EP map uses a two-level structure to compress what would otherwise be a very large lookup table:
- **Coarse entries**: low-resolution index covering large time/sector ranges
- **Fine entries**: high-resolution entries within each coarse range
Each coarse entry points to a range of fine entries via `ref_to_fine_id`.
### Coarse Entries (8 bytes each)
Located immediately after the fine table start offset (4 bytes) in the per-stream EP map:
```
Bits Field
------ -----
[31:14] ref_to_fine_id (18 bits) -- index of first fine entry in this range
[13:0] pts_coarse (14 bits) -- upper bits of PTS
```
Second dword:
```
Bits Field
------ -----
[31:0] spn_coarse (32 bits) -- upper bits of SPN
```
### Fine Entries (4 bytes each)
Located at the fine table start offset within the per-stream EP map:
```
Bits Field
------ -----
[31] is_angle_change_point (1 bit)
[30:28] I_end_position_offset (3 bits)
[27:17] pts_fine (11 bits) -- lower bits of PTS
[16:0] spn_fine (17 bits) -- lower bits of SPN
```
## Reconstructing Full PTS and SPN
The full timestamp and packet number are assembled by combining the coarse and fine components:
### Full PTS
```
full_pts = (pts_coarse << 19) + (pts_fine << 8)
```
The coarse component provides the upper bits, shifted left by 19. The fine component provides mid-range bits, shifted left by 8. The resulting PTS is in 45kHz ticks, matching the MPLS timestamp format.
### Full SPN
```
full_spn = (spn_coarse & 0xFFFE0000) + spn_fine
```
The coarse SPN provides the upper 15 bits (masked to clear the lower 17). The fine SPN provides the lower 17 bits. The result is a Source Packet Number -- each source packet is 192 bytes in the M2TS file.
### Resolved EP Map
The `resolved_ep_map()` method iterates through all coarse entries and their associated fine entries to produce a flat list of `(PTS, SPN)` pairs. For each coarse entry at index `ci`:
- Fine entries range from `coarse[ci].ref_to_fine_id` to `coarse[ci+1].ref_to_fine_id` (or end of fine table for the last coarse entry).
- Each fine entry is combined with its parent coarse entry to produce one full `(PTS, SPN)` pair.
## Deriving Sector Extents for Ripping
Given a playlist's in_time and out_time (from MPLS), the CLPI EP map provides the sector ranges to read from disc. The `get_extents()` method does this in three steps:
### Step 1: PTS to SPN
Binary search the resolved EP map for the in_time and out_time:
- **start_spn**: the SPN at or before in_time (seek backward to the nearest I-frame)
- **end_spn**: the SPN at or after out_time (include the full GOP)
### Step 2: SPN to Byte Offset
Each Source Packet is 192 bytes (188 bytes MPEG-TS payload + 4 bytes M2TS header):
```
byte_offset = spn * 192
```
### Step 3: Byte Offset to Sector
M2TS files are stored contiguously on disc. Sectors are 2048 bytes:
```
start_sector = start_byte / 2048
end_sector = (end_byte + 2047) / 2048
sector_count = end_sector - start_sector
```
The resulting `Extent` contains `start_lba` (relative to the M2TS file's starting sector on disc) and `sector_count`. The caller adds the file's absolute starting LBA from UDF to get disc-absolute sector numbers.
### Alignment Note
The 192-byte source packet size and 2048-byte sector size share no common factor beyond 1. A single sector contains roughly 10.67 source packets. The conversion rounds start down and end up to ensure complete coverage.
## Putting It Together
The full ripping pipeline chains three parsers:
1.**MPLS** provides clip IDs and in/out timestamps.
2.**CLPI** converts those timestamps to SPN ranges, then to sector extents.
3.**UDF** provides the file's starting LBA on disc for absolute sector addressing.
The `Disc::scan()` method in `src/disc/mod.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
## References
- BD-ROM Part 3, Section 5.5: Clip Information file format
MPLS (Movie PlayList) files define playback titles on a Blu-ray disc. Each `.mpls` file in `BDMV/PLAYLIST/` describes one title -- a sequence of clips with precise in/out timestamps and stream information. The main movie, bonus features, trailers, and menus each have their own MPLS file.
A disc may contain dozens of MPLS files. Most are short (menus, logos, transitions). The main movie is typically the longest playlist. libfreemkv filters out playlists shorter than 30 seconds.
## File Structure
MPLS files use big-endian byte order throughout. The file has three main sections:
```
Offset Size Field
------ ---- -----
0 4 Magic: "MPLS"
4 4 Version: "0200" (BD) or "0300" (UHD BD)
8 4 PlayList start offset (absolute from file start)
12 4 PlayListMark start offset
16 4 ExtensionData start offset
```
### PlayList Section
Located at the PlayList start offset. Contains all play items and sub-path entries:
```
Offset Size Field
------ ---- -----
0 4 PlayList length
4 2 Reserved
6 2 Number of PlayItems
8 2 Number of SubPaths
10 ... PlayItem entries (variable length)
```
### Play Items
Each play item references one clip and specifies what portion to play:
```
Offset Size Field
------ ---- -----
0 2 PlayItem length (bytes after this field)
2 5 Clip ID (ASCII, e.g. "00001")
7 4 Codec ID ("M2TS")
11 1 Connection condition (lower 4 bits)
12 1 Ref to STC_id
14 4 IN_time (45kHz PTS ticks)
18 4 OUT_time (45kHz PTS ticks)
22 8 UO_mask_table
30 1 Misc flags
31 1 still_mode
32 ... STN_table (first play item only is parsed)
```
**Connection condition** values:
- `1` = seamless connection (no gap between clips)
- `5`, `6` = non-seamless connection
**Timestamps** use 45kHz PTS (Presentation Time Stamp) ticks, the same timebase as MPEG transport streams. Duration of a play item = `OUT_time - IN_time`. To convert to seconds: divide by 45000.
A playlist's total duration is the sum of all play item durations.
### Clip ID Mapping
The Clip ID (e.g. "00001") maps to:
- `BDMV/STREAM/00001.m2ts` -- the transport stream
- `BDMV/CLIPINF/00001.clpi` -- the clip info (EP map, stream details)
## STN Table
The Stream Number Table describes all elementary streams available in the clip. It is parsed from the first play item (which defines the title's stream layout).
```
Offset Size Field
------ ---- -----
0 2 STN table length
2 2 Reserved
4 1 Number of primary video streams
5 1 Number of primary audio streams
6 1 Number of PG (subtitle) streams
7 1 Number of IG (interactive graphics) streams
8 ... Stream entries
```
### Stream Entries
Each stream entry has two parts: a stream reference and a stream attributes block.
**Stream reference:**
```
Offset Size Field
------ ---- -----
0 1 Entry length
1 1 Stream type (1=PlayItem, 2=SubPath, 3=InMux)
2 2 PID (MPEG-TS packet ID, big-endian)
```
**Stream attributes** (immediately follows the reference):
```
Offset Size Field
------ ---- -----
0 1 Attributes length
1 1 Coding type
2+ ... Type-specific fields
```
The layout of type-specific fields depends on the stream category:
`libfreemkv` supports a multi-stage rip model for damaged or protection-bearing
discs: a fast forward sweep that tolerates read failures, in-loop request-size
adaptation that survives transient drive trouble without bailing, and targeted
retry passes against a persistent bad-range map. The stream pipeline
(`DiscStream` + `input`/`output`) operates against the resulting ISO image, so
the mux stage never touches the drive.
Recovery is layered cleanly. Each layer has one responsibility and does not
reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
UDF (Universal Disc Format) is the filesystem standard for optical media. It is defined by ECMA-167 with extensions from the OSTA (Optical Storage Technology Association). BD-ROM discs use UDF revision 2.50, which introduces the **metadata partition** -- a critical feature that separates file metadata from file content on disc.
## Why UDF 2.50 for Blu-ray?
Older UDF revisions (1.02, 1.50) scatter ICBs (file metadata) and file data across the same partition. On a high-capacity Blu-ray disc (25-100 GB), this creates excessive seeking when the drive needs to read a directory listing or locate a file. UDF 2.50 solves this by placing all metadata into a contiguous region near the beginning of the disc. The drive reads metadata from one compact area and streams file data from another -- no interleaved seeks.
BD-ROM Part 3 of the Blu-ray specification mandates UDF 2.50.
## Metadata Partitions
A UDF 2.50 BD-ROM has two logical partitions:
- **Partition 0 (Type 1)** -- the physical partition. Contains actual file data (m2ts streams, playlist files, etc.). Mapped directly to disc sectors starting at the Partition Descriptor's `partitionStartingLocation`.
- **Partition 1 (Type 2)** -- the metadata partition. Contains all ICBs (Inode-like structures), directory data, and the File Set Descriptor. The metadata partition is itself stored as a file within the physical partition. Its location is found by reading an Extended File Entry at LBA 0 of the physical partition.
The key rule: **ICBs and directory data live in the metadata partition. File content lives in the physical partition.** When an ICB's allocation descriptor gives an LBA, the partition it refers to depends on what the LBA describes -- metadata-relative for directory entries, physical-partition-relative for file data extents.
## Pointer Chain
Reading a UDF 2.50 filesystem follows a fixed chain of pointers. Each step reads one or two 2048-byte sectors:
```
Sector 256: AVDP (Anchor Volume Descriptor Pointer, tag 2)
|-- Each FID names a file/subdirectory and points to its ICB
|-- Recurse into subdirectories to build the full file tree
```
### AVDP (Sector 256)
The Anchor Volume Descriptor Pointer is always at sector 256 (ECMA-167 section 10.2). It points to the Main Volume Descriptor Sequence. Tag identifier = 2.
### VDS (Sectors 32+)
The Volume Descriptor Sequence contains:
- **Partition Descriptor (tag 5)**: byte offset 188 holds `partitionStartingLocation` -- the absolute sector where the physical partition begins.
- **Logical Volume Descriptor (tag 6)**: byte offset 268 holds the number of partition maps. The partition maps themselves start at offset 440. For BD-ROM, map 0 is Type 1 (physical) and map 1 is Type 2 (metadata).
- **Terminating Descriptor (tag 8)**: signals the end of the VDS.
### Metadata File
When two partition maps exist and the second is Type 2, the metadata partition content is located by reading the Extended File Entry at the first sector of the physical partition (partition_start + 0). This ICB's allocation descriptor gives the offset and length of the metadata content within the physical partition.
### File Set Descriptor
The FSD (tag 256) sits at metadata-relative LBA 0 (the first sector of the metadata content). It contains a long allocation descriptor at offset 400 pointing to the root directory ICB. The LBA in this long_ad is at bytes 404-407.
### Directory Traversal
Each directory is an ICB (Extended File Entry, tag 266, or File Entry, tag 261) whose allocation extent points to directory data. The directory data is a sequence of File Identifier Descriptors (FIDs, tag 257):
| FID Field | Offset | Size | Description |
|-----------|--------|------|-------------|
| Tag | 0 | 2 | Always 257 |
| File characteristics | 18 | 1 | Bit 1 = directory, bit 3 = parent |
Allocation descriptors use the Short Allocation Descriptor format: 4 bytes extent length (upper 2 bits = type), 4 bytes extent position (LBA).
## How read_filesystem() Works
The `read_filesystem()` function in `src/udf.rs` follows the pointer chain above:
1. Reads sector 256, validates AVDP (tag 2).
2. Scans sectors 32-63 for the Partition Descriptor and Logical Volume Descriptor.
3. If two partition maps exist and the second is Type 2, reads the metadata file ICB at partition_start to find metadata_start.
4. Reads the FSD at metadata_start, extracts the root directory ICB LBA.
5. Calls `read_directory()` recursively (max depth 3) to build the full file tree.
Each directory read involves two sector reads: one for the ICB, then one or more for the directory data. File sizes are read from info_length in each file's ICB.
`read_file()` reads a file by navigating the directory tree, reading the file's ICB to get its data extent, then reading the data sector by sector from the **physical partition** (partition_start + LBA, not metadata_start).
## Buffered Sector Reads
USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_filesystem()` and `read_file()` issue one SCSI READ per sector, a full disc scan can require hundreds of commands -- taking 10+ minutes on USB.
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
### UDF Filename Encoding
UDF filenames use a compression ID as the first byte:
- `8` = 8-bit characters (ASCII)
- `16` = 16-bit big-endian Unicode (UTF-16BE)
The parser handles both encodings. All path lookups are case-insensitive.
## BD-ROM Directory Structure
A typical Blu-ray disc has this directory layout:
```
/
+-- BDMV/
| +-- index.bdmv Disc index (title list, first play)
| +-- MovieObject.bdmv Movie objects (navigation commands)
| +-- PLAYLIST/
| | +-- 00000.mpls Main movie playlist
| | +-- 00001.mpls Director's commentary
| | +-- ...
| +-- CLIPINF/
| | +-- 00001.clpi Clip info for 00001.m2ts
| | +-- 00002.clpi
| | +-- ...
| +-- STREAM/
| | +-- 00001.m2ts Transport stream (video/audio/subtitle data)
| | +-- 00002.m2ts
| | +-- ...
| +-- BACKUP/ Duplicate of index, MovieObject, playlists, clip info
|
+-- AACS/ AACS encryption data (encrypted discs only)
| +-- Unit_Key_RO.inf Unit key file (encrypted)
| +-- MKB_RW.inf Media Key Block
| +-- Content000.cer Content certificate
| +-- DUPLICATE/ Backup copies
|
+-- CERTIFICATE/ BD+ certificate data (some discs)
```
The parser reads from `BDMV/PLAYLIST/` and `BDMV/CLIPINF/` to discover titles and their sector layouts. The `BDMV/STREAM/` directory contains the actual transport streams but is not parsed by the UDF layer -- stream data is read by LBA directly using extents computed from CLPI EP maps.
## References
- ECMA-167: Volume and File Structure of Write-Once and Rewritable Media
- UDF 2.50 (OSTA): Universal Disk Format Specification
- BD-ROM Part 3: Blu-ray Disc Read-Only Format, File System Specifications
/// Mutation: in the `key` array drop the `title_key[i] ^` term -> both
/// keys give the same body, assert fires.
#[test]
fndescramble_output_depends_on_title_key(){
letseed=[0xDE,0xAD,0xBE,0xEF,0x42];
letmake=|k: &[u8;5]|{
letmuts=vec![0x00u8;2048];
s[0x14]=0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(k,&muts);
s
};
leta=make(&[0x01,0x02,0x03,0x04,0x05]);
letb=make(&[0x01,0x02,0x03,0x04,0x06]);// differs in last byte
assert_ne!(
&a[128..2048],
&b[128..2048],
"different title keys must descramble differently"
);
}
/// Descramble is keyed by the sector seed too: same title key, different
/// seed -> different body. Pins that bytes 0x54..0x59 actually feed the
/// keystream (not just the per-sector XOR key).
///
/// Mutation: replace `seed` array reads with a constant -> both seeds give
/// the same body, assert fires.
#[test]
fndescramble_output_depends_on_seed(){
letkey=[0x01,0x02,0x03,0x04,0x05];
letmake=|seed: [u8;5]|{
letmuts=vec![0x00u8;2048];
s[0x14]=0x30;
s[0x54..0x59].copy_from_slice(&seed);
descramble_sector(&key,&muts);
s
};
leta=make([0x11,0x22,0x33,0x44,0x55]);
letb=make([0x11,0x22,0x33,0x44,0x56]);
assert_ne!(
&a[128..2048],
&b[128..2048],
"different seeds must descramble differently"
);
}
}
#[path = "lfsr_tests.rs"]
modtests;
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Matthew Jackson