FMTS (AACS 2.1) now decodes per (LBA, phase): Phase enum + AacsKeyMap::
from_ranges_phased, decrypt only the variant's parity half. resolve_fmts_key_map
does a 2-phase index-1 anchor then per-index phase probe, and sizes the forensic
set to whatever the source returns (no hardcoded 32). KeyFetch is now two explicit
operations (unit_keys / fmts_indexes) and KeySource splits get_uk into
get_unit_keys + get_fmts_indexes. BYPASS_FMTS_KEY gate removed (first-class format).
Teed up for 1.4.5. Local WIP baseline.
The keyserver protocol now returns all 32 index keys as an array for a
forensic content sample (and a single-element array for plain content).
resolve_fmts_key_map sends one forensic batch and maps array element i to
segment index i+1, replacing the per-index blind-probe collection loop
that repeatedly hit the key service. Segment/index parsing and the
aligned-unit content classification are reworked to support this:
- rename variant_select -> index_select (per-index, not per-variant)
- content classification moves to is_clean(buf, ContentFormat) so the
unit selector emits only units the key service accepts
- segment.rs: parse IndividualSegment.tbl index tags + SPN ranges,
build contiguous LBA key ranges from the resolved 32-key array
- decrypt/decorator plumbing for the resolved per-index keys
Fail loud (FmtsKeyMissing) when the forensic query returns < 32 keys or
any segment index stays unresolved.
decrypt:
- decrypt_sectors is now a pure decrypt (apply key, leave plaintext, report
unverified bytes); TS-structure is a separate primitive (is_clean_ts/ps) used
only for key selection and read-verify. The mux passes decrypted bytes through
(the demuxer drops non-conforming packets), ending the NULL-TS conceal loop and
the per-unit key-server refetch storm. Key-proof floor replaces the 75%
supermajority.
recovery:
- Removed the post-read decrypt-verify gate (verify.rs) that mis-aligned the
disc-absolute unit grid against clip-anchored AACS units and false-failed good
clips (e.g. Dunkirk's orphan-CPS clip). Bad sectors are marked by physical read
result; decryptability is proven at scan + mux time.
HD DVD (first-class AACS):
- Role-based candidate-list file sourcing so an HD DVD's /ANY!/ files
(MKBROM.AACS, VTKF000.AACS, CONTENT_CERT.AACS) are found with no disc-type
branch. parse_vtkf parses VTKF000.AACS into the same UnitKeyFile as a BD
Unit_Key_RO.inf, so the shared VUK unwrap applies unchanged. set_unit_base
clip-anchoring. Two decrypt-axis assumptions remain UNVERIFIED-HDDVD-DECRYPT
(no encrypted disc to test).
mux:
- MVC (Blu-ray 3D) track signals unified into one MVCDecoderConfigurationRecord;
release-safe track_vint (3-byte VINT) and pid_index (i32) guards.
hardening:
- Container-aware is_clean / encryption detection; bytes_bad_in_title fail-safe
on a corrupt mapfile; CSS crack gated on DiscFormat::Dvd (HD DVD excluded);
non-vacuous CSS tests; patch NOT_READY/HARDWARE/ILLEGAL_REQUEST/ABORTED
sense-path tests.
decrypt_sectors is now a pure decrypt — apply the CPS unit key, leave the
plaintext, report how many bytes did not reach clean TS ("unverified"). It
never restores ciphertext, nulls, or re-fetches. "Did a key produce clean TS?"
is a key-selection / read-verify signal, not the verdict "did we decrypt?": a
correct key can decrypt a bad-encoded region, and broken TS is a muxer concern
(the demuxer drops the packet and resyncs).
Callers own the policy:
- mux (read > decrypt > mux): pass the decrypted bytes to the muxer, whatever
they are; fail loud only on a genuine can't-decrypt (no key / misaligned).
- sweep/patch (reading from a disc): an unverified unit is a bad read — recover
a fresh key and retry, or fail loud so disc-recovery re-reads it.
Removes three duplicated decisions — the decrypt-time ciphertext restore, the
mux NULL-TS conceal loop, and the per-unit key-server refetch — plus the dead
aacs_unit_still_ciphertext predicate. Key-fetch recovery now samples the on-disc
ciphertext explicitly (a pure decrypt leaves the buffer plaintext) and lives
only on the rip/verify path, never the mux.
Fixes the 30-90s/region mux stalls and key-server storm on bad-encoded UHD runs
that 1.4.1 left behind (it relaxed the gate but not the surrounding machinery).
AACS content decryption rejected a whole 6144-byte aligned unit unless
EVERY content packet was conformant MPEG-TS. One authored-bad packet (a
pressing/encoding defect or an AACS 2.1 forensic-variant frame) made the
mux conceal the entire unit as NULL TS — destroying up to 31/32 good
packets and tallying them as loss, surfacing as false "corruption" on
otherwise-clean discs (observed across two UHD titles).
decrypt_unit now asks only "did a key OPEN this unit?" — a padding-aware
>=75% supermajority of content packets restoring their 0x47 sync, a gate
no wrong key can reach (uniform-AES noise floor) yet one that tolerates a
minority of authored-bad packets. Opened units pass through VERBATIM; a
non-conforming packet is left for the demuxer to drop on sync-loss and
resync past. TS-sync conformance is a muxer concern, never a decryption
verdict. The post-read verify/sweep gate now shares the same primitive so
it can never disagree with the mux decrypt.
Also unify the MVC (Blu-ray 3D) track signals: the mvcC CodecPrivate
extension, the BlockAdditionMapping, and each per-frame BlockAdditional
all derive from one MVCDecoderConfigurationRecord built once per track, so
a malformed dependent-view parameter set can no longer orphan a BlockAddID.
Second audit round converged (severity collapsed 6 HIGH -> 1; the one
HIGH was a bounded 32-element scan, not a defect; the sole spec MEDIUM
was the same false-positive re-raised — 0xBF matches ISO/IEC 14496-15
§7.6.2 verbatim). One genuine robustness fix plus coverage:
- extract_mvc_params: skip a zero-length NAL instead of abandoning the
scan, so a stray length prefix before the subset SPS/PPS no longer
silently drops 3D signalling. Test proves params after a zero-length
NAL are still found.
- Tests: parser_for_mvc_dependent routes H.264 to a passthrough parser;
passthrough with an IDR does not re-assert param sets (the keyframe &&
!mvc branch).
- Document the per-playlist (not per-clip) is_3d latching as a known
limitation (real main-feature playlists are uniformly 3D).
Triage of a 10-lens code audit of the 3D branch. Fixes for real defects;
rejected three spec false-positives that matched the ISO/IEC 14496-15
§7.6.2 record verbatim.
Robustness / correctness:
- Never panic when a title's only video is the MVC dependent view: the
base is now the first NON-dependent video, so a dependent-only title
sets up no merge (muxed as an ordinary track) instead of hitting an
`expect` on the skipped track slot.
- Drop a per-frame BlockAdditional (BlockAddID=2) when the track declared
no mvcC mapping (dependent params not captured before the header) — a
plain block keeps the file conforming instead of an orphaned add.
- A non-keyframe MVC base frame always carries a ReferenceBlock (fall back
to a 0 offset in the pre-first-keyframe corner) so it is never mistaken
for a seek point.
- Reference the last keyframe on the PRIMARY video track only, so a
secondary video track's keyframe can't become a cross-track reference.
- dep_by_pts overflow: bound BEFORE inserting so the just-arrived
dependent survives the drift-clear; count a displaced duplicate-PTS
dependent as an orphan instead of losing it silently.
API / docs:
- Fold write_frame_with_additional into write_frame(..., Option<&[u8]>)
per the "no foo_with_X" convention.
- Fix mvc_params doc (StereoMode is intentionally not emitted); remove a
stale PAT/PMT comment describing an approach that was never taken.
Tests: MVCDecoderConfigurationRecord over-length guards; write_int minimal
two's-complement widths; BlockGroup/BlockAdditions/BlockAdditional +
ReferenceBlock emission; additional dropped without a mapping; h264 MVC
passthrough keeps param sets in-band; extract_mvc_params no-panic on
truncated/empty input; pairing window + dep-overflow edges; no-panic on a
dependent-only title.
Fold the MVC dependent (right-eye) view into the base H.264 track as a
per-frame BlockAdditional under an mvcC BlockAdditionMapping, so a 3D
title produces one MVC video track instead of two independent H.264
tracks.
- h264: MVC-passthrough parser mode keeps the dependent view's subset
SPS/PPS in-band, so each emitted frame is a self-contained dependent
access unit for a BlockAdditional
- resolve: route the dependent stream through the passthrough parser
- mkvstream: detect the dependent view, pair it to the base frame by
PTS (bounded FIFO), attach it as a BlockAdditional (BlockAddID=2),
and skip building its own track; build the mvcC
MVCDecoderConfigurationRecord from the captured subset SPS/PPS and set
it on the base track at activation
- mkv: emit the mvcC BlockAdditionMapping and BlockGroup/BlockAdditions,
with a ReferenceBlock on non-keyframe base frames
- ebml: add BlockAdditions/BlockMore/BlockAdditional/BlockAddID/
BlockAddIDValue/ReferenceBlock elements and a signed-int writer
Verified against a Blu-ray 3D ISO: ffprobe shows a single MVC track,
the mvcC mapping is present, ~144k BlockAdditionals carry the dependent
view (8.7 GB), and the base view decodes cleanly with no regression.
MVCDecoderConfigurationRecord follows ISO/IEC 14496-15 7.6.2; StereoMode
is intentionally omitted (no enum value describes MVC-in-BlockAdditional;
the mvcC mapping is the primary 3D signal per RFC 9559).
Round-11 findings from the 10-phase release audit (no real HIGH):
- When a no-declared-duration source (HD-DVD) muxes a degenerate single
frame at tick 0 with no per-frame duration, max_block_ticks stays 0 and
the reserved DURATION placeholder was left as a literal 0.0 (players read
that as a zero-length file). Void the element instead, so the Segment
omits DURATION as an unknown-duration source did before the back-patch.
- Add a regression test for the sniff_video_codec overlap fix (a
picture_start_code whose payload begins 00 00 followed by a real start
code) so the i+=4 marker skip can't silently regress to i+=3.
Round-10 findings from the 10-phase release audit:
- A finder claimed the DTS SFREQ→rate table was wrong at 11/12; verified
it against ffmpeg's avpriv_dca_sample_rates (12k/24k/48k/96k/192k at
11-15) — the table is CORRECT. Added a test that locks the full table so
it can't be mis-"fixed".
- sniff_video_codec advanced 3 bytes after a matched start code, re-reading
the code byte as an overlapping start code; skip the full 4-byte marker.
- Guard the HD-DVD next_id title counter with saturating_add so a crafted
disc with >65536 clips can't overflow (panic in debug).
- Add a test that an undecryptable unit (DecryptFailed) is zero-filled and
counted as loss through ExtractResult (complete=false, bytes_lost>0) —
the recovery-seam consolidation folded that bucket into bytes_unreadable.
Round-9 findings from the 10-phase release audit (no HIGH):
- Remove the MAX_PTS_MARKS backstop and its tautological test: an empty
DTS PES returns before recording a mark, and a non-empty run is already
bounded by the MAX_AU_BYTES buffer clear (which clears pts_marks) — so
the deque cannot grow unbounded and the cap was dead code.
- AuAssembler::for_codec no longer reserves 256 KiB for a Passthrough
stream (audio/subtitle, and every TS/BD stream) whose buf is never
written; only the reassembling modes reserve.
- Correct the scan comment that claimed region is computed (it is a
Region-free stub until region detection lands) and drop a public-repo
reference to internal "private refactor notes" in the mkb module doc.
Round-8 findings from the 10-phase release audit:
- detect_disc_format's BDMV fallback passed detect_format's result
through unchanged, so an SD bonus/menu title could tag a BD-tree disc as
DVD (mis-sizing the ECC sweep) — violating its own "never below Blu-ray"
invariant. Clamp anything but UHD up to Blu-ray.
- Track the MPEG-2 GOP byte total incrementally instead of re-summing the
whole gop_buf on every pushed picture (was O(pictures²) on any MPEG-2
disc, not just adversarial input).
- Back the DTS pts_marks deque with a VecDeque so the over-cap prune is an
O(1) pop_front, not an O(n) Vec::remove(0).
- Add a test exercising parse_stream_id_extension's PTS/DTS skip branches
(the real AU-opening 0xFD video PES path) — previously untested.
Round-7 findings from the 10-phase release audit (no HIGH; convergence):
- Cap DtsParser.pts_marks (MAX_PTS_MARKS): a run of zero-length timed PES
packets grew no buffer bytes, so the drain_front mark-prune never ran —
the deque could accumulate without bound on hostile PS input.
- detect_disc_format tested HVDVD_TS before BDMV while the title-scan
dispatch tests BDMV first, so a disc with both trees would be classified
HD-DVD but enumerated as Blu-ray. Align both to BDMV → HVDVD_TS →
VIDEO_TS.
- Document why the DTS new-PES re-base can emit a locally-decreasing PTS
(the muxer's block_ts applies the strictly-monotonic audio nudge, tested
in mkv.rs) — this is by design, not a mux defect.
- Fix stale aacs/keys.rs comment references (functions moved to
aacs/inf.rs / aacs::resolve/derive in the module split).
Round-6 findings from the 10-phase release audit:
- Wire the documented MAX_PENDING_BYTES byte cap into the MPEG-2 GOP
buffer (it was dead code) and add an equivalent MAX_GOP_BYTES cap to the
sparse-PTS reorder, so a crafted stream of few-but-huge access units
cannot over-allocate — both were bounded only by frame count before.
- probe_evo_streams defaulted an unsniffable HD-DVD video stream to H.264,
which mis-parses a VC-1 (or still-encrypted) clip into a corrupt track.
Emit the video stream only when the codec is actually identified — the
honest outcome, matching the audio path (a real clear clip always carries
its sequence header at the head).
- Resume the AU-opener search from a cursor (like the boundary search), so
a long unsynced junk run is O(bytes), not O(buffer) per push.
- Mark mpeg2's now-dead MAX_AU_BUFFER test-only; restore #[doc(hidden)] on
the aacs probe harness module.
- Add regression tests: the 0xFD video-routing guard, the FMTS-is-UHD key
state, and the GOP byte caps.
Round-5 findings from the 10-phase release audit:
- collect_es routed EVERY extended-stream-id (0xFD) PES into the video ES
buffer, so a 0xFD HD-audio sub-stream (MLP/TrueHD) could pollute the
video sample and — if it preceded the video PES — stamp the video track
with the audio PID, losing the video. Only the VC-1 extension (0x55) is
now treated as video; routing 0xFD audio to its own track is deferred to
the HD-DVD program-chain follow-up.
- The sparse-PTS reorder now carries its calibrated per-frame duration onto
each frame, so the muxer emits a BlockDuration and the back-patched
Segment Duration covers the final frame instead of understating it.
- Add regression tests for the MAX_MARKS and MAX_VTI_HITS caps (promote
MAX_VTI_HITS to module scope); make the differential-test factory array a
named type; drop an identity-op in a reorder test.
Round-4 findings from the 10-phase release audit (the first fully clean
round; it dug into the new #22/#18 refactor code):
- AuAssembler closed each AU from only the FRONT mark's fields, so when
one PES fragment carried the source and a later fragment of the same AU
carried the PTS, the second field was dropped — a regression vs the old
separate pts/source mark deques. Now merge the first Some of each field
across all in-range marks.
- parse_vti_clip_order picked the largest residue bucket with
HashMap::into_values().max_by_key(), nondeterministic on a size tie
(randomized HashMap iteration) — could select a different clip table
run-to-run. Break ties by smallest offset.
- Bound the marks/disc_marks deques (MAX_MARKS): the buf-size cap prunes
marks only when bytes accumulate, so a run of zero-length timed
fragments could grow them without bound on hostile input.
- Add push_owned so the PS path moves the PES payload into a passthrough
AU with no copy (MPEG-2 video + all audio), removing a per-PES
malloc+memcpy the refactor had introduced on the DVD path.
- Back-patch the MKV duration from the block END (start + its own
duration) so it covers the final frame instead of understating by one.
- Add direct tests for the MKB record-framing walker; drop a stale
drain_complete_aus doc comment left on process_au.
AuAssembler::drain rescanned the whole buffered access unit from a fixed
offset on every push, so reassembling one AU split across N program-stream
PES fragments cost O(bytes²/fragment) — amplified on the HD-DVD PS path
where H.264/HEVC/VC-1 frames are large and now flow through this shared
assembler (unlike the TS path, which delivers one AU per PES).
Carry a scan_pos cursor (and, for the stateful VC-1/MPEG-2 rules, a
seen_unit flag) so each push resumes the boundary search where the last
one stopped instead of restarting. Total scan work for one AU is now
O(AU bytes). The from-scratch scanners are retained as a #[cfg(test)]
oracle; a new differential test asserts the resumable path yields
byte-identical AUs at every fragment granularity for all three modes.
Round-1 findings from the 10-phase release audit:
- SparsePtsReorder buffered its current GOP with no bound, draining only on
a keyframe — an open-GOP or crafted program stream that never signals one
could hold the whole title in RAM. Force-complete the GOP at
MAX_GOP_FRAMES, matching the MPEG-2 parser's backstop.
- inject_unit_keys labelled a 2.1 FMTS disc as AACS 1.0 / bus-encryption
off; FMTS is UHD-family, so synthesize the UHD version + bus encryption.
- The compiled Key Correction Data was a non-zero 16-byte constant fed into
the Media Key derivation. Per the no-compiled-keys rule it is now all-zero;
the chain still cannot complete on a real disc (documented), so this is
behaviour-neutral — all variant tests pass unchanged.
- Fix stale doc references (broken `super::variants` intra-doc links, and
`aacs::keys` comments) left by the module rename.
A title whose scan yields no duration (HD-DVD — its `.MAP` timemaps are
not parsed, so DiscTitle.duration_secs is 0) produced an MKV with no
Segment Duration element, so players/MediaInfo reported an unknown
runtime.
Reserve a DURATION placeholder when the source declares none, track the
highest block timestamp written, and back-patch the placeholder at
finish() with the real muxed runtime (also enabling the per-track BPS
tags for these titles). Gated on duration_secs == 0, so BD/UHD/DVD —
which carry a real mpls/IFO duration — write it up-front exactly as
before, unchanged.
The MPEG-2 parser hand-rolled its own PES reassembly — a byte buffer plus
parallel PTS / source / discontinuity mark queues keyed by absolute
offset — duplicating what AuAssembler already does for H.264/HEVC/VC-1.
Add a Mode::Mpeg2 to AuAssembler (picture 0x00 with preceding sequence
0xB3 / GOP 0xB8 headers — the same headers-precede-picture shape as the
VC-1 mode) and have the MPEG-2 parser own one via AuAssembler::mpeg2().
parse() now feeds fragments to the assembler and processes each complete
access unit; the buffer, base offset, and three mark queues are gone. The
GOP-buffered temporal_reference reorder and PTS origin-locking are
unchanged. The parser's external contract is unchanged, so all existing
MPEG-2 parser tests pass as-is; new AuAssembler tests cover the MPEG-2
boundary rule directly.
HD-DVD EVO (and DVD VOB) program streams timestamp video at GOP
granularity: only one access unit per GOP carries a PES PTS. The H.264 /
HEVC / VC-1 parsers collapsed a missing PTS to 0, so on such a source
every non-anchor frame landed on the same block timestamp and a decoder
reported "non monotonically increasing dts".
Add a shared SparsePtsReorder that rebuilds a display-order PTS per frame
from the coded picture type (I/P/B) plus the sparse anchor PTS, with a
per-frame duration self-calibrated from the spacing between consecutive
GOP anchors (no external frame-rate needed). Display order is derived via
the classic single-anchor-delay rule (an anchor displays only after the
previously-held anchor; a B displays immediately), exact for the
non-hierarchical GOP structures HD-DVD H.264/VC-1 use. It mirrors the
MPEG-2 parser's GOP-buffered origin-locking.
Gated to the program-stream path only: the three parsers enable it via
with_ps_reorder(is_dvd_ps), so the BD/UHD transport path (per-frame PTS)
is byte-identical and untouched.
VC-1 HD-DVDs (e.g. Shaun of the Dead) carry video on MPEG-PS extended
stream id 0xFD, with the real stream selector in stream_id_extension
inside the PES extension. Parse that field so the video routes to a
distinct track (pid 0xFD00|ext) instead of being dropped.
Reframe VC-1 access units in AuAssembler with a dedicated Mode::Vc1:
an AU is delimited by the next frame BDU (0x0D) once a frame has already
been seen, so the sequence (0x0F) and entry-point (0x0E) headers that
precede an I-frame stay attached to the frame they describe. The old
single-start-code split stranded those headers on the prior AU, which
the decoder reported as bits-overconsumption and hard decode failures.
hddvd probe now tracks the video pid it detects and emits VC-1 on 0xFD.
- aacs/resolve: a media-keys-only provider missing the VID classifies as
VidUnavailable, not NoMaterial (an MK derives the VUK once the VID
arrives).
- disc/bluray: mark a clip seen only after its .clpi parses, so a
transient parse failure on the first PlayItem cannot suppress the
clip's extents for a later PlayItem referencing it that succeeds.
- disc/patch: log rather than swallow mapfile record/flush failures on a
reverify downgrade, so a failed persist cannot silently mismark a bad
unit good on resume.
- mux/ts: flag a discontinuity when a partial PES is dropped, matching
the other partial-drop paths.
- mux/demux_thread: the no-demuxer branch forwards an empty batch for
early consumer-disconnect detection instead of reading the whole disc.
- io/pipeline: correct the send-timing log (as_secs_f64, not as_micros
printed as ms).
- aacs/derive, aacs/variant, disc/read_error, keysource: comment/doc
accuracy. sector/prefetched, udf: remove dead fields/functions.
- mux/disc: assert unit-aligned read counts in the test.
Replace the AACS-specific inline key-fetch in the decrypt decorator with
a scheme-neutral recovery seam: the input stream (L3) installs a Recover
closure (none / AACS key-fetch) and the decorator (L2) runs it at the
single decrypt-miss point. FMTS (AACS 2.1) forensic-segment units that no
key opens are just undecryptable units, concealed and counted as ordinary
decrypt loss with no FMTS-specific branch ("a loss is a loss"), so the
separate bytes_undecryptable bucket collapses into one loss count.
- sector/recovery.rs: the seam (MissOutcome, none/key_fetch factories),
naming no encryption scheme in its type.
- FMTS: segment routing primitives + BYPASS_FMTS_KEY, and an upfront
ensure_forensic_segments_decryptable gate (Error::FmtsKeyMissing) in
the mux input path, parallel to the unit-key gate.
- CSS descramble/rekey moves from decrypt_sectors into
css::descramble_region: CSS self-recovers from the data itself, so it
stays OFF the seam (which is only for external inputs).
- disc/mod.rs also: main-title selection aligned to largest physical
size; is_regular read from the open file handle, not metadata(path),
fixing a swallowed sync_all on a fresh-rip ISO. decrypt_threads()
resolved once via OnceLock off the per-buffer hot path.
The first cut used a global running clock (max(next, own-PES PTS) + advance),
which fixed the same-PES collision but DRIFTED: once accumulated frame
durations exceeded the PES-timestamp spacing, it never re-based, so a
feature-long DVD DTS track ran minutes past its real length (2h44 for a 2h03
film) while AC-3 from the same source stayed exact.
Match the AC-3 path: re-base to each PES's own container timestamp, and advance
by one frame duration ONLY within a run of AUs sharing one PES. Fixes the DVD
multi-frame-per-PES collision without drift; the UHD DTS-HD MA per-PES
attribution (da85f56) is preserved (each AU still takes its own core PES's PTS).
Adds new_pes_rebases_to_its_own_pts_no_drift; full mux suite green (905).
DVD packs several DTS core frames into one PES; the parser stamped every
access unit with that single PES PTS and duration_ns=None, so consecutive
frames collided on one timestamp — ffmpeg rejected the output as 'non
monotonically increasing dts to muxer: X >= X' (deep-decode = corrupt,
e.g. The Punisher). The UHD DTS-HD MA path (one AU per PES, distinct PTS)
was unaffected, which is why this only surfaced on DVD.
Parse the DTS core header for samples ((NBLKS+1)*32) and sample rate
(SFREQ, 48kHz fallback) to derive each AU's duration, and stamp a running
monotonic PTS: max(next_clock, own-core-PES PTS), then advance by the
frame duration. A later PES whose PTS is ahead of the clock still wins
(preserves the UHD per-PES attribution from da85f56/c49a180); frames
sharing one PES advance frame-by-frame instead of colliding.
Tests: the 3 that encoded 'same PES -> same PTS' now assert monotonic
advance; new dvd_many_cores_one_pes_are_strictly_monotonic reproduces the
Punisher bug; duration/SFREQ-fallback unit tests added.
DiscStream (the live-drive single-pass path) enables AACS decrypt-loss
concealment (NULL-TS fill on an undecryptable unit) but, unlike the file-backed
PipelinedPesStream, had no B1 resync gate — so after a concealed gap it forwarded
inter-coded video frames referencing the now-missing data, producing
decode-broken MKV output (dangling-reference frames).
Mirror PipelinedPesStream: add per-stream ResyncGate + is_video, and route every
codec-frame emit on the TS path (in-stream parse, demuxer EOF flush, parser EOF
flush) through gate.admit(is_video, frame.discontinuity, frame.keyframe) so a
video track drops forward to the next keyframe after a concealment event. Warn
once if a gate is still armed at EOF (a concealed gap in the final GOP with no
trailing keyframe). Audio/subtitle always admit.
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):
1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
never be stranded by an empty post-gap PES. The demuxer only emits
non-empty PES today; this is defense-in-depth for any future caller.
2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
`flush_pts_ns`, dts continues from the most recent known base; truehd
already kept its running cadence on a None PTS. Matches OSS behavior
(PTS rebases off the next PES that actually carries a PTS).
Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.
The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.
Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.
Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.
Three coordinated changes:
1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
an adaptation-field discontinuity_indicator; the demuxer recognises a
0x1FFF packet carrying it as a concealed gap and forces a discontinuity
on every tracked PID (the lost unit's PID is unknowable). This survives
a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
— hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
also drops any open, potentially-truncated partial PES.
2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
STARTING after it, not the one flushed at the boundary (hole 1) —
stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
and admit the real post-gap inter frame.
3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
flag stamped the previous picture. consume_ts (and the EOF flush drain)
gate on frame.discontinuity.
Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.
- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
pending_discontinuity flag set on CC gap / discontinuity_indicator and
carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
until the next keyframe disarms and resumes. Logs the resync + drop
count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
consume_ts. Out-of-range track index emits as-is (defensive).
Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:
- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
undecryptable in-content unit is tallied, overwritten with valid NULL
TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
its LBA, and the read returns Ok — the stream keeps flowing. The rip
paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
(build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
lost video/audio PID packets surface as a CC gap the TS assembler
already drops a partial PES on (the B1 foundation). Ciphertext is never
passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).
Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.
Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).
libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.
Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).
DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.
Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
Surgical fixes (each with a regression test that fails without the change):
mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.
mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.
disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.
mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).
mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.
mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.
mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).
io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).
mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).
Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).
Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).
Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
(0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.
Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.
Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.
Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.
Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.
disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.
disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.
aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.
Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
public fields) to rc6's authoritative crate::mux::codec::PictureInfo
in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
that carries coding (every video codec now fills it), with the
keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
members derived through the accessors: field_order (tff/bff/progressive)
and progressive, emitted ONLY when the codec measured the signal
(Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
/ coding_type_only(..); added measured_cicp: None to VideoStream
literals for rc6's struct.
Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
gop_start), so key is set from the frame's intra / decode-restart flag
(frame.keyframe == coding.keyframe() for video), NOT a fabricated
clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
is updated to document this as a limitation: key is an intra picture /
parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
(closed_gop) is not currently distinguished. §7.1 rewritten for the
new field_order/progressive/nb_fields members.
Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
(#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
for side-channel reuse, sink builds records directly); pub use
fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
scheme(), path_str(), input() (write-only reject) and output()
(constructs FviSink), mirroring the mkv:///demux:// patterns.
Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
frames reached the mux/index with no provenance (FVI src null). Carry
ps.source through, matching the TS path; the real-pipeline fvi test now
sees the stamped src sectors.
Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).
- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
→ coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
absence; interlaced honest-decline.
Honest PictureInfo population for HEVC, on the same principle as H.264.
- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
parser); h264 just gained one in startcode. Promote startcode's to the single
shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
(H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
With no active PPS we decline rather than guess: coding stays None, honestly
absent. Set coding = coding_type_only(...) and source = pes.source; field
order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
the no-PPS honest-omission case.
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).
- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
order is NOT decoded here, so field_order() stays None — honestly absent,
never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
absence; BitReader Exp-Golomb table + truncation.
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.
- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
video track's FieldOrder from the MEASURED value, THEN write the header —
right the first time, no guess, no seek-back. An interlaced track that arrives
with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
(the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
faked). Two obsolete tests updated only after confirming (their own comments)
they existed to enforce the deleted hardcoded-TFF.
Matthew Jackson