Round-11 findings from the 10-phase release audit (no real HIGH):
- When a no-declared-duration source (HD-DVD) muxes a degenerate single
frame at tick 0 with no per-frame duration, max_block_ticks stays 0 and
the reserved DURATION placeholder was left as a literal 0.0 (players read
that as a zero-length file). Void the element instead, so the Segment
omits DURATION as an unknown-duration source did before the back-patch.
- Add a regression test for the sniff_video_codec overlap fix (a
picture_start_code whose payload begins 00 00 followed by a real start
code) so the i+=4 marker skip can't silently regress to i+=3.
Round-10 findings from the 10-phase release audit:
- A finder claimed the DTS SFREQ→rate table was wrong at 11/12; verified
it against ffmpeg's avpriv_dca_sample_rates (12k/24k/48k/96k/192k at
11-15) — the table is CORRECT. Added a test that locks the full table so
it can't be mis-"fixed".
- sniff_video_codec advanced 3 bytes after a matched start code, re-reading
the code byte as an overlapping start code; skip the full 4-byte marker.
- Guard the HD-DVD next_id title counter with saturating_add so a crafted
disc with >65536 clips can't overflow (panic in debug).
- Add a test that an undecryptable unit (DecryptFailed) is zero-filled and
counted as loss through ExtractResult (complete=false, bytes_lost>0) —
the recovery-seam consolidation folded that bucket into bytes_unreadable.
Round-9 findings from the 10-phase release audit (no HIGH):
- Remove the MAX_PTS_MARKS backstop and its tautological test: an empty
DTS PES returns before recording a mark, and a non-empty run is already
bounded by the MAX_AU_BYTES buffer clear (which clears pts_marks) — so
the deque cannot grow unbounded and the cap was dead code.
- AuAssembler::for_codec no longer reserves 256 KiB for a Passthrough
stream (audio/subtitle, and every TS/BD stream) whose buf is never
written; only the reassembling modes reserve.
- Correct the scan comment that claimed region is computed (it is a
Region-free stub until region detection lands) and drop a public-repo
reference to internal "private refactor notes" in the mkb module doc.
Round-8 findings from the 10-phase release audit:
- detect_disc_format's BDMV fallback passed detect_format's result
through unchanged, so an SD bonus/menu title could tag a BD-tree disc as
DVD (mis-sizing the ECC sweep) — violating its own "never below Blu-ray"
invariant. Clamp anything but UHD up to Blu-ray.
- Track the MPEG-2 GOP byte total incrementally instead of re-summing the
whole gop_buf on every pushed picture (was O(pictures²) on any MPEG-2
disc, not just adversarial input).
- Back the DTS pts_marks deque with a VecDeque so the over-cap prune is an
O(1) pop_front, not an O(n) Vec::remove(0).
- Add a test exercising parse_stream_id_extension's PTS/DTS skip branches
(the real AU-opening 0xFD video PES path) — previously untested.
Round-7 findings from the 10-phase release audit (no HIGH; convergence):
- Cap DtsParser.pts_marks (MAX_PTS_MARKS): a run of zero-length timed PES
packets grew no buffer bytes, so the drain_front mark-prune never ran —
the deque could accumulate without bound on hostile PS input.
- detect_disc_format tested HVDVD_TS before BDMV while the title-scan
dispatch tests BDMV first, so a disc with both trees would be classified
HD-DVD but enumerated as Blu-ray. Align both to BDMV → HVDVD_TS →
VIDEO_TS.
- Document why the DTS new-PES re-base can emit a locally-decreasing PTS
(the muxer's block_ts applies the strictly-monotonic audio nudge, tested
in mkv.rs) — this is by design, not a mux defect.
- Fix stale aacs/keys.rs comment references (functions moved to
aacs/inf.rs / aacs::resolve/derive in the module split).
Round-6 findings from the 10-phase release audit:
- Wire the documented MAX_PENDING_BYTES byte cap into the MPEG-2 GOP
buffer (it was dead code) and add an equivalent MAX_GOP_BYTES cap to the
sparse-PTS reorder, so a crafted stream of few-but-huge access units
cannot over-allocate — both were bounded only by frame count before.
- probe_evo_streams defaulted an unsniffable HD-DVD video stream to H.264,
which mis-parses a VC-1 (or still-encrypted) clip into a corrupt track.
Emit the video stream only when the codec is actually identified — the
honest outcome, matching the audio path (a real clear clip always carries
its sequence header at the head).
- Resume the AU-opener search from a cursor (like the boundary search), so
a long unsynced junk run is O(bytes), not O(buffer) per push.
- Mark mpeg2's now-dead MAX_AU_BUFFER test-only; restore #[doc(hidden)] on
the aacs probe harness module.
- Add regression tests: the 0xFD video-routing guard, the FMTS-is-UHD key
state, and the GOP byte caps.
Round-5 findings from the 10-phase release audit:
- collect_es routed EVERY extended-stream-id (0xFD) PES into the video ES
buffer, so a 0xFD HD-audio sub-stream (MLP/TrueHD) could pollute the
video sample and — if it preceded the video PES — stamp the video track
with the audio PID, losing the video. Only the VC-1 extension (0x55) is
now treated as video; routing 0xFD audio to its own track is deferred to
the HD-DVD program-chain follow-up.
- The sparse-PTS reorder now carries its calibrated per-frame duration onto
each frame, so the muxer emits a BlockDuration and the back-patched
Segment Duration covers the final frame instead of understating it.
- Add regression tests for the MAX_MARKS and MAX_VTI_HITS caps (promote
MAX_VTI_HITS to module scope); make the differential-test factory array a
named type; drop an identity-op in a reorder test.
Round-4 findings from the 10-phase release audit (the first fully clean
round; it dug into the new #22/#18 refactor code):
- AuAssembler closed each AU from only the FRONT mark's fields, so when
one PES fragment carried the source and a later fragment of the same AU
carried the PTS, the second field was dropped — a regression vs the old
separate pts/source mark deques. Now merge the first Some of each field
across all in-range marks.
- parse_vti_clip_order picked the largest residue bucket with
HashMap::into_values().max_by_key(), nondeterministic on a size tie
(randomized HashMap iteration) — could select a different clip table
run-to-run. Break ties by smallest offset.
- Bound the marks/disc_marks deques (MAX_MARKS): the buf-size cap prunes
marks only when bytes accumulate, so a run of zero-length timed
fragments could grow them without bound on hostile input.
- Add push_owned so the PS path moves the PES payload into a passthrough
AU with no copy (MPEG-2 video + all audio), removing a per-PES
malloc+memcpy the refactor had introduced on the DVD path.
- Back-patch the MKV duration from the block END (start + its own
duration) so it covers the final frame instead of understating by one.
- Add direct tests for the MKB record-framing walker; drop a stale
drain_complete_aus doc comment left on process_au.
AuAssembler::drain rescanned the whole buffered access unit from a fixed
offset on every push, so reassembling one AU split across N program-stream
PES fragments cost O(bytes²/fragment) — amplified on the HD-DVD PS path
where H.264/HEVC/VC-1 frames are large and now flow through this shared
assembler (unlike the TS path, which delivers one AU per PES).
Carry a scan_pos cursor (and, for the stateful VC-1/MPEG-2 rules, a
seen_unit flag) so each push resumes the boundary search where the last
one stopped instead of restarting. Total scan work for one AU is now
O(AU bytes). The from-scratch scanners are retained as a #[cfg(test)]
oracle; a new differential test asserts the resumable path yields
byte-identical AUs at every fragment granularity for all three modes.
Round-1 findings from the 10-phase release audit:
- SparsePtsReorder buffered its current GOP with no bound, draining only on
a keyframe — an open-GOP or crafted program stream that never signals one
could hold the whole title in RAM. Force-complete the GOP at
MAX_GOP_FRAMES, matching the MPEG-2 parser's backstop.
- inject_unit_keys labelled a 2.1 FMTS disc as AACS 1.0 / bus-encryption
off; FMTS is UHD-family, so synthesize the UHD version + bus encryption.
- The compiled Key Correction Data was a non-zero 16-byte constant fed into
the Media Key derivation. Per the no-compiled-keys rule it is now all-zero;
the chain still cannot complete on a real disc (documented), so this is
behaviour-neutral — all variant tests pass unchanged.
- Fix stale doc references (broken `super::variants` intra-doc links, and
`aacs::keys` comments) left by the module rename.
A title whose scan yields no duration (HD-DVD — its `.MAP` timemaps are
not parsed, so DiscTitle.duration_secs is 0) produced an MKV with no
Segment Duration element, so players/MediaInfo reported an unknown
runtime.
Reserve a DURATION placeholder when the source declares none, track the
highest block timestamp written, and back-patch the placeholder at
finish() with the real muxed runtime (also enabling the per-track BPS
tags for these titles). Gated on duration_secs == 0, so BD/UHD/DVD —
which carry a real mpls/IFO duration — write it up-front exactly as
before, unchanged.
The MPEG-2 parser hand-rolled its own PES reassembly — a byte buffer plus
parallel PTS / source / discontinuity mark queues keyed by absolute
offset — duplicating what AuAssembler already does for H.264/HEVC/VC-1.
Add a Mode::Mpeg2 to AuAssembler (picture 0x00 with preceding sequence
0xB3 / GOP 0xB8 headers — the same headers-precede-picture shape as the
VC-1 mode) and have the MPEG-2 parser own one via AuAssembler::mpeg2().
parse() now feeds fragments to the assembler and processes each complete
access unit; the buffer, base offset, and three mark queues are gone. The
GOP-buffered temporal_reference reorder and PTS origin-locking are
unchanged. The parser's external contract is unchanged, so all existing
MPEG-2 parser tests pass as-is; new AuAssembler tests cover the MPEG-2
boundary rule directly.
HD-DVD EVO (and DVD VOB) program streams timestamp video at GOP
granularity: only one access unit per GOP carries a PES PTS. The H.264 /
HEVC / VC-1 parsers collapsed a missing PTS to 0, so on such a source
every non-anchor frame landed on the same block timestamp and a decoder
reported "non monotonically increasing dts".
Add a shared SparsePtsReorder that rebuilds a display-order PTS per frame
from the coded picture type (I/P/B) plus the sparse anchor PTS, with a
per-frame duration self-calibrated from the spacing between consecutive
GOP anchors (no external frame-rate needed). Display order is derived via
the classic single-anchor-delay rule (an anchor displays only after the
previously-held anchor; a B displays immediately), exact for the
non-hierarchical GOP structures HD-DVD H.264/VC-1 use. It mirrors the
MPEG-2 parser's GOP-buffered origin-locking.
Gated to the program-stream path only: the three parsers enable it via
with_ps_reorder(is_dvd_ps), so the BD/UHD transport path (per-frame PTS)
is byte-identical and untouched.
VC-1 HD-DVDs (e.g. Shaun of the Dead) carry video on MPEG-PS extended
stream id 0xFD, with the real stream selector in stream_id_extension
inside the PES extension. Parse that field so the video routes to a
distinct track (pid 0xFD00|ext) instead of being dropped.
Reframe VC-1 access units in AuAssembler with a dedicated Mode::Vc1:
an AU is delimited by the next frame BDU (0x0D) once a frame has already
been seen, so the sequence (0x0F) and entry-point (0x0E) headers that
precede an I-frame stay attached to the frame they describe. The old
single-start-code split stranded those headers on the prior AU, which
the decoder reported as bits-overconsumption and hard decode failures.
hddvd probe now tracks the video pid it detects and emits VC-1 on 0xFD.
- aacs/resolve: a media-keys-only provider missing the VID classifies as
VidUnavailable, not NoMaterial (an MK derives the VUK once the VID
arrives).
- disc/bluray: mark a clip seen only after its .clpi parses, so a
transient parse failure on the first PlayItem cannot suppress the
clip's extents for a later PlayItem referencing it that succeeds.
- disc/patch: log rather than swallow mapfile record/flush failures on a
reverify downgrade, so a failed persist cannot silently mismark a bad
unit good on resume.
- mux/ts: flag a discontinuity when a partial PES is dropped, matching
the other partial-drop paths.
- mux/demux_thread: the no-demuxer branch forwards an empty batch for
early consumer-disconnect detection instead of reading the whole disc.
- io/pipeline: correct the send-timing log (as_secs_f64, not as_micros
printed as ms).
- aacs/derive, aacs/variant, disc/read_error, keysource: comment/doc
accuracy. sector/prefetched, udf: remove dead fields/functions.
- mux/disc: assert unit-aligned read counts in the test.
Replace the AACS-specific inline key-fetch in the decrypt decorator with
a scheme-neutral recovery seam: the input stream (L3) installs a Recover
closure (none / AACS key-fetch) and the decorator (L2) runs it at the
single decrypt-miss point. FMTS (AACS 2.1) forensic-segment units that no
key opens are just undecryptable units, concealed and counted as ordinary
decrypt loss with no FMTS-specific branch ("a loss is a loss"), so the
separate bytes_undecryptable bucket collapses into one loss count.
- sector/recovery.rs: the seam (MissOutcome, none/key_fetch factories),
naming no encryption scheme in its type.
- FMTS: segment routing primitives + BYPASS_FMTS_KEY, and an upfront
ensure_forensic_segments_decryptable gate (Error::FmtsKeyMissing) in
the mux input path, parallel to the unit-key gate.
- CSS descramble/rekey moves from decrypt_sectors into
css::descramble_region: CSS self-recovers from the data itself, so it
stays OFF the seam (which is only for external inputs).
- disc/mod.rs also: main-title selection aligned to largest physical
size; is_regular read from the open file handle, not metadata(path),
fixing a swallowed sync_all on a fresh-rip ISO. decrypt_threads()
resolved once via OnceLock off the per-buffer hot path.
The first cut used a global running clock (max(next, own-PES PTS) + advance),
which fixed the same-PES collision but DRIFTED: once accumulated frame
durations exceeded the PES-timestamp spacing, it never re-based, so a
feature-long DVD DTS track ran minutes past its real length (2h44 for a 2h03
film) while AC-3 from the same source stayed exact.
Match the AC-3 path: re-base to each PES's own container timestamp, and advance
by one frame duration ONLY within a run of AUs sharing one PES. Fixes the DVD
multi-frame-per-PES collision without drift; the UHD DTS-HD MA per-PES
attribution (da85f56) is preserved (each AU still takes its own core PES's PTS).
Adds new_pes_rebases_to_its_own_pts_no_drift; full mux suite green (905).
DVD packs several DTS core frames into one PES; the parser stamped every
access unit with that single PES PTS and duration_ns=None, so consecutive
frames collided on one timestamp — ffmpeg rejected the output as 'non
monotonically increasing dts to muxer: X >= X' (deep-decode = corrupt,
e.g. The Punisher). The UHD DTS-HD MA path (one AU per PES, distinct PTS)
was unaffected, which is why this only surfaced on DVD.
Parse the DTS core header for samples ((NBLKS+1)*32) and sample rate
(SFREQ, 48kHz fallback) to derive each AU's duration, and stamp a running
monotonic PTS: max(next_clock, own-core-PES PTS), then advance by the
frame duration. A later PES whose PTS is ahead of the clock still wins
(preserves the UHD per-PES attribution from da85f56/c49a180); frames
sharing one PES advance frame-by-frame instead of colliding.
Tests: the 3 that encoded 'same PES -> same PTS' now assert monotonic
advance; new dvd_many_cores_one_pes_are_strictly_monotonic reproduces the
Punisher bug; duration/SFREQ-fallback unit tests added.
DiscStream (the live-drive single-pass path) enables AACS decrypt-loss
concealment (NULL-TS fill on an undecryptable unit) but, unlike the file-backed
PipelinedPesStream, had no B1 resync gate — so after a concealed gap it forwarded
inter-coded video frames referencing the now-missing data, producing
decode-broken MKV output (dangling-reference frames).
Mirror PipelinedPesStream: add per-stream ResyncGate + is_video, and route every
codec-frame emit on the TS path (in-stream parse, demuxer EOF flush, parser EOF
flush) through gate.admit(is_video, frame.discontinuity, frame.keyframe) so a
video track drops forward to the next keyframe after a concealment event. Warn
once if a gate is still armed at EOF (a concealed gap in the final GOP with no
trailing keyframe). Audio/subtitle always admit.
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):
1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
never be stranded by an empty post-gap PES. The demuxer only emits
non-empty PES today; this is defense-in-depth for any future caller.
2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
`flush_pts_ns`, dts continues from the most recent known base; truehd
already kept its running cadence on a None PTS. Matches OSS behavior
(PTS rebases off the next PES that actually carries a PTS).
Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.
The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.
Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.
Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.
Three coordinated changes:
1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
an adaptation-field discontinuity_indicator; the demuxer recognises a
0x1FFF packet carrying it as a concealed gap and forces a discontinuity
on every tracked PID (the lost unit's PID is unknowable). This survives
a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
— hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
also drops any open, potentially-truncated partial PES.
2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
STARTING after it, not the one flushed at the boundary (hole 1) —
stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
and admit the real post-gap inter frame.
3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
flag stamped the previous picture. consume_ts (and the EOF flush drain)
gate on frame.discontinuity.
Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.
- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
pending_discontinuity flag set on CC gap / discontinuity_indicator and
carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
until the next keyframe disarms and resumes. Logs the resync + drop
count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
consume_ts. Out-of-range track index emits as-is (defensive).
Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:
- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
undecryptable in-content unit is tallied, overwritten with valid NULL
TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
its LBA, and the read returns Ok — the stream keeps flowing. The rip
paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
(build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
lost video/audio PID packets surface as a CC gap the TS assembler
already drops a partial PES on (the B1 foundation). Ciphertext is never
passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).
Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.
Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).
libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.
Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).
DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.
Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
Surgical fixes (each with a regression test that fails without the change):
mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.
mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.
disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.
mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).
mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.
mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.
mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).
io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).
mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).
Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).
Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).
Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
(0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.
Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.
Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.
Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.
Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.
disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.
disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.
aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.
Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
public fields) to rc6's authoritative crate::mux::codec::PictureInfo
in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
that carries coding (every video codec now fills it), with the
keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
members derived through the accessors: field_order (tff/bff/progressive)
and progressive, emitted ONLY when the codec measured the signal
(Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
/ coding_type_only(..); added measured_cicp: None to VideoStream
literals for rc6's struct.
Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
gop_start), so key is set from the frame's intra / decode-restart flag
(frame.keyframe == coding.keyframe() for video), NOT a fabricated
clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
is updated to document this as a limitation: key is an intra picture /
parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
(closed_gop) is not currently distinguished. §7.1 rewritten for the
new field_order/progressive/nb_fields members.
Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
(#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
for side-channel reuse, sink builds records directly); pub use
fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
scheme(), path_str(), input() (write-only reject) and output()
(constructs FviSink), mirroring the mkv:///demux:// patterns.
Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
frames reached the mux/index with no provenance (FVI src null). Carry
ps.source through, matching the TS path; the real-pipeline fvi test now
sees the stamped src sectors.
Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).
- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
→ coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
absence; interlaced honest-decline.
Honest PictureInfo population for HEVC, on the same principle as H.264.
- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
parser); h264 just gained one in startcode. Promote startcode's to the single
shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
(H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
With no active PPS we decline rather than guess: coding stays None, honestly
absent. Set coding = coding_type_only(...) and source = pes.source; field
order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
the no-PPS honest-omission case.
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).
- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
order is NOT decoded here, so field_order() stays None — honestly absent,
never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
absence; BitReader Exp-Golomb table + truncation.
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.
- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
video track's FieldOrder from the MEASURED value, THEN write the header —
right the first time, no guess, no seek-back. An interlaced track that arrives
with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
(the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
faked). Two obsolete tests updated only after confirming (their own comments)
they existed to enforce the deleted hardcoded-TFF.
- FieldOrder now derives from the bitstream's measured top_field_first
(Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
interlaced content; falls back to TFF when unmeasured. Adds
VideoStream::top_field_first; DVD/BD scan sets None with a precise
TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
enum, so the container stops assuming a colour space the stream may
contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
parser→title channel as top_field_first).
Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
Delete re-implementations in the demux:// sink and wire to proven helpers;
keep only genuinely-new functionality.
- AnnexB reframing: delete the sink's local length_prefixed_to_annexb (it
break'd on a zero-length NAL, dropping the rest of the access unit) and
call the canonical append_length_prefixed_as_annex_b in mux::hevc, which
skips just the empty NAL.
- HEVC param sets: delete hvcc_param_sets; reuse hvcc_to_annex_b.
- avcC param sets: hoist as the new canonical avcc_to_annex_b in mux::hevc,
next to hvcc_to_annex_b (the symmetry point); the sink calls it.
- PGS .sup: emit a synthetic clear display set (empty PCS + END) at
pts + duration_ns so subtitles time out instead of lingering to EOF.
- TimelineContinuity: move verbatim into the shared mux::timeline module
(with the prev_offset straggler-remap intact) and use it from both the
MKV muxer and the demux sink; delete the sink's drifted TimelineRebase
copy (which lacked the straggler branch).
- VobSub .idx: emit the conventional 'id: <lang2>, index: 0' line mkvmerge
reads to assign the subtitle language; palette reuse unchanged.
- output(): seed DemuxOptions.base from title.playlist when non-empty.
New constants for the PGS clear-segment framing and avcC header cite the
public HDMV PGS (BD-ROM Part 3) and ISO/IEC 14496-15 specs.
Tests: a zero-length NAL mid-frame no longer truncates the AU; a frame with
duration_ns produces a .sup clear segment; existing demux tests stay green.
The DVD seek-index concern is that the mux highway
(PsDemuxer -> PipelinedPesStream codec parse -> frame out -> muxer)
might drop the keyframe flag or per-frame duration that Mpeg2Parser sets
on each Frame, which would stop the muxer's cluster/cue open
(keyframe && track 0) from ever firing and leave a DVD MKV with many
clusters and zero cues.
Add dvd_highway_preserves_video_keyframe_and_duration: drives a real
Mpeg2Parser through PipelinedPesStream via DemuxBatch::Ps batches (one
PTS-stamped MPEG-2 PS video PES per GOP, decode-order I + P/B), reads the
frames back through the highway's read(), and asserts the GOP-opening
I-frames arrive as keyframes, every frame keeps its duration (BlockGroup
path), and video routes to track 0. Guards from_codec_frame's
keyframe/duration propagation across the demux-thread + recycled-buffer
highway, which the prior cue tests (codec parser straight into the muxer)
did not exercise.
Verified against the real current-VFR-build output
(Greenland-feature.mkv, today): 3365 clusters / 3365 CuePoints, all
BlockGroups, cues resolve to clusters and track the video — the highway
preserves the flags and the seek index is complete.
The existing cue tests (cue_count_equals_cluster_count,
cue_positions_resolve_to_clusters, cue_times_match_cluster_timestamps)
all feed frames with duration_ns=None, so they exercise only the
SimpleBlock write path (UHD/HEVC). DVD MPEG-2 video is now VFR: every
coded picture carries a per-frame duration_ns=Some(..), so it is written
as a BlockGroup, not a SimpleBlock. That cue path was untested.
Add cue_count_equals_cluster_count_blockgroup_vfr, which drives the real
Mpeg2Parser end-to-end (decode-order frames, non-monotonic B-frame
display PTS, telecine field durations) into the muxer and asserts the
output is wholly BlockGroup (no SimpleBlock) and that the Cues index has
exactly one cue per cluster, with every cue resolving to a real cluster.
Guards the DVD seek index (scrub/fast-forward) against regressing to the
chapter-seek-only, zero-cue state.
New write-only pes::Stream sink that taps the per-track PesFrame stream
(the seam right before MKV muxing) and writes each track to its own
elementary-stream file, plus chapters and per-audio-track delay metadata.
Purely additive — the MKV mux path is untouched.
- mux/demux_sink.rs: DemuxSink + EsWriter dispatch. Pass-through for
codecs whose Frame.data is already standalone ES (MPEG-2, VC-1, AC3/
E-AC3, DTS/DTS-HD, TrueHD, LPCM). Non-trivial writers:
- AnnexBWriter: reframes hvcC/avcC 4-byte-length-prefixed NALs to
Annex-B and prepends VPS/SPS/PPS parsed out of the codec_private
configuration record (HEVC .hevc / H.264 .h264).
- PgsSupWriter: rebuilds the HDMV 'PG' segment framing the parser
strips, with 90kHz PTS/DTS (.sup).
- VobSubWriter: writes raw SPUs to .sub and synthesizes the .idx
sidecar (palette + per-SPU timestamp/filepos).
- Delay-in-filename (mkvmerge-readable 'DELAY <n>ms') + chapter XML/OGM
export. TimelineRebase ports the MKV muxer's seamless-branch epoch
logic so per-track ES timestamps stay continuous across clip joins.
- mux/resolve.rs: StreamUrl::Demux variant + scheme/path_str/parse_url/
input(write-only)/output arms.
- mux/mod.rs: module + public type re-exports.
16 unit tests: Annex-B reframing, hvcC/avcC param extraction, delay
sign/rounding + mkvmerge-regex match, PGS .sup framing, VobSub .idx
synthesis, chapter XML/OGM, timeline rebase, and end-to-end file-keying
by track + track selection.
A genuinely-clear or uncrackable extra title (a tiny menu/nav stub) no
longer poisons a multi-title rip with a false CssKeyMissing (E7023).
- decrypt_keys_for_title_checked: re-crack a non-overlapping VTS via
crack_key_outcome and report title_is_clear when the title's own
extents show no scrambling. A genuinely-clear stub on an otherwise-CSS
disc needs no key.
- ensure_title_decryptable: pass a clear stub without a key; a scrambled-
but-uncrackable title still hard-fails with CssKeyMissing.
- is_scrambled_pack: hardened scramble-evidence gate for the crack scan —
requires the MPEG-PS pack-start signature before trusting the 0x14
scramble bits, so a clear stub with stray 0x14 bits can't flip
saw_scrambled. The descramble loop keeps the looser is_scrambled.
- mux/resolve: ISO per-title gate routes through the clear-aware check.
Sibling of Disc::copy specialized to write per-file instead of a whole ISO
image, decrypting on the way out: walk the UDF tree, read each file's extents
through the shared DecryptingSectorSource (AACS unit-aligned, CSS per-VTS),
strip AACS/, sanitize host paths per component, .partial+rename, 1-shot with
per-file loss accounting (no mapfile; recovery stays the iso:// multipass
path). Reuses UdfFs + the decrypt seam; only the per-file orchestration is new.
Adds Disc::ensure_decryptable / ensure_decryptable_keys, the single decrypt
gate consulted before any copy or mux. When the source is encrypted and no
key resolved (and not --raw), abort with a typed error and write nothing,
instead of silently emitting ciphertext at exit 0. Unifies the prior ad-hoc
CSS/AACS checks.
Matthew Jackson