Reshape the Unlocker seam so every unlocker is dispatched at ONE place from
ONE ordered registry — the firmware, cert, and CSS routes are all "remove the
bus-encryption barrier", differing only in what they key off. matches() and
unlock() now take an UnlockCtx { drive_id, kind: DiscKind } instead of a bare
DriveId: a firmware unlocker keys off drive_id (kind irrelevant), the cert
unlocker will match DiscKind::Aacs, the CSS unlocker DiscKind::Css. UnlockCtx
is #[non_exhaustive] so a host-cert source can be added without breaking
external unlockers. Drive-prep dispatch passes DiscKind::Unknown (no disc
probed yet); the cert/CSS registry impls + the single post-probe dispatch
point follow in subsequent commits.
Thread &mut dyn ScsiTransport through unlock_css_reads and its bus-auth /
disc-key primitives instead of &mut Drive — they only ever issued SCSI via
drive.scsi_mut(), so this is mechanical and lossless, and it makes the CSS
unlock speak the same transport interface as the registry Unlocker trait
(prerequisite for CSS becoming a uniform unlocker).
Also clean up the badly-named, half-dead bus_auth:
- rename bus_auth -> establish_authenticated_session: it is run for its
side effect (sets the drive's ASF=1, unlocking scrambled-sector reads),
which the name now states.
- drop the derived CSS bus key from the return + computation: it had no
consumer (descrambling is keyless via the Stevenson attack), so it was
dead crypto computed on every DVD unlock. Return just the negotiated AGID,
which the caller genuinely needs for the best-effort disc-key REPORT KEY.
The AACS cert-auth primitives (aacs_authenticate, the AACS 2.0 P-256
variants, read_volume_id, read_data_keys) and their scsi_read/scsi_write
helpers touched the drive ONLY through Drive::scsi_execute — a pure
pass-through to the transport. Thread &mut dyn ScsiTransport instead of
&mut Drive so these primitives are transport-level, matching the firmware
Unlocker seam (which hands out &mut dyn ScsiTransport for testability).
Pure mechanical signature change, no logic change; the cert orchestrator
(do_handshake_cert) keeps &mut Drive for the OEM-VID shortcut and passes
session.scsi_mut() into the primitives. Step toward making the cert
handshake a uniform registry unlocker.
Generalize the unlock seam toward uniform unlockers (firmware / cert / CSS).
`unlock` now reports what it LEARNED — a Volume ID and/or an AACS bus key —
instead of just a Vid; libfreemkv files those onto the disc/drive in one
place, so a plugin never touches Disc. The hardware side-effect (extended
mode) still happens inside unlock. route_unlock walks the registry in
registration order (the single ordered place that decides run order); no
priority field. Firmware route reports {vid: Some, read_data_key: None}.
The bus-key gate only credited the cert handshake's read_data_key as proof
bus encryption was removed. A firmware unlocker removes it AT THE DRIVE
(serves clear content) and yields no read_data_key — so a SUCCESSFUL
firmware unlock (VID present, read_data_key None) tripped the gate and
blocked ALL key resolution, including the online source. That was the
root cause of live UHD discs reporting "missing keys" after an unlock.
Now a single predicate answers "is bus encryption gone?": never-had-it ||
file/ISO || firmware-unlocked || cert-bus-key. The gate is just
`if !bus_encryption_removed { error }` — no enumerated cases. HandshakeResult
gains `drive_unlocked`, and the read_data_key failure reason is captured so
the warn says WHY the bus key is missing.
Also: reword the first hardware-sense escalation as "fast-fail escalation"
(it is often transient — the drive recovers), reserving "wedge" for a
persistent run; and scrub the product name from core comments (it belongs
only in the unlocker crate).
When a bus-encrypted disc's handshake yields no read_data_key, the gate
logged a bare "bus_key_unavailable" with no indication of WHY — turning
every occurrence into archaeology (is the bus key not-attempted, or did
the read fail?).
Now:
- read_data_keys failure is captured (error code) instead of swallowed by
`.ok()`, and logged at the handshake with its consequence.
- HandshakeResult carries `read_data_key_err: Option<u16>` so the gate
distinguishes "never attempted" (None — VID-only/OEM path) from "read
FAILED" (a code), and the bus_key_unavailable warn now reports the code
plus whether a Volume ID was present.
No behavior change — purely diagnostic. The handshake_ok debug also now
reports has_volume_id.
Two defensive hardenings from the post-fix audit (vs FFmpeg/GStreamer):
1. Move the `pes.discontinuity` partial-drop ABOVE the empty-data guard in
all three audio parsers (ac3/dts/truehd), so a discontinuity signal can
never be stranded by an empty post-gap PES. The demuxer only emits
non-empty PES today; this is defense-in-depth for any future caller.
2. A PES with no PTS must not reset the timeline to 0. ac3 now carries
`flush_pts_ns`, dts continues from the most recent known base; truehd
already kept its running cadence on a None PTS. Matches OSS behavior
(PTS rebases off the next PES that actually carries a PTS).
Adds an ac3 regression test (empty-payload discontinuity PES still drops
the stranded partial). Loss accounting was reviewed: TS-demux CC-gaps are
NOT counted toward lost_video_secs / abort (that is sector-based via
DiscStream::errors / mapfile bytes_unreadable), so a source splice never
inflates loss — no gating needed there.
The AC-3, DTS and TrueHD parsers buffer access units across PES
boundaries. At a concealed-loss gap the buffered unit is truncated:
splicing post-gap bytes onto it manufactures a corrupt frame on top of
the real loss (FFmpeg "Failed to decode block code(s)" / "Invalid data
found" at the gap) and, for TrueHD, strands the PTS cadence into the
non-monotonic audio-DTS band seen on multi-clip titles.
The video parsers already handle this via the ResyncGate, but the
discontinuity signal was only wired into video — audio parsers ignored
pes.discontinuity and spliced across the gap.
Now, when pes.discontinuity is set, each audio parser drops the partial
(clears buf, and for DTS its PTS marks / pending base) so the post-gap
PES re-bases a fresh unit. A lost gap degrades to a clean single-frame
drop instead of a corrupt spliced frame. No effect on perfect rips: the
branch only runs when concealment inserted a discontinuity marker.
Adds a per-parser test feeding a partial frame then a discontinuity PES,
asserting the truncated partial is dropped (not spliced) and the post-gap
PTS is adopted.
Pass N (patch) pinned the slow recovery speed (0x0000) for the whole pass.
But Pass 1's damage-jump overshoots, so most of a jumped range is clean data
the reverse-walk reads first — grinding it at slow speed wastes minutes per
gap. Now each range mirrors Pass 1's model:
- Enter at max speed (0xFFFF) + reset to the initial batch (current_batch
carries across ranges, so a prior single-sector grind would otherwise start
the next range slow). Reads the clean overshoot fast.
- First read failure in the range → drop to 0x0000 (once; idempotent SET CD
SPEED) for the rest of that range — grind only the genuine damage.
- 10 s halt-responsive cooldown between ranges, gated on "this range actually
grinded" so a many-small-range pass (100+ ranges) doesn't stall ~20 min on
unconditional pauses.
No change to good/bad classification, the mapfile, or the abort math — purely
read speed + an inter-range cooldown. Synthetic speed-transition test to follow.
Non-blocking follow-ups from the 1.2.0 audit:
- conceal loop: if decrypt reported loss but the padding-aware predicate
matched nothing to conceal (a ~256^-31 contradiction), fall back to the
strict predicate and conceal whatever it flags, loudly — belt-and-
suspenders so ciphertext can never reach the mux.
- decrypt_dropped doc: reflect 1.2.0 (mux-path loss is concealed + tallied,
not silently dropped).
- direct unit test for aacs_unit_still_ciphertext (the padding-aware
conceal predicate): clear/all-zero/full-decrypted/full-ciphertext/
decrypted-short-tail.
- fix three stale "v1.1.1" comment refs (the fragment-tail fix ships in
1.2.0; there is no v1.1.1 release).
Closes the three residual holes where a concealed/lost gap could still let
a dangling-reference frame reach the muxer (degraded/undecryptable-disc
path only; clean rips are byte-identical and untouched). Root cause: the
discontinuity signal was reconstructed from the 4-bit continuity counter
and applied per-PES, both of which are lossy.
Three coordinated changes:
1. CC-INDEPENDENT marker. fill_null_ts_unit now tags its NULL packets with
an adaptation-field discontinuity_indicator; the demuxer recognises a
0x1FFF packet carrying it as a concealed gap and forces a discontinuity
on every tracked PID (the lost unit's PID is unknowable). This survives
a loss that is an exact multiple of 16 packets (CC aliases to in-sequence
— hole 3) and a loss at a PID's very start (no prior CC — hole 4); it
also drops any open, potentially-truncated partial PES.
2. PUSI ATTRIBUTION. A gap landing on a PES boundary now flags the PES
STARTING after it, not the one flushed at the boundary (hole 1) —
stamping the pre-gap frame could arm-then-disarm the gate on a keyframe
and admit the real post-gap inter frame.
3. PER-FRAME signal. codec::Frame gains `discontinuity`; each parser
propagates it onto the first post-gap frame. MPEG-2 buffers whole GOPs
asynchronously, so it associates the gap by ES OFFSET (like PTS/source),
landing it on the exact post-gap picture mid-GOP (hole 2) — a per-PES
flag stamped the previous picture. consume_ts (and the EOF flush drain)
gate on frame.discontinuity.
Tests: CC-independent marker with in-sequence CC + leading-loss; PUSI
attribution flags the post-gap PES; MPEG-2 offset-mark stamps the post-gap
picture through GOP reorder, not the previous one. Existing B1 gate + EOF
tests still green (2270 lib tests).
Per the release decision: ship the AACS-version-stride API changes under
the 1.2.0 minor bump, with an explicit Breaking section (DiscInputs new
field, DiscInputsCtx::new arity, read_aacs_inputs* 3-tuple return).
The P3 concealment loop in DecryptingSectorSource::read_sectors keyed on
aacs_unit_needs_decrypt, whose sync check is the majority-vote
ts_sync_destroyed (<=16 of 32 syncs). A successfully padding-aware-
decrypted content-fragment TAIL unit (e.g. 11 content packets + 21 zero
padding) has only 11 syncs, so the majority vote called it "still
encrypted" — and when such a good unit shared a read buffer with a
genuinely-undecryptable one (dropped>0), the loop overwrote the GOOD
decrypted tail with NULL-TS, silently discarding correct video and
over-counting concealed units vs the tallied dropped bytes.
Add aacs_unit_still_ciphertext (padding-aware): encrypted AND at least
one non-zero (non-padding) 192-byte packet missing its 0x47 sync — the
same discriminator decrypt_unit uses to accept a fragment tail. The
conceal loop now uses it, so only genuinely-unrestored ciphertext is
concealed. Full and fully-ciphertext units are unchanged. Regression
test: a decrypted short-padding-tail co-resident with a failed unit is
left byte-for-byte intact while the failed unit is concealed.
Pairs with A2 (read-path NULL-TS concealment). When the demux assembler
sees a TS continuity gap it now stamps `discontinuity` on the next
completed PES; the codec-parse stage carries that onto a per-track
ResyncGate. After a gap on an inter-coded video track the gate drops
forward to the next IRAP/IDR keyframe so no frame with a dangling
reference reaches the muxer (an ffmpeg deep scan would otherwise report
a missing-reference / non-existing-PPS error). Audio and subtitle tracks
have no cross-frame references, so the gate is a no-op there.
- ts.rs: PesPacket gains `discontinuity`; PesAssembler tracks a sticky
pending_discontinuity flag set on CC gap / discontinuity_indicator and
carried to the next completed/flushed PES.
- resync.rs (new): ResyncGate — per-track arm-on-gap, drop non-keyframes
until the next keyframe disarms and resumes. Logs the resync + drop
count once at the keyframe.
- pipelined_stream.rs: precompute per-track is_video, apply the gate in
consume_ts. Out-of-range track index emits as-is (defensive).
Tests: ResyncGate unit tests; ts.rs gap-stamps-discontinuity; end-to-end
B1 video-drops-to-keyframe and audio-never-drops through PipelinedPesStream.
Decrypt-verify is a RIP gate, not a MUX gate. On the mux read path an
undecryptable content unit must never abort the mux:
- DecryptingSectorSource gains tolerate_decrypt_loss(): when set, an
undecryptable in-content unit is tallied, overwritten with valid NULL
TS packets (PID 0x1FFF) via aacs::fill_null_ts_unit, logged loud with
its LBA, and the read returns Ok — the stream keeps flowing. The rip
paths keep the fail-loud DECRYPT_VERIFY_READ decorator (re-read off the
disc); only the mux opts in.
- Wire it into both mux read paths: the file-backed highway
(build_iso_pipeline) and the inline DiscStream.
- NULL-TS fill keeps the demuxer byte-synced on the 192-byte stride; the
lost video/audio PID packets surface as a CC gap the TS assembler
already drops a partial PES on (the B1 foundation). Ciphertext is never
passed downstream either way.
- Fix stale resolve_vid_only no-cert test: default is UHD (audit #4).
Tests: conceal-as-NULL-TS, fill well-formedness, fail-loud still holds.
- One workspace hex parser (libfreemkv::hex): the keydb / online / mapfile
parsers had divergent prefix handling (0x vs 0X vs none) — a key written with
a prefix one parser didn't expect was silently dropped. All three now call it.
- read_encrypted_units probes several points spread across each extent instead
of only the midpoint-forward window, so a late-starting / sparse encrypted
body still yields samples (empty samples make decrypt_with skip wrong-key
validation). A read error at one probe no longer abandons the extent.
- read_aacs_inputs* now returns the AACS major version; DiscInputs carries it,
and DiscInputsCtx parses Unit_Key_RO.inf at the disc's own stride (fixes the
hardcoded-V20 read-time fetch for V10 discs). One source of truth, no version
argument to drift.
- Disc::inputs() is the single complete AACS-input source (inf/MKB/VID/hash/
version); the out-of-band duplicate readers go away.
- Named constants for AACS file paths (aacs::PATH_*) and the AACS majors
(aacs::AACS_MAJOR_*, AacsVersion::major/from_major) replace magic strings/ints.
- push_ranges saturating (corrupt-disc panic guard).
- decrypt_unit: padding-aware acceptance — recover real video at content-
fragment tails (the phantom mux-loss class) without weakening wrong-key
rejection (a full content unit still needs all 32 TS syncs).
- scan: read the MKB via the bounded read_mkb_content so Disc::inputs()
carries it. Online key resolution was shipping mkb=0 (a full read of the
~128 MiB MKB_RO allocation fails) → the decode service 404'd.
- resolve_vid_only: surface an MKB read error instead of silently emptying.
- fetch: a per-sample dry-set replaces the global fetch_spent latch, so a
second CPS unit's key can still be fetched after the first came back empty.
- verify::push_ranges: saturating arithmetic (corrupt-disc panic guard).
- Tests for all of the above.
ContainerKind {Ts,Ps} + ClipLayout.container thread the post-decrypt structural check per clip; decryptability() dispatches it (TS: unit_is_clean_ts, PS: unit_is_clean_ps). New decrypt_unit_checked(unit,key,accept) decouples the container-agnostic AACS crypto from the format-specific acceptance (decrypt_unit delegates with the TS check). unit_is_clean_ps is the MPEG-2 PS pack-start check, documented UNVALIDATED for HD-DVD (.evo unit/seed/pack alignment must be confirmed on real media). clip_layouts assigns Ts today; .evo->Ps is the one-line HD-DVD hook.
reverify_iso now takes an is_finished predicate and SKIPS any unit with a non-Finished backing sector: we can't verify what wasn't read (a non-Finished sector is zero-filled because the drive read failed there), and must never waste a key lookup on a block the read already knows is bad. observe() (sweep) was already safe (only fed Good bytes).
Post-read verify gate (new src/disc/verify.rs): UnitVerifier buffers/aligns the disc-absolute read stream into clip-file 6144-byte units, then makes one decryptability() decision per unit (CPI gate -> held keys -> key_fetch -> strict TS). POST_READ_VERIFY const kill-switch; fail-safe contract (only ever downgrades units it is confident are undecryptable; every doubt skips). Hooked into Disc::sweep (producer observes ciphertext -> WorkItem::MarkBad after the Good, FIFO-ordered) and Disc::patch (post-loop reverify_iso reads recovered units whole from the patched ISO). extract::clip_layouts enumerates AACS clips for the gate.
Standards-correct AACS verify: aacs::unit_is_clean_ts is a strict port of libaacs _verify_ts (all 32 TS syncs, not a majority vote); decrypt_unit accepts a key only on it; the majority verify_ts is removed. Deleted the Disc::verify_clips post-pass bolt-on (its primitive is absorbed by the read-path gate).
libaacs/DVD audit fixes: content-cert bus_encryption flag now read from bit 7 (was bit 0 - defeated the bus-key fail-loud gate); cc_id read from offset 14; title_cps_unit range-validated + 1->0 index-converted per libaacs. Corrected attack_crib ("functionally-equivalent" not "exact" port) and read_disc_key (READ DVD STRUCTURE 0xAD, not REPORT KEY) doc comments.
Also includes accumulated uncommitted work: key-fetch seam and TrueHD/DTS audio fix.
Use 'Semantics / reference' as the last-column header across all spec
tables, and break enum/option/note cells onto separate lines with <br>
for readability. Formatting only; no semantic changes.
The CLI --version, the MKV muxing/writing-application field, and the FVI generator
all derive from one libfreemkv const, so a binary reports the exact same label it
stamps into the files it produces — no split-brain where an MKV claims one version
and the binary another.
Reformat all three JSON code blocks (section 8 ext example, Appendix A
Header schema, Appendix A Record schema) with consistent 2-space
indentation and valid syntax. No field or semantic changes.
build.rs emits FREEMKV_VERSION = FREEMKV_BUILD_LABEL when that env is set
(non-empty), else the Cargo package version. The muxing/writing-application
field and the FVI generator tag use it, so a pre-release/test build can be
stamped without bumping Cargo.toml and disturbing the tag-pinned [patch]
version matching. The git short hash is still appended either way.
TrueHD: when the PES PTS lags the access-unit cadence, resync to the PTS
but never snap the running timestamp backward, so the emitted DTS stays
monotonic across the resync (next_pts_ns = max(next_pts_ns, pts)).
DTS-HD MA: size each EXSS extension substream exactly from its header
(exss_frame_size) and skip it as a unit, so a false 0x7FFE8001 core sync
inside the lossless extension payload can no longer split the access unit
and truncate the extension. Falls back to a bounded scan when the header
is unparseable.
Provenance: build.rs bakes the git short hash into GIT_SUFFIX; the muxing/
writing-application field and the FVI generator tag now record the exact
build (e.g. "freemkv 1.1.0-beta.1 (g835cc99)"), so any output file is
traceable to the revision that produced it.
keydb I/O moves out of libfreemkv into freemkv-keysources
(KeydbSource::save / ::update). Delete src/keydb.rs entirely (save,
http_get, default_path, write_atomic, UpdateResult) and drop `pub mod
keydb;` — http_get had no real callers. The shared Keydb* Error variants
stay in error.rs (keysources raises them; the every-error-has-a-code
contract depends on them). flate2 is no longer used here, so drop it
from Cargo.toml (zip stays for labels/jar.rs).
CHANGELOG: note the keydb-I/O move; reword the rc.5.2
DefaultDecodedFieldDuration entry to state only the action taken (the
revert) rather than an unverified Windows-fps outcome.
Surgical fixes (each with a regression test that fails without the change):
mux/mkv.rs, mux/demux_sink.rs: drive the clip-boundary timeline epoch
off the resolved PRIMARY VIDEO track, not the literal stream index 0.
An M2TS/PMT title can list an audio ES before video, so streams[0] may
be audio; a non-video epoch driver ratchets the frontier and inflates
the timeline. mkv cluster-opening falls back to track 0 for audio-only
titles so they still open clusters.
mux/codec/ac3.rs: correct ACMOD_CHANNELS — acmod=5 (3/1) is 4 channels,
not 3 (was undercounting a 3/1 stream); fix the A/52 Table 5.8 doc.
disc/mod.rs: HDMV coding_type 0x91 (Interactive Graphics / menus) no
longer maps to PGS subtitle — it falls through to Unknown so the PMT/STN
walker drops it instead of surfacing a bogus subtitle track.
mux/videomap.rs + mux/mkv.rs: FVI colour now mirrors the MKV muxer's CICP
precedence (measured CICP authoritative; HDR-driven PQ/HLG transfer
override) via a shared cicp_for_video helper, so the two sinks can't
disagree (HDR10 BT.2020 no longer emits SDR transfer 14).
mux/mkvstream.rs: saturating_add on cluster_ts + rel_ts so an adversarial
CLUSTER_TIMESTAMP near i64::MAX can't overflow/panic before the existing
saturating_mul.
mux/timeline.rs: tighten the tail-straggler clamp so a normal new-epoch
non-video frame leading the sparse video frontier by >3s is not demoted
into the previous clip's epoch.
mux/m2ts_mux/mod.rs: re-stamp PCR per video TS packet (mid-PES), not only
at PES boundaries, so a large UHD I-frame can't open a multi-second PCR
gap; modular 33-bit PTS rebasing so a real 90 kHz clock wrap is not
collapsed to PTS 0 (pre-base frames still floor to 0).
io/byte_prefetcher.rs, sector/prefetched.rs: wrap the producer feed loop
in catch_unwind and emit a typed error sentinel on panic, so a mid-stream
producer panic is not read as a clean EOF at the demux boundary (which
would silently truncate the mux).
mux/codec/h264.rs: extend HIGH_PROFILES to the full ISO/IEC 14496-15 set
that mandates the avcC chroma/bit-depth extension (adds 244 et al.).
Doc/comment accuracy: css/mod.rs (50000 sectors, not scrambled-sectors),
aacs/decrypt.rs (decrypt_unit already-clear path), ifo.rs (TT_SRPT at
0xC4), css/lfsr.rs (LFSR0 24-bit; TAB1-then-XOR cipher; real scramble-flag
predicate), disc/read_error.rs (for_sweep does bounded transient retries).
Skipped: keydb.rs SSRF guard (low/latent, no live caller) — a hard
loopback block breaks an existing behavioral test that exercises the
header-EOF path over a loopback server; a clean fix needs a resolver test
seam beyond this surgical pass. The sibling keydb_fetch.rs comment fix is
out of scope (freemkv crate).
Parse the two HDR10 HEVC SEI messages and emit the corresponding
Matroska Colour metadata, only when actually present in the bitstream
(SDR / no-SEI tracks omit it; nothing is fabricated).
Parse (Rec. ITU-T H.265 Annex D):
- Mastering Display Colour Volume SEI, payloadType 137 (D.2.28):
display_primaries_x/y[3] (SEI order G,B,R), white_point_x/y
(0.00002 units), max/min_display_mastering_luminance (0.0001 cd/m²).
- Content Light Level Info SEI, payloadType 144 (D.2.35):
MaxCLL / MaxFALL (cd/m² integers).
HevcParser::scan_sei walks the sei_rbsp ff-extension payloadType/
payloadSize coding and de-emulates (00 00 03) before reading, reusing
the existing strip_emulation_prevention helper. Both SEI are required
before any metadata is surfaced; SEI NALs still pass through unchanged.
Carry: the measured Hdr10Metadata rides PictureInfo (the same per-coded-
picture seam FieldOrder uses), flowing through from_codec_frame onto
PesFrame.coding to the deferred-muxer activate path, where
apply_coding_to_track stamps it on the video track before the header is
written. Set only when both SEI were seen.
Emit (RFC 9559 / Matroska): new Colour children in ebml.rs
(MasteringMetadata 0x55D0, Primary R/G/B + WhitePoint chromaticity
0x55D1..0x55D8, Luminance max/min 0x55D9/0x55DA, MaxCLL 0x55BC,
MaxFALL 0x55BD). write_hdr10 converts chromaticity SEI int × 0.00002 →
Matroska float, luminance SEI int × 0.0001 → cd/m² float; MaxCLL/MaxFALL
are uints verbatim. SEI primary index 0/1/2 (G/B/R) mapped to the
Matroska R/G/B element layout. Emitted only when hdr10 is present.
Tests: SEI parse with exact raw values, requires-both-SEI, SDR omission,
and emulation-prevention stripping (hevc.rs); muxer emit with exact unit
scaling + SDR omission of MasteringMetadata/MaxCLL/MaxFALL (mkv.rs);
apply_coding_to_track HDR10 plumbing (mkvstream.rs).
Owner-flagged invariant: freemkv -t 1 ALWAYS selects the main feature because the
CLI's title 1 maps to titles[0] and the list is ordered by canonical_title_order
(main feature first). Adds a DVD-shaped contract pin asserting titles[0] is the
movie after sorting (a regression there is a title-ordering bug, not a remux issue),
complementing the existing branching-UHD / normal-disc order tests.
Also clears two pre-existing test-only clippy nits surfaced under --all-targets:
unused `lba` in ClearStubReader::read_sectors, and an unneeded `mut` on the
h264 population test's closure.
mux/mkv: assert emitted CODEC_PRIVATE bytes verbatim for H.264/HEVC/VC-1/
MPEG-2 (direct TrackEntry child, not nested in Video) and DefaultDuration ns
for all eight frame rates, read back out of a real MkvMuxer.
disc/sweep: end-to-end Disc::sweep against a synthetic MockReader with an
injected bad region, asserting the resulting mapfile marks the clean lead
Finished and the failed batch + zero-filled skip-ahead gap NonTrimmed,
proving the Pass-1 damage-jump engaged.
disc/patch: introduce a minimal clock seam (fn() -> Instant on the internal
PatchLoopState, defaulting to Instant::now) so the per-range and whole-pass
watchdogs are deterministically testable; public API and callers unchanged,
production behavior identical. Add tests that advance a fake clock to trip the
range budget and whole-pass stall predicate.
aacs: add an AES-128-CBC known-answer test for aes_cbc_decrypt using the
published NIST SP 800-38A F.2.2 vector (blocks 1..3 exact; block 0 via the
documented fixed-AACS-IV substitution).
Port + adapt the freemkv native per-picture video index (FVI) from the
old feat/fvi-sink branch onto rc6's codec-agnostic PictureInfo model.
This is a surgical adaptation, not a merge.
Adaptations (fvi_sink.rs, videomap.rs, tests/fvi_pipeline.rs):
- Retarget from the removed crate::mux::codec::mpeg2::PictureInfo (raw
public fields) to rc6's authoritative crate::mux::codec::PictureInfo
in codec/coding.rs, via its accessors.
- type from coding_type() -> CodingType{I,P,B}; emitted for ANY frame
that carries coding (every video codec now fills it), with the
keyframe-flag I/P fallback only when coding is absent.
- Replace the mpeg2-only tff/rff/progressive members with codec-agnostic
members derived through the accessors: field_order (tff/bff/progressive)
and progressive, emitted ONLY when the codec measured the signal
(Option::Some) and omitted otherwise; plus nb_fields.
- Test fixtures rebuilt via PictureInfo::mpeg2(CodingType, Mpeg2Coding{..})
/ coding_type_only(..); added measured_cicp: None to VideoStream
literals for rc6's struct.
Honesty decision (key / random-access):
- The codec-agnostic PictureInfo carries NO GOP-closure (no closed_gop /
gop_start), so key is set from the frame's intra / decode-restart flag
(frame.keyframe == coding.keyframe() for video), NOT a fabricated
clean-RAP claim. The old gop member is honestly omitted. FVI_FORMAT.md
is updated to document this as a limitation: key is an intra picture /
parser-flagged decode-restart point; MPEG-2 open-GOP clean-RAP precision
(closed_gop) is not currently distinguished. §7.1 rewritten for the
new field_order/progressive/nb_fields members.
Wiring:
- mux/mod.rs: pub(crate) mod fvi_sink; pub(crate) mod videomap
(#[allow(dead_code)] on videomap — the VideoMap accumulator is staged
for side-channel reuse, sink builds records directly); pub use
fvi_sink::FviSink.
- mux/resolve.rs: add the fvi:// output scheme to StreamUrl, parse_url,
scheme(), path_str(), input() (write-only reject) and output()
(constructs FviSink), mirroring the mkv:///demux:// patterns.
Provenance fix surfaced by the end-to-end test:
- pipelined_stream::consume_ps was dropping the PS demuxer's byte-exact
source stamp (source: None) when rebuilding PesPacket, so PS/DVD-path
frames reached the mux/index with no provenance (FVI src null). Carry
ps.source through, matching the TS path; the real-pipeline fvi test now
sees the stamped src sectors.
Gate: cargo +1.86 fmt + clippy --lib -D warnings clean; cargo +1.86 test
--lib (2182 passed) and --test fvi_pipeline (2 passed); precommit.sh
libfreemkv green.
Honest PictureInfo population for VC-1 — completes the set (MPEG-2, H.264, HEVC,
VC-1 all populate now).
- Read the advanced-profile sequence header INTERLACE flag (SMPTE 421M §6.1.1,
bit 41), de-escaping emulation-prevention bytes as parse_vc1_resolution does.
- For a PROGRESSIVE sequence, decode the picture-layer PTYPE VLC (0=P, 10=B,
110=I, 1110=BI→I, 1111=Skipped→P) from the first bits after the frame start
code. For INTERLACED (FCM/FPTYPE precede PTYPE) or simple/main/unknown, decline
→ coding stays None: honestly absent, never read at a wrong bit offset.
- Set coding = coding_type_only(...) and source = pes.source; field order is not
decoded, so field_order() is honestly None.
- Reuses the shared startcode::BitReader (no new primitive).
- Tests: progressive I/P/B from real PTYPE VLCs + source carry + field-order
absence; interlaced honest-decline.
Honest PictureInfo population for HEVC, on the same principle as H.264.
- Consolidate the bit reader: hevc.rs had its own BitReader (used by the SPS
parser); h264 just gained one in startcode. Promote startcode's to the single
shared pub(crate) reader (adds read_bits) and delete hevc's copy — one proven
primitive, reused (SPS parse + both slice-type decoders).
- hevc: decode slice_type from the first coded slice's slice_segment_header
(H.265 §7.3.6.1) → I/P/B (§7.4.7.1). The offset to slice_type depends on
num_extra_slice_header_bits, which lives in the PPS — so we parse it from the
ACTIVE PPS (§7.3.2.3) and only measure slice_type when that PPS is known.
With no active PPS we decline rather than guess: coding stays None, honestly
absent. Set coding = coding_type_only(...) and source = pes.source; field
order (pic_struct SEI) is not decoded, so field_order() is honestly None.
- Tests: I/P/B from real slice headers, source carry, field-order absence, and
the no-PPS honest-omission case.
Honest PictureInfo population for H.264 — the foundation that lets the video
index claim a frame's type and be believed (a faked "P" on a B-frame would make
the index confidently wrong).
- startcode: shared minimal MSB-first BitReader with Exp-Golomb ue(v), reused by
H.264 (and next HEVC). Documents the emulation-prevention caveat: only the
leading slice-header fields are read, where 00 00 03 cannot intervene.
- h264: decode first_mb_in_slice + slice_type (H.264 §7.3.3) from the first
coded slice and map to I/P/B (§7.4.3 Table 7-6; SP→P, SI→I). Set
coding = PictureInfo::coding_type_only(...) and source = pes.source. Field
order is NOT decoded here, so field_order() stays None — honestly absent,
never guessed.
- Tests: I/P/B from real ue-encoded slice headers, source carry, field-order
absence; BitReader Exp-Golomb table + truncation.
Carry per-picture truth and byte-exact source provenance THROUGH the stream so
the muxer (and the upcoming video index) read MEASURED facts instead of
assuming them. Honest data in, honest data out.
- codec/coding.rs: codec-agnostic PictureInfo (CodingType / FieldOrder + the
accessors field_order/coding_type/nb_fields/progressive/keyframe). Each codec
folds its raw signals in; consumers use only accessors, never branch on codec.
- mpeg2: builds PictureInfo from the picture coding extension and carries it +
SourcePos (source_marks, parallel to pts_marks) on every emitted frame.
- pes / codec::Frame: additive `coding` + `source`, forwarded through the
highway; None for audio/subtitle and the network/stdio deserialize hop.
- mkvstream: DEFER muxer construction until the first coded picture, set the
video track's FieldOrder from the MEASURED value, THEN write the header —
right the first time, no guess, no seek-back. An interlaced track that arrives
with no measured order is LOGGED loudly and left UNDETERMINED, never faked.
- mkv: MkvTrack::video no longer guesses TFF (a bitstream property the scan
cannot know is UNDETERMINED at build). Removed VideoStream::top_field_first
(the dead scan-time guess) crate-wide.
- Tests: parser population (every PictureInfo facet + per-PES source carry) and
mux-stream consumption (measured -> correct; missing -> UNDETERMINED, not
faked). Two obsolete tests updated only after confirming (their own comments)
they existed to enforce the deleted hardcoded-TFF.
- FieldOrder now derives from the bitstream's measured top_field_first
(Some(true)→TFF, Some(false)→BFF) instead of hardcoding TFF for all
interlaced content; falls back to TFF when unmeasured. Adds
VideoStream::top_field_first; DVD/BD scan sets None with a precise
TODO(spec) for parser→title plumbing.
- CICP (matrix/transfer/primaries/range) now prefers measured CICP from
the bitstream (VideoStream::measured_cicp) over the coarse ColorSpace
enum, so the container stops assuming a colour space the stream may
contradict. Enum remains the fallback.
- VobSub S_VOBSUB CodecPrivate now emits a `size: WxH` line ahead of the
palette per the .idx format so players place/scale subs correctly.
- SeekHead: when zero cues are written, the CUES Seek entry is Voided
instead of leaving a dangling pointer to the Cues offset (now Tags/EOF).
- AC-3 Channels back-patch offset is captured from the writer instead of
the hardcoded chan_elem_pos+2 (decoupled from the VINT width choice).
- Hoisted inline CICP codes and the dvcC fourcc to named constants citing
ITU-T H.273 / RFC 9559; fixed the stale FieldOrder comment.
- DefaultDuration vs pulldown: precise TODO(spec) left (needs the same
parser→title channel as top_field_first).
Tests: BFF-from-measured-flag, measured-CICP-overrides-enum, VobSub size:
line present/omitted, zero-cue SeekHead Void. precommit (1.86) green.
Delete re-implementations in the demux:// sink and wire to proven helpers;
keep only genuinely-new functionality.
- AnnexB reframing: delete the sink's local length_prefixed_to_annexb (it
break'd on a zero-length NAL, dropping the rest of the access unit) and
call the canonical append_length_prefixed_as_annex_b in mux::hevc, which
skips just the empty NAL.
- HEVC param sets: delete hvcc_param_sets; reuse hvcc_to_annex_b.
- avcC param sets: hoist as the new canonical avcc_to_annex_b in mux::hevc,
next to hvcc_to_annex_b (the symmetry point); the sink calls it.
- PGS .sup: emit a synthetic clear display set (empty PCS + END) at
pts + duration_ns so subtitles time out instead of lingering to EOF.
- TimelineContinuity: move verbatim into the shared mux::timeline module
(with the prev_offset straggler-remap intact) and use it from both the
MKV muxer and the demux sink; delete the sink's drifted TimelineRebase
copy (which lacked the straggler branch).
- VobSub .idx: emit the conventional 'id: <lang2>, index: 0' line mkvmerge
reads to assign the subtitle language; palette reuse unchanged.
- output(): seed DemuxOptions.base from title.playlist when non-empty.
New constants for the PGS clear-segment framing and avcC header cite the
public HDMV PGS (BD-ROM Part 3) and ISO/IEC 14496-15 specs.
Tests: a zero-length NAL mid-frame no longer truncates the AU; a frame with
duration_ns produces a .sup clear segment; existing demux tests stay green.
The DVD seek-index concern is that the mux highway
(PsDemuxer -> PipelinedPesStream codec parse -> frame out -> muxer)
might drop the keyframe flag or per-frame duration that Mpeg2Parser sets
on each Frame, which would stop the muxer's cluster/cue open
(keyframe && track 0) from ever firing and leave a DVD MKV with many
clusters and zero cues.
Add dvd_highway_preserves_video_keyframe_and_duration: drives a real
Mpeg2Parser through PipelinedPesStream via DemuxBatch::Ps batches (one
PTS-stamped MPEG-2 PS video PES per GOP, decode-order I + P/B), reads the
frames back through the highway's read(), and asserts the GOP-opening
I-frames arrive as keyframes, every frame keeps its duration (BlockGroup
path), and video routes to track 0. Guards from_codec_frame's
keyframe/duration propagation across the demux-thread + recycled-buffer
highway, which the prior cue tests (codec parser straight into the muxer)
did not exercise.
Verified against the real current-VFR-build output
(Greenland-feature.mkv, today): 3365 clusters / 3365 CuePoints, all
BlockGroups, cues resolve to clusters and track the video — the highway
preserves the flags and the seek index is complete.
Matthew Jackson