Free-format MPEG-audio (bitrate_index 0) is a legal, decodable mode — the
decoder derives the frame size from the sync spacing. Dropping it was a
false positive on a clean stream, so it now passes the gate.
Three TrueHD state-machine fixes from an adversarial audit:
- A corrupt access unit drops forward to the next major sync, but only
the individually-verified corruption now feeds the whole-track poison
verdict; the resync run is collateral. A couple of transient errors can
no longer poison and discard an otherwise-good multi-hour track. The
shared drop tally gains a verified/collateral split for this.
- The per-AU PTS rate is refined only from a CRC-validated major sync, so
a corrupt major sync whose rate nibble decodes to another rate family
can no longer shift the resumed audio — a drop stays a silence gap.
- The resync clears only on a CRC-validated major sync, never on a runt
too short to hold and validate its header.
Flag a PGS subtitle track FlagForced when it displays subtitles and every
one carries the HDMV forced_on_flag (a dedicated forced/narrative track),
independent of the disc's vendor label metadata. The track header
reserves a FlagForced byte up front and it is promoted at finish() from
the accumulated display-set state. Only ever promotes — a track already
forced from the playlist metadata is never demoted.
A damaged audio access unit is now dropped rather than muxed as a
decoder-choking glitch. Sync is preserved — a drop becomes a silence
gap, never a shift — and every drop is logged. Detection is per-codec,
each mirroring the format's authoritative integrity check:
DTS core-header validity gates
AC-3/E-AC-3 native frame CRC-16 + bitstream-id range
FLAC whole-frame CRC-16 residue
MP2/MP3 header sanity + free-format reject
AAC-ADTS header sanity (raw AAC passes through untouched)
TrueHD/MLP major-sync CRC-16 + AU parity; corrupt AUs drop forward
to the next major sync, since decode state carries
across access units
LPCM and video are excluded by design (no in-frame integrity data;
inter-frame prediction). A shared DropTally handles counting, logging,
and a whole-track fallback for a mostly-undecodable track.
DTS-HD MA access units are a lossy core frame followed by trailing
extension substreams up to the next core sync. On source-damaged discs
(observed on the Bourne UHDs) the bytes where the XLL extension belongs
are neither a core sync nor an extension sync -- pure garbage -- which
desyncs ffmpeg's XLL decoder and cascades into 'Read past end of XLL
band data' / 'DSYNC check failed' across the whole track.
next_core_boundary now distinguishes three boundary states via a new
ext_clean flag on NextCore::Found:
- precise/recognized extension sync -> ext_clean=true (keep full AU)
- garbage at the boundary byte -> ext_clean=false (drop the ext)
When ext_clean is false we emit the DTS core alone (drop the smallest
junk piece, keep the frame and its PTS) and drain past the garbage to
the next core. Recognized-but-unsizeable extensions still ride the
heuristic scan and are kept intact, so lossless tracks are unaffected --
only genuinely corrupt extension bytes are dropped.
Bourne s1.dts: 1606 damaged frames / 691620 (0.232%), 93% isolated
single frames, worst run 3 in a row (~32ms lossy blip).
Parse the DTS core header (SFREQ/AMODE/RATE/LFF/NBLKS/FSIZE) → a ddts box
(sample rate, channel layout mask, core size, computed bitrate, LFE);
whole access units (core + DTS-HD extension substreams) pass through, so
an HD decoder finds the extension. dtsh when an extension sync follows the
core, else dtsc. Fit oracle now carries DTS / DTS-HD MA / DTS-HD HR.
Validated on 300 (real DTS-HD MA 7.1): freemkv's mp4 DTS track is
byte-identical to ffmpeg -c copy under ffprobe (dts / 48000 / 8ch / 7.1)
and decodes clean (exit 0). Channel layout correct.
moov now precedes mdat. At create() reserve a moov-sized hole (a free
box) between ftyp and mdat: reserve = round_up_4MB(16 B/sample ×
est_samples) + 4MB buffer, floored at 8MB. Because the hole precedes
mdat, sample offsets are fixed from the start — no rewrite, no offset
patch. finish() writes moov into the hole and pads the slack with a free
box; if the estimate is blown it falls back to moov-at-end. Streams over
HTTP without a pre-fetch. Reserve-math + box-order tests added.
Read side of mp4://: parse moov/trak/stbl (stsd codecs+hvcC/avcC/channel
info, stsz/stco/co64+stsc → per-sample offsets, stts+ctts → decode/
composition timing, stss → sync), rebuild a DiscTitle, and emit samples
as PesFrames in global decode order. Video NALs are length-prefixed in
MP4 — the exact framing the MKV muxer wants — so no reframing. Wired into
input() so mp4:// flows to every sink (mkv://, audio://, json://, …).
In-memory write→read round-trip test proves symmetry (streams, hvcC,
sample sizes survive). Progressive MP4 only; fragmented (moof) is future.
Make Mp4Sink generic over a seekable writer; the CLI output() arm wraps
the file in WritebackFile (as mkv:// does) so a UHD-scale mux to slow /
NFS staging avoids the dirty-page burst. The mdat backpatch is an
ordinary seek WritebackFile already handles (seek_then_patch_roundtrip).
Tests switched to in-memory Cursor writers.
Generalize the sink to N tracks: one video + every MP4-mappable audio
track. Audio sample entries built from the first frame's bitstream —
AC-3 (ac-3/dac3) and E-AC-3 (ec-3/dec3), parsing the (E-)AC-3 BSI for
fscod/bsid/bsmod/acmod/lfeon and the data rate. Per-sample audio
durations from PTS deltas (no reorder → no ctts, all sync). Fit oracle
(mp4_fit_report, exported): video HEVC/H264, audio AC-3/E-AC-3;
TrueHD/DTS/LPCM and bitmap subs are excluded with a typed reason so the
CLI can report exclusions — never a silent drop.
Verified vs ffmpeg -c copy on real discs: AVC+AC3 and HEVC(HDR10)+AC3
both frame-exact on every track (video 2650/4270, audio 5567/3454),
duration and colour identical, clean decode. On a pathological 4-clip
title ffmpeg's OWN output emits the same DTS-monotonicity warnings (more
of them) — source-inherent, not a muxer defect.
New mux/mp4: writes ftyp+mdat+moov (moov-at-end), streaming samples into
a 64-bit mdat and building the sample tables in memory, patched at
finish(). Video track (HEVC/AVC): passthrough length-prefixed NALs
(already MP4 framing), full stts/stsz/stsc/co64/stss and signed ctts,
CFR-derived decode timeline (pipeline carries presentation PTS only), and
a colr box for HDR10 colour signalling. hvc1/avc1 sample entry from the
hvcC/avcC codec_private. Fail-loud on codecs with no MP4 mapping.
Verified against ffmpeg -c copy on real discs: AVC-SDR (300) and
HEVC-HDR10 UHD (Dune) both frame-exact (8159 / 8160 frames), colour-exact
(bt2020/smpte2084/bt2020nc), and clean-decoding. Audio + fit oracle land
in M2.
Completes the per-track-class trio with audio:// / sub://. video:// is
the demux path with a TrackKind::Video kind filter — each video track to
its own native elementary stream (.hevc/.h264/.vc1/.m2v/.obu), no audio
or subtitles. Reuses the existing kind_filter machinery and extension
map; scheme/parse/output/write-only-guard arms added symmetrically.
The json:// sink previously emitted only codec + language + pid per
stream. Fill it out to the full DiscTitle: video carries resolution
(+ pixel dims, interlaced), frame rate (+ fraction), HDR, colour space,
display aspect, and measured CICP; audio carries channels (+ count),
sample rate (+ Hz), and editorial purpose; subtitles carry the
qualifier. Add the clip list and chapter names. This is everything the
scan resolved, so json:// is a lossless machine-readable view of a
title rather than a summary.
Two write-only file sinks that ignore the PES stream and emit the title
metadata at construction (fvi-style, wired through output()):
- chapters:// — chapter markers in the format the extension picks: .xml
(Matroska, default), .txt/.ogm (OGM simple), .vtt (WebVTT). Reuses the
existing chapters_xml/chapters_ogm writers; adds a WebVTT writer.
- json:// — one title's model (playlist, duration, size, format, streams,
chapters) as pretty JSON via serde_json.
New mux/meta_sink.rs; StreamUrl gains Chapters/Json; codec_label +
chapters_xml/ogm promoted to pub(crate) for reuse.
Tests: chapters_format_selected_by_extension, title_json_carries_streams_and_chapters.
Two extraction sinks built on the existing demux machinery: audio:// keeps
only audio tracks (native containers .thd/.dts/.ac3/.eac3/.pcm...), sub://
only subtitle tracks (PGS .sup, VobSub .idx+.sub, text .srt) — one file per
track, no video, no chapters sidecar. DemuxOptions gains a kind_filter; the
DemuxSink writers (already complete) are reused verbatim. StreamUrl gains
Audio/Sub variants; output() builds the filtered DemuxSink.
Test: kind_filter_keeps_only_the_selected_class.
The read_plan fix landed on the file-backed highway (build_iso_pipeline)
but the inline DiscStream (live single-pass) still passed the alternate
device-group half to the demux — a known FMTS bug on rip_mode="single".
DiscStream::with_key_map installs the proactive map and rewrites the
extent walk to the read plan, so the live single-pass path reads ONLY
our-phase units, exactly like the highway. DecryptingSectorSource gains
set_key_map for the already-constructed decorator. Non-forensic maps
return the extents unchanged, so every non-FMTS disc is byte-identical.
Test: with_key_map_reads_only_our_phase_units — a forensic Even segment
drops exactly its alternate units from the extent walk. Precommit green
on 1.86.
An FMTS forensic segment interleaves our device group's variant with a
foreign group's at the aligned-unit level. The mux was decrypting only
our phase but leaving the alternate (foreign) units in the buffer as
ciphertext, trusting the demux to 'drop untouched ciphertext cleanly'.
It doesn't: random 0x47 bytes at the 192-byte stride hit tracked PIDs,
mis-parse, and trip the demux's concealed-gap keyframe-resync — which
drops GOOD frames of ours around every segment (349 resyncs / ~6391
packets on Stand by Me, visible as playback flaws).
The map already knows which unit each LBA is and, for a forensic
segment, which phase is ours. AacsKeyMap::read_plan turns that into the
title's read plan: every default/CPS unit, plus inside a segment ONLY
our-phase units. The alternate units are never fetched, decrypted, or
handed to the demux — the demux sees one gapless our-variant stream.
- read_plan is general (single-CPS, multi-CPS, FMTS): a map with no
Even/Odd range returns the extents unchanged, so DVD/CSS, single-CPS
UHD and multi-CPS Blu-ray read byte-for-byte as before.
- Wired into build_iso_pipeline (the file-backed highway that muxes
resumed ISOs). Producer re-anchors unit_base per extent, so per-unit
segment reads stay unit-aligned and decrypt correctly.
- Extent gains PartialEq/Eq for the read_plan tests.
Tests: read_plan non-forensic unchanged; forensic omits exactly the
alternate units, kept units match the decrypt gate unit-for-unit. All
2314 lib tests pass; precommit green on 1.86.
Pending: end-to-end ISO re-mux validation (concealed gaps 349 -> ~0).
Verified zero callers across all consumer crates AND libfreemkv integration tests:
MUX_APP, Disc::{aacs_disc_hash,encrypted_content_ranges,inject_unit_keys},
locate_ranges, mapfile::{MapEntry,entries}, diag::dump_mkv_track,
DiscStream::{errors,lost_bytes} (read via accessors). Removed the unused mux
DemuxSink/FviSink crate-root re-exports (constructed internally by output() via
the direct module path). Staged demux option variants marked allow(dead_code).
Removed the test-only unchecked wrapper and renamed decrypt_keys_for_title_checked
-> decrypt_keys_for_title (one method returning (keys, title_is_clear)). Production
already used the checked form; the _checked suffix had no counterpart. Callers +
docs updated; two tests take .0.
crack_key_halt had no caller except the crack_key wrapper — a needless middle
layer. crack_key now calls crack_key_scan directly; crack_key (Option) and
crack_key_outcome (full CrackOutcome + halt) remain as the two real entry points.
create_with_source (full SourceInfo) was only ever called by the create wrapper
with a partial SourceInfo — the extra capability was unused (YAGNI). Inlined the
constructor body into create and deleted the _with_source variant; callers
unchanged.
One method per action (CLAUDE.md), not a foo_with_X pair — create_at was the real
constructor and create was a None wrapper. Callers pass Option<&Path> directly.
Adds parse_hex_u16/u32/u8 and exposes strip_hex_prefix so callers stop hand-rolling
from_str_radix(trim_start_matches("0x")) — a case-sensitive strip that this module
exists to prevent. disc::aacs_disc_hash now uses strip_hex_prefix.
AacsState (public via Disc.aacs) and Key (the decrypt_with key-transport enum)
are crate-root re-exported and carried VUK/unit/read-data keys + volume id on
#[derive(Debug)]; HandshakeResult carried the VID + AACS 2.0 bus key. Manual
Debug impls print shape only, guarded by red->green tests.
These carry raw unit-key / VUK / processing-key bytes on their Debug; manual impls
print shape only (unit_keys_len, redacted markers). Each has a red→green test.
CssState is reachable via the public Disc.css field; #[derive(Debug)] leaked the
raw CSS title key on any {:?} of a Disc. Manual Debug prints crack_span only.
DeviceKey/HostCert/Vid/MediaKey/Vuk/ProcessingKey/UnitKey/DiscEntry carried key
material on #[derive(Debug)] — a stray {:?} would leak device/host-private/media/
volume/unit keys. Manual Debug impls print shape only. redaction_tests asserts no
key bytes appear and a redaction marker is present (fails if a derive returns).
FMTS (AACS 2.1) now decodes per (LBA, phase): Phase enum + AacsKeyMap::
from_ranges_phased, decrypt only the variant's parity half. resolve_fmts_key_map
does a 2-phase index-1 anchor then per-index phase probe, and sizes the forensic
set to whatever the source returns (no hardcoded 32). KeyFetch is now two explicit
operations (unit_keys / fmts_indexes) and KeySource splits get_uk into
get_unit_keys + get_fmts_indexes. BYPASS_FMTS_KEY gate removed (first-class format).
Teed up for 1.4.5. Local WIP baseline.
The keyserver protocol now returns all 32 index keys as an array for a
forensic content sample (and a single-element array for plain content).
resolve_fmts_key_map sends one forensic batch and maps array element i to
segment index i+1, replacing the per-index blind-probe collection loop
that repeatedly hit the key service. Segment/index parsing and the
aligned-unit content classification are reworked to support this:
- rename variant_select -> index_select (per-index, not per-variant)
- content classification moves to is_clean(buf, ContentFormat) so the
unit selector emits only units the key service accepts
- segment.rs: parse IndividualSegment.tbl index tags + SPN ranges,
build contiguous LBA key ranges from the resolved 32-key array
- decrypt/decorator plumbing for the resolved per-index keys
Fail loud (FmtsKeyMissing) when the forensic query returns < 32 keys or
any segment index stays unresolved.
decrypt:
- decrypt_sectors is now a pure decrypt (apply key, leave plaintext, report
unverified bytes); TS-structure is a separate primitive (is_clean_ts/ps) used
only for key selection and read-verify. The mux passes decrypted bytes through
(the demuxer drops non-conforming packets), ending the NULL-TS conceal loop and
the per-unit key-server refetch storm. Key-proof floor replaces the 75%
supermajority.
recovery:
- Removed the post-read decrypt-verify gate (verify.rs) that mis-aligned the
disc-absolute unit grid against clip-anchored AACS units and false-failed good
clips (e.g. Dunkirk's orphan-CPS clip). Bad sectors are marked by physical read
result; decryptability is proven at scan + mux time.
HD DVD (first-class AACS):
- Role-based candidate-list file sourcing so an HD DVD's /ANY!/ files
(MKBROM.AACS, VTKF000.AACS, CONTENT_CERT.AACS) are found with no disc-type
branch. parse_vtkf parses VTKF000.AACS into the same UnitKeyFile as a BD
Unit_Key_RO.inf, so the shared VUK unwrap applies unchanged. set_unit_base
clip-anchoring. Two decrypt-axis assumptions remain UNVERIFIED-HDDVD-DECRYPT
(no encrypted disc to test).
mux:
- MVC (Blu-ray 3D) track signals unified into one MVCDecoderConfigurationRecord;
release-safe track_vint (3-byte VINT) and pid_index (i32) guards.
hardening:
- Container-aware is_clean / encryption detection; bytes_bad_in_title fail-safe
on a corrupt mapfile; CSS crack gated on DiscFormat::Dvd (HD DVD excluded);
non-vacuous CSS tests; patch NOT_READY/HARDWARE/ILLEGAL_REQUEST/ABORTED
sense-path tests.
decrypt_sectors is now a pure decrypt — apply the CPS unit key, leave the
plaintext, report how many bytes did not reach clean TS ("unverified"). It
never restores ciphertext, nulls, or re-fetches. "Did a key produce clean TS?"
is a key-selection / read-verify signal, not the verdict "did we decrypt?": a
correct key can decrypt a bad-encoded region, and broken TS is a muxer concern
(the demuxer drops the packet and resyncs).
Callers own the policy:
- mux (read > decrypt > mux): pass the decrypted bytes to the muxer, whatever
they are; fail loud only on a genuine can't-decrypt (no key / misaligned).
- sweep/patch (reading from a disc): an unverified unit is a bad read — recover
a fresh key and retry, or fail loud so disc-recovery re-reads it.
Removes three duplicated decisions — the decrypt-time ciphertext restore, the
mux NULL-TS conceal loop, and the per-unit key-server refetch — plus the dead
aacs_unit_still_ciphertext predicate. Key-fetch recovery now samples the on-disc
ciphertext explicitly (a pure decrypt leaves the buffer plaintext) and lives
only on the rip/verify path, never the mux.
Fixes the 30-90s/region mux stalls and key-server storm on bad-encoded UHD runs
that 1.4.1 left behind (it relaxed the gate but not the surrounding machinery).
Matthew Jackson