Regression since the previous release, which added an MLP major-sync checksum
gate to drop genuinely-undecodable audio frames. The checksum itself was computed
with mismatched byte order: `crc16_mlp` is the correct crc_2D table (poly 0x2D,
MSB-first) but returns its two bytes in the OPPOSITE order to libavutil's
`av_crc`, and `mlp_major_sync_crc_ok` then folded in the pre-trailer word
little-endian while comparing the trailer big-endian. The net result never
matched a real major sync, so EVERY major sync was judged corrupt. That armed the
drop-forward on the first AU and, since no major sync ever validated to clear it,
collateral-dropped every following AU forever — the entire TrueHD track was
silently dropped. Its AUs then flushed only at mux end, so the track's blocks
landed physically after all the video: a decoder reading video+TrueHD had to
buffer the whole title to reach the first audio block and spiralled into an
unbounded memory runaway ("decoder ran out of memory"). Every TrueHD title
produced after the gate landed was affected; a title from the release before it
is clean. (The header-size parse — a frequent suspect for extended 7.1/Atmos
headers — is NOT the bug; it already matches ffmpeg's `mlp_get_major_sync_size`
byte-for-byte.)
Fix: compute the checksum exactly as ffmpeg's `ff_mlp_checksum16` —
`crc16_mlp(body).swap_bytes() ^ AV_RL16(word) == AV_RL16(trailer)`. Cross-verified
byte-exact against two real discs (a 7.1/Atmos title and a 5.1 title, independent
32-byte headers both validate). With the checksum correct, major syncs validate
and the drop-forward corruption protection works as intended.
Defence in depth: a major-sync checksum that STILL can't be validated (a genuinely
corrupt or as-yet-unparsed header) no longer arms the drop-forward until we hold a
validated baseline (`num_substreams` from a prior clean major sync) — so a single
bad header can never again silently drop an entire track.
Tests: the `finalize_major_sync` fixture now builds the checksum the corrected way;
a synthetic checksum-failed head major sync is kept, not dropped; the existing
baseline-then-corrupt drop-forward tests still pass. Verified end to end against a
real disc: the TrueHD track demuxes to a full, cleanly-decodable 48 kHz 8-channel
stream, interleaved with the video, instead of 0 bytes.
The HD DVD Title Key File (VTKF*.AACS) stores 64 title-key entries of 36 bytes
each — 1-byte BIFO + 3 reserved + 16-byte encrypted key + 16-byte binding MAC —
per AACS "HD DVD and DVD Pre-recorded Book" Table 3-8, confirmed byte-exact
against real discs (Freedom VTKF090, Dukes VTKF000: every 36-byte slot has
BIFO=0x80, a clean key, and a 0xFF binding MAC).
The parser used a 32-byte stride (a 12-byte pad in place of the 16-byte binding
MAC) with flag-based termination. That aligns entry #1 (key at offset 132, where
both strides agree) but drifts +4 bytes per entry after it and never terminates
(the previous entry's 0xFF MAC reads as a set present-flag), so it recovered a
correct key only for single-CPS-unit discs and garbage for CPS unit >=2. Every
multi-title HD DVD (Freedom, Harry Potter) was affected.
Fix: 36-byte stride, iterate the fixed 64 slots, take slots whose BIFO AV_FLG
(bit 7) is set, key at offset 4, slot index = CPS unit (skip empty slots rather
than terminate so a gap can't renumber later keys), and never read the trailing
16-byte TKF MAC as a key. Tests rebuilt on the real layout, including a full
64-entry file.
Also correct the VTKF-selection TODO in mod.rs: the AACS HD DVD Book gives the
selector explicitly (match the TKF's PLAYLIST_NAME field to the active
playlist), not the "validate against an encrypted unit" placeholder.
Reconciled against the new HD DVD reference (freemkv.org/docs/hddvd/); the spec
source is archived in freemkv-private/spec/.
The HD DVD AACS directory name and title-key filename are chosen by the
authoring house, but the resolver hardcoded a single spelling
(/ANY!/VTKF000.AACS, /ANY!/MKBROM.AACS, /ANY!/CONTENT_CERT.AACS). Real discs
diverge: Freedom (Memory-Tech) names its AACS dir AAC! and ships VTKF090.AACS
+ VTKF100.AACS; Harry Potter carries VTKF000/001/002/099. On such a disc the
hardcoded path finds nothing, so no MKB/title-key/cert is read and decryption
silently can't engage.
Replace the fixed HD DVD path constants with structural discovery:
- find_hddvd_aacs_dir() locates the AACS dir as the root child dir ending in
'!' that contains MKBROM.AACS (so the ..._BAK mirror is skipped; the dozens
of decoy advanced-content '!' dirs are excluded by the MKBROM.AACS guard).
- role_paths(udf, role) builds the ordered candidate list per role: the static
BD/UHD /AACS/ paths first, then the discovered HD DVD files — MKBROM.AACS,
CONTENT_CERT.AACS, and every VTKF*.AACS (sorted), not just VTKF000.
- read_first() is now generic over &str / String so it takes the Vec<String>.
BD/UHD unaffected (no '!' dir → discovery returns None, list is the /AACS/
constants exactly as before). Verified on real Freedom (AAC!/VTKF090+100) and
Dukes (ANY!/VTKF000) ISOs; unit tests cover both shapes.
Open item (TODO(hddvd-encrypted)): when a disc has multiple VTKF variants the
correct one must be chosen by validating its VUK-derived key against a real
encrypted unit rather than first-that-reads. Blocked on obtaining a genuinely
encrypted HD DVD image — all HD DVD ISOs on hand are already-decrypted rips.
A read was only ever judged "good" by the drive's SCSI GOOD status — with no
integrity check on the bytes. On dirty/marginal media a drive can silently
return best-effort ECC data (occasionally mis-corrected) with GOOD status, so a
rip could "pass clean" yet decode with errors, pushing corruption downstream to
the mux instead of catching it at the read.
Enable recovered-error REPORTING at drive-prep: MODE SELECT the Read-Write
Error Recovery page with PER=1 (TB on / DTE off so the data still comes back),
preserving the drive's own retry count. Marginal reads now surface as
CHECK CONDITION / RECOVERED ERROR. Best-effort — a drive that doesn't honor it
keeps its defaults (no regression), and on a clean disc nothing changes.
A recovered error is distrusted: it marks just that ECC block NonTrimmed for a
Pass N re-read (a clean re-read wins; a persistent marginal becomes an honest
concealed gap), and is counted in the pass summary. Crucially it takes a
per-block SkipBlock, NOT the damage-jump path — a single marginal sector must
not trigger a 64 MB skip that would discard good data on a lightly-smudged disc.
Give `info` the same forced-subtitle verdict the muxer derives during a
rip, so the two agree. A shared classifier (mux::codec::pgs::ForcedTracker)
folds a PGS track's display sets — forced iff every one carries the
forced_on_flag — and is used by BOTH the MKV writer and a new scan-time
probe that reads the title's PGS streams (reusing the TS demuxer and PGS
parser). The probe only overrides a track it actually observed content
for, so an undecrypted/unread stream keeps its vendor-derived flag. Gated
behind ScanOptions::probe_forced_subtitles (off for the rip path, which
detects forced while muxing without a second read).
Free-format MPEG-audio (bitrate_index 0) is a legal, decodable mode — the
decoder derives the frame size from the sync spacing. Dropping it was a
false positive on a clean stream, so it now passes the gate.
Three TrueHD state-machine fixes from an adversarial audit:
- A corrupt access unit drops forward to the next major sync, but only
the individually-verified corruption now feeds the whole-track poison
verdict; the resync run is collateral. A couple of transient errors can
no longer poison and discard an otherwise-good multi-hour track. The
shared drop tally gains a verified/collateral split for this.
- The per-AU PTS rate is refined only from a CRC-validated major sync, so
a corrupt major sync whose rate nibble decodes to another rate family
can no longer shift the resumed audio — a drop stays a silence gap.
- The resync clears only on a CRC-validated major sync, never on a runt
too short to hold and validate its header.
Flag a PGS subtitle track FlagForced when it displays subtitles and every
one carries the HDMV forced_on_flag (a dedicated forced/narrative track),
independent of the disc's vendor label metadata. The track header
reserves a FlagForced byte up front and it is promoted at finish() from
the accumulated display-set state. Only ever promotes — a track already
forced from the playlist metadata is never demoted.
A damaged audio access unit is now dropped rather than muxed as a
decoder-choking glitch. Sync is preserved — a drop becomes a silence
gap, never a shift — and every drop is logged. Detection is per-codec,
each mirroring the format's authoritative integrity check:
DTS core-header validity gates
AC-3/E-AC-3 native frame CRC-16 + bitstream-id range
FLAC whole-frame CRC-16 residue
MP2/MP3 header sanity + free-format reject
AAC-ADTS header sanity (raw AAC passes through untouched)
TrueHD/MLP major-sync CRC-16 + AU parity; corrupt AUs drop forward
to the next major sync, since decode state carries
across access units
LPCM and video are excluded by design (no in-frame integrity data;
inter-frame prediction). A shared DropTally handles counting, logging,
and a whole-track fallback for a mostly-undecodable track.
DTS-HD MA access units are a lossy core frame followed by trailing
extension substreams up to the next core sync. On source-damaged discs
(observed on the Bourne UHDs) the bytes where the XLL extension belongs
are neither a core sync nor an extension sync -- pure garbage -- which
desyncs ffmpeg's XLL decoder and cascades into 'Read past end of XLL
band data' / 'DSYNC check failed' across the whole track.
next_core_boundary now distinguishes three boundary states via a new
ext_clean flag on NextCore::Found:
- precise/recognized extension sync -> ext_clean=true (keep full AU)
- garbage at the boundary byte -> ext_clean=false (drop the ext)
When ext_clean is false we emit the DTS core alone (drop the smallest
junk piece, keep the frame and its PTS) and drain past the garbage to
the next core. Recognized-but-unsizeable extensions still ride the
heuristic scan and are kept intact, so lossless tracks are unaffected --
only genuinely corrupt extension bytes are dropped.
Bourne s1.dts: 1606 damaged frames / 691620 (0.232%), 93% isolated
single frames, worst run 3 in a row (~32ms lossy blip).
Parse the DTS core header (SFREQ/AMODE/RATE/LFF/NBLKS/FSIZE) → a ddts box
(sample rate, channel layout mask, core size, computed bitrate, LFE);
whole access units (core + DTS-HD extension substreams) pass through, so
an HD decoder finds the extension. dtsh when an extension sync follows the
core, else dtsc. Fit oracle now carries DTS / DTS-HD MA / DTS-HD HR.
Validated on 300 (real DTS-HD MA 7.1): freemkv's mp4 DTS track is
byte-identical to ffmpeg -c copy under ffprobe (dts / 48000 / 8ch / 7.1)
and decodes clean (exit 0). Channel layout correct.
moov now precedes mdat. At create() reserve a moov-sized hole (a free
box) between ftyp and mdat: reserve = round_up_4MB(16 B/sample ×
est_samples) + 4MB buffer, floored at 8MB. Because the hole precedes
mdat, sample offsets are fixed from the start — no rewrite, no offset
patch. finish() writes moov into the hole and pads the slack with a free
box; if the estimate is blown it falls back to moov-at-end. Streams over
HTTP without a pre-fetch. Reserve-math + box-order tests added.
Read side of mp4://: parse moov/trak/stbl (stsd codecs+hvcC/avcC/channel
info, stsz/stco/co64+stsc → per-sample offsets, stts+ctts → decode/
composition timing, stss → sync), rebuild a DiscTitle, and emit samples
as PesFrames in global decode order. Video NALs are length-prefixed in
MP4 — the exact framing the MKV muxer wants — so no reframing. Wired into
input() so mp4:// flows to every sink (mkv://, audio://, json://, …).
In-memory write→read round-trip test proves symmetry (streams, hvcC,
sample sizes survive). Progressive MP4 only; fragmented (moof) is future.
Make Mp4Sink generic over a seekable writer; the CLI output() arm wraps
the file in WritebackFile (as mkv:// does) so a UHD-scale mux to slow /
NFS staging avoids the dirty-page burst. The mdat backpatch is an
ordinary seek WritebackFile already handles (seek_then_patch_roundtrip).
Tests switched to in-memory Cursor writers.
Generalize the sink to N tracks: one video + every MP4-mappable audio
track. Audio sample entries built from the first frame's bitstream —
AC-3 (ac-3/dac3) and E-AC-3 (ec-3/dec3), parsing the (E-)AC-3 BSI for
fscod/bsid/bsmod/acmod/lfeon and the data rate. Per-sample audio
durations from PTS deltas (no reorder → no ctts, all sync). Fit oracle
(mp4_fit_report, exported): video HEVC/H264, audio AC-3/E-AC-3;
TrueHD/DTS/LPCM and bitmap subs are excluded with a typed reason so the
CLI can report exclusions — never a silent drop.
Verified vs ffmpeg -c copy on real discs: AVC+AC3 and HEVC(HDR10)+AC3
both frame-exact on every track (video 2650/4270, audio 5567/3454),
duration and colour identical, clean decode. On a pathological 4-clip
title ffmpeg's OWN output emits the same DTS-monotonicity warnings (more
of them) — source-inherent, not a muxer defect.
New mux/mp4: writes ftyp+mdat+moov (moov-at-end), streaming samples into
a 64-bit mdat and building the sample tables in memory, patched at
finish(). Video track (HEVC/AVC): passthrough length-prefixed NALs
(already MP4 framing), full stts/stsz/stsc/co64/stss and signed ctts,
CFR-derived decode timeline (pipeline carries presentation PTS only), and
a colr box for HDR10 colour signalling. hvc1/avc1 sample entry from the
hvcC/avcC codec_private. Fail-loud on codecs with no MP4 mapping.
Verified against ffmpeg -c copy on real discs: AVC-SDR (300) and
HEVC-HDR10 UHD (Dune) both frame-exact (8159 / 8160 frames), colour-exact
(bt2020/smpte2084/bt2020nc), and clean-decoding. Audio + fit oracle land
in M2.
Completes the per-track-class trio with audio:// / sub://. video:// is
the demux path with a TrackKind::Video kind filter — each video track to
its own native elementary stream (.hevc/.h264/.vc1/.m2v/.obu), no audio
or subtitles. Reuses the existing kind_filter machinery and extension
map; scheme/parse/output/write-only-guard arms added symmetrically.
The json:// sink previously emitted only codec + language + pid per
stream. Fill it out to the full DiscTitle: video carries resolution
(+ pixel dims, interlaced), frame rate (+ fraction), HDR, colour space,
display aspect, and measured CICP; audio carries channels (+ count),
sample rate (+ Hz), and editorial purpose; subtitles carry the
qualifier. Add the clip list and chapter names. This is everything the
scan resolved, so json:// is a lossless machine-readable view of a
title rather than a summary.
Two write-only file sinks that ignore the PES stream and emit the title
metadata at construction (fvi-style, wired through output()):
- chapters:// — chapter markers in the format the extension picks: .xml
(Matroska, default), .txt/.ogm (OGM simple), .vtt (WebVTT). Reuses the
existing chapters_xml/chapters_ogm writers; adds a WebVTT writer.
- json:// — one title's model (playlist, duration, size, format, streams,
chapters) as pretty JSON via serde_json.
New mux/meta_sink.rs; StreamUrl gains Chapters/Json; codec_label +
chapters_xml/ogm promoted to pub(crate) for reuse.
Tests: chapters_format_selected_by_extension, title_json_carries_streams_and_chapters.
Two extraction sinks built on the existing demux machinery: audio:// keeps
only audio tracks (native containers .thd/.dts/.ac3/.eac3/.pcm...), sub://
only subtitle tracks (PGS .sup, VobSub .idx+.sub, text .srt) — one file per
track, no video, no chapters sidecar. DemuxOptions gains a kind_filter; the
DemuxSink writers (already complete) are reused verbatim. StreamUrl gains
Audio/Sub variants; output() builds the filtered DemuxSink.
Test: kind_filter_keeps_only_the_selected_class.
The read_plan fix landed on the file-backed highway (build_iso_pipeline)
but the inline DiscStream (live single-pass) still passed the alternate
device-group half to the demux — a known FMTS bug on rip_mode="single".
DiscStream::with_key_map installs the proactive map and rewrites the
extent walk to the read plan, so the live single-pass path reads ONLY
our-phase units, exactly like the highway. DecryptingSectorSource gains
set_key_map for the already-constructed decorator. Non-forensic maps
return the extents unchanged, so every non-FMTS disc is byte-identical.
Test: with_key_map_reads_only_our_phase_units — a forensic Even segment
drops exactly its alternate units from the extent walk. Precommit green
on 1.86.
An FMTS forensic segment interleaves our device group's variant with a
foreign group's at the aligned-unit level. The mux was decrypting only
our phase but leaving the alternate (foreign) units in the buffer as
ciphertext, trusting the demux to 'drop untouched ciphertext cleanly'.
It doesn't: random 0x47 bytes at the 192-byte stride hit tracked PIDs,
mis-parse, and trip the demux's concealed-gap keyframe-resync — which
drops GOOD frames of ours around every segment (349 resyncs / ~6391
packets on Stand by Me, visible as playback flaws).
The map already knows which unit each LBA is and, for a forensic
segment, which phase is ours. AacsKeyMap::read_plan turns that into the
title's read plan: every default/CPS unit, plus inside a segment ONLY
our-phase units. The alternate units are never fetched, decrypted, or
handed to the demux — the demux sees one gapless our-variant stream.
- read_plan is general (single-CPS, multi-CPS, FMTS): a map with no
Even/Odd range returns the extents unchanged, so DVD/CSS, single-CPS
UHD and multi-CPS Blu-ray read byte-for-byte as before.
- Wired into build_iso_pipeline (the file-backed highway that muxes
resumed ISOs). Producer re-anchors unit_base per extent, so per-unit
segment reads stay unit-aligned and decrypt correctly.
- Extent gains PartialEq/Eq for the read_plan tests.
Tests: read_plan non-forensic unchanged; forensic omits exactly the
alternate units, kept units match the decrypt gate unit-for-unit. All
2314 lib tests pass; precommit green on 1.86.
Pending: end-to-end ISO re-mux validation (concealed gaps 349 -> ~0).
Verified zero callers across all consumer crates AND libfreemkv integration tests:
MUX_APP, Disc::{aacs_disc_hash,encrypted_content_ranges,inject_unit_keys},
locate_ranges, mapfile::{MapEntry,entries}, diag::dump_mkv_track,
DiscStream::{errors,lost_bytes} (read via accessors). Removed the unused mux
DemuxSink/FviSink crate-root re-exports (constructed internally by output() via
the direct module path). Staged demux option variants marked allow(dead_code).
Removed the test-only unchecked wrapper and renamed decrypt_keys_for_title_checked
-> decrypt_keys_for_title (one method returning (keys, title_is_clear)). Production
already used the checked form; the _checked suffix had no counterpart. Callers +
docs updated; two tests take .0.
crack_key_halt had no caller except the crack_key wrapper — a needless middle
layer. crack_key now calls crack_key_scan directly; crack_key (Option) and
crack_key_outcome (full CrackOutcome + halt) remain as the two real entry points.
create_with_source (full SourceInfo) was only ever called by the create wrapper
with a partial SourceInfo — the extra capability was unused (YAGNI). Inlined the
constructor body into create and deleted the _with_source variant; callers
unchanged.
One method per action (CLAUDE.md), not a foo_with_X pair — create_at was the real
constructor and create was a None wrapper. Callers pass Option<&Path> directly.
Adds parse_hex_u16/u32/u8 and exposes strip_hex_prefix so callers stop hand-rolling
from_str_radix(trim_start_matches("0x")) — a case-sensitive strip that this module
exists to prevent. disc::aacs_disc_hash now uses strip_hex_prefix.
AacsState (public via Disc.aacs) and Key (the decrypt_with key-transport enum)
are crate-root re-exported and carried VUK/unit/read-data keys + volume id on
#[derive(Debug)]; HandshakeResult carried the VID + AACS 2.0 bus key. Manual
Debug impls print shape only, guarded by red->green tests.
These carry raw unit-key / VUK / processing-key bytes on their Debug; manual impls
print shape only (unit_keys_len, redacted markers). Each has a red→green test.
CssState is reachable via the public Disc.css field; #[derive(Debug)] leaked the
raw CSS title key on any {:?} of a Disc. Manual Debug prints crack_span only.
DeviceKey/HostCert/Vid/MediaKey/Vuk/ProcessingKey/UnitKey/DiscEntry carried key
material on #[derive(Debug)] — a stray {:?} would leak device/host-private/media/
volume/unit keys. Manual Debug impls print shape only. redaction_tests asserts no
key bytes appear and a redaction marker is present (fails if a derive returns).
Matthew Jackson