A read was only ever judged "good" by the drive's SCSI GOOD status — with no
integrity check on the bytes. On dirty/marginal media a drive can silently
return best-effort ECC data (occasionally mis-corrected) with GOOD status, so a
rip could "pass clean" yet decode with errors, pushing corruption downstream to
the mux instead of catching it at the read.
Enable recovered-error REPORTING at drive-prep: MODE SELECT the Read-Write
Error Recovery page with PER=1 (TB on / DTE off so the data still comes back),
preserving the drive's own retry count. Marginal reads now surface as
CHECK CONDITION / RECOVERED ERROR. Best-effort — a drive that doesn't honor it
keeps its defaults (no regression), and on a clean disc nothing changes.
A recovered error is distrusted: it marks just that ECC block NonTrimmed for a
Pass N re-read (a clean re-read wins; a persistent marginal becomes an honest
concealed gap), and is counted in the pass summary. Crucially it takes a
per-block SkipBlock, NOT the damage-jump path — a single marginal sector must
not trigger a 64 MB skip that would discard good data on a lightly-smudged disc.
Give `info` the same forced-subtitle verdict the muxer derives during a
rip, so the two agree. A shared classifier (mux::codec::pgs::ForcedTracker)
folds a PGS track's display sets — forced iff every one carries the
forced_on_flag — and is used by BOTH the MKV writer and a new scan-time
probe that reads the title's PGS streams (reusing the TS demuxer and PGS
parser). The probe only overrides a track it actually observed content
for, so an undecrypted/unread stream keeps its vendor-derived flag. Gated
behind ScanOptions::probe_forced_subtitles (off for the rip path, which
detects forced while muxing without a second read).
Free-format MPEG-audio (bitrate_index 0) is a legal, decodable mode — the
decoder derives the frame size from the sync spacing. Dropping it was a
false positive on a clean stream, so it now passes the gate.
Three TrueHD state-machine fixes from an adversarial audit:
- A corrupt access unit drops forward to the next major sync, but only
the individually-verified corruption now feeds the whole-track poison
verdict; the resync run is collateral. A couple of transient errors can
no longer poison and discard an otherwise-good multi-hour track. The
shared drop tally gains a verified/collateral split for this.
- The per-AU PTS rate is refined only from a CRC-validated major sync, so
a corrupt major sync whose rate nibble decodes to another rate family
can no longer shift the resumed audio — a drop stays a silence gap.
- The resync clears only on a CRC-validated major sync, never on a runt
too short to hold and validate its header.
Flag a PGS subtitle track FlagForced when it displays subtitles and every
one carries the HDMV forced_on_flag (a dedicated forced/narrative track),
independent of the disc's vendor label metadata. The track header
reserves a FlagForced byte up front and it is promoted at finish() from
the accumulated display-set state. Only ever promotes — a track already
forced from the playlist metadata is never demoted.
A damaged audio access unit is now dropped rather than muxed as a
decoder-choking glitch. Sync is preserved — a drop becomes a silence
gap, never a shift — and every drop is logged. Detection is per-codec,
each mirroring the format's authoritative integrity check:
DTS core-header validity gates
AC-3/E-AC-3 native frame CRC-16 + bitstream-id range
FLAC whole-frame CRC-16 residue
MP2/MP3 header sanity + free-format reject
AAC-ADTS header sanity (raw AAC passes through untouched)
TrueHD/MLP major-sync CRC-16 + AU parity; corrupt AUs drop forward
to the next major sync, since decode state carries
across access units
LPCM and video are excluded by design (no in-frame integrity data;
inter-frame prediction). A shared DropTally handles counting, logging,
and a whole-track fallback for a mostly-undecodable track.
DTS-HD MA access units are a lossy core frame followed by trailing
extension substreams up to the next core sync. On source-damaged discs
(observed on the Bourne UHDs) the bytes where the XLL extension belongs
are neither a core sync nor an extension sync -- pure garbage -- which
desyncs ffmpeg's XLL decoder and cascades into 'Read past end of XLL
band data' / 'DSYNC check failed' across the whole track.
next_core_boundary now distinguishes three boundary states via a new
ext_clean flag on NextCore::Found:
- precise/recognized extension sync -> ext_clean=true (keep full AU)
- garbage at the boundary byte -> ext_clean=false (drop the ext)
When ext_clean is false we emit the DTS core alone (drop the smallest
junk piece, keep the frame and its PTS) and drain past the garbage to
the next core. Recognized-but-unsizeable extensions still ride the
heuristic scan and are kept intact, so lossless tracks are unaffected --
only genuinely corrupt extension bytes are dropped.
Bourne s1.dts: 1606 damaged frames / 691620 (0.232%), 93% isolated
single frames, worst run 3 in a row (~32ms lossy blip).
Parse the DTS core header (SFREQ/AMODE/RATE/LFF/NBLKS/FSIZE) → a ddts box
(sample rate, channel layout mask, core size, computed bitrate, LFE);
whole access units (core + DTS-HD extension substreams) pass through, so
an HD decoder finds the extension. dtsh when an extension sync follows the
core, else dtsc. Fit oracle now carries DTS / DTS-HD MA / DTS-HD HR.
Validated on 300 (real DTS-HD MA 7.1): freemkv's mp4 DTS track is
byte-identical to ffmpeg -c copy under ffprobe (dts / 48000 / 8ch / 7.1)
and decodes clean (exit 0). Channel layout correct.
moov now precedes mdat. At create() reserve a moov-sized hole (a free
box) between ftyp and mdat: reserve = round_up_4MB(16 B/sample ×
est_samples) + 4MB buffer, floored at 8MB. Because the hole precedes
mdat, sample offsets are fixed from the start — no rewrite, no offset
patch. finish() writes moov into the hole and pads the slack with a free
box; if the estimate is blown it falls back to moov-at-end. Streams over
HTTP without a pre-fetch. Reserve-math + box-order tests added.
Read side of mp4://: parse moov/trak/stbl (stsd codecs+hvcC/avcC/channel
info, stsz/stco/co64+stsc → per-sample offsets, stts+ctts → decode/
composition timing, stss → sync), rebuild a DiscTitle, and emit samples
as PesFrames in global decode order. Video NALs are length-prefixed in
MP4 — the exact framing the MKV muxer wants — so no reframing. Wired into
input() so mp4:// flows to every sink (mkv://, audio://, json://, …).
In-memory write→read round-trip test proves symmetry (streams, hvcC,
sample sizes survive). Progressive MP4 only; fragmented (moof) is future.
Make Mp4Sink generic over a seekable writer; the CLI output() arm wraps
the file in WritebackFile (as mkv:// does) so a UHD-scale mux to slow /
NFS staging avoids the dirty-page burst. The mdat backpatch is an
ordinary seek WritebackFile already handles (seek_then_patch_roundtrip).
Tests switched to in-memory Cursor writers.
Generalize the sink to N tracks: one video + every MP4-mappable audio
track. Audio sample entries built from the first frame's bitstream —
AC-3 (ac-3/dac3) and E-AC-3 (ec-3/dec3), parsing the (E-)AC-3 BSI for
fscod/bsid/bsmod/acmod/lfeon and the data rate. Per-sample audio
durations from PTS deltas (no reorder → no ctts, all sync). Fit oracle
(mp4_fit_report, exported): video HEVC/H264, audio AC-3/E-AC-3;
TrueHD/DTS/LPCM and bitmap subs are excluded with a typed reason so the
CLI can report exclusions — never a silent drop.
Verified vs ffmpeg -c copy on real discs: AVC+AC3 and HEVC(HDR10)+AC3
both frame-exact on every track (video 2650/4270, audio 5567/3454),
duration and colour identical, clean decode. On a pathological 4-clip
title ffmpeg's OWN output emits the same DTS-monotonicity warnings (more
of them) — source-inherent, not a muxer defect.
New mux/mp4: writes ftyp+mdat+moov (moov-at-end), streaming samples into
a 64-bit mdat and building the sample tables in memory, patched at
finish(). Video track (HEVC/AVC): passthrough length-prefixed NALs
(already MP4 framing), full stts/stsz/stsc/co64/stss and signed ctts,
CFR-derived decode timeline (pipeline carries presentation PTS only), and
a colr box for HDR10 colour signalling. hvc1/avc1 sample entry from the
hvcC/avcC codec_private. Fail-loud on codecs with no MP4 mapping.
Verified against ffmpeg -c copy on real discs: AVC-SDR (300) and
HEVC-HDR10 UHD (Dune) both frame-exact (8159 / 8160 frames), colour-exact
(bt2020/smpte2084/bt2020nc), and clean-decoding. Audio + fit oracle land
in M2.
Completes the per-track-class trio with audio:// / sub://. video:// is
the demux path with a TrackKind::Video kind filter — each video track to
its own native elementary stream (.hevc/.h264/.vc1/.m2v/.obu), no audio
or subtitles. Reuses the existing kind_filter machinery and extension
map; scheme/parse/output/write-only-guard arms added symmetrically.
The json:// sink previously emitted only codec + language + pid per
stream. Fill it out to the full DiscTitle: video carries resolution
(+ pixel dims, interlaced), frame rate (+ fraction), HDR, colour space,
display aspect, and measured CICP; audio carries channels (+ count),
sample rate (+ Hz), and editorial purpose; subtitles carry the
qualifier. Add the clip list and chapter names. This is everything the
scan resolved, so json:// is a lossless machine-readable view of a
title rather than a summary.
Two write-only file sinks that ignore the PES stream and emit the title
metadata at construction (fvi-style, wired through output()):
- chapters:// — chapter markers in the format the extension picks: .xml
(Matroska, default), .txt/.ogm (OGM simple), .vtt (WebVTT). Reuses the
existing chapters_xml/chapters_ogm writers; adds a WebVTT writer.
- json:// — one title's model (playlist, duration, size, format, streams,
chapters) as pretty JSON via serde_json.
New mux/meta_sink.rs; StreamUrl gains Chapters/Json; codec_label +
chapters_xml/ogm promoted to pub(crate) for reuse.
Tests: chapters_format_selected_by_extension, title_json_carries_streams_and_chapters.
Two extraction sinks built on the existing demux machinery: audio:// keeps
only audio tracks (native containers .thd/.dts/.ac3/.eac3/.pcm...), sub://
only subtitle tracks (PGS .sup, VobSub .idx+.sub, text .srt) — one file per
track, no video, no chapters sidecar. DemuxOptions gains a kind_filter; the
DemuxSink writers (already complete) are reused verbatim. StreamUrl gains
Audio/Sub variants; output() builds the filtered DemuxSink.
Test: kind_filter_keeps_only_the_selected_class.
The read_plan fix landed on the file-backed highway (build_iso_pipeline)
but the inline DiscStream (live single-pass) still passed the alternate
device-group half to the demux — a known FMTS bug on rip_mode="single".
DiscStream::with_key_map installs the proactive map and rewrites the
extent walk to the read plan, so the live single-pass path reads ONLY
our-phase units, exactly like the highway. DecryptingSectorSource gains
set_key_map for the already-constructed decorator. Non-forensic maps
return the extents unchanged, so every non-FMTS disc is byte-identical.
Test: with_key_map_reads_only_our_phase_units — a forensic Even segment
drops exactly its alternate units from the extent walk. Precommit green
on 1.86.
An FMTS forensic segment interleaves our device group's variant with a
foreign group's at the aligned-unit level. The mux was decrypting only
our phase but leaving the alternate (foreign) units in the buffer as
ciphertext, trusting the demux to 'drop untouched ciphertext cleanly'.
It doesn't: random 0x47 bytes at the 192-byte stride hit tracked PIDs,
mis-parse, and trip the demux's concealed-gap keyframe-resync — which
drops GOOD frames of ours around every segment (349 resyncs / ~6391
packets on Stand by Me, visible as playback flaws).
The map already knows which unit each LBA is and, for a forensic
segment, which phase is ours. AacsKeyMap::read_plan turns that into the
title's read plan: every default/CPS unit, plus inside a segment ONLY
our-phase units. The alternate units are never fetched, decrypted, or
handed to the demux — the demux sees one gapless our-variant stream.
- read_plan is general (single-CPS, multi-CPS, FMTS): a map with no
Even/Odd range returns the extents unchanged, so DVD/CSS, single-CPS
UHD and multi-CPS Blu-ray read byte-for-byte as before.
- Wired into build_iso_pipeline (the file-backed highway that muxes
resumed ISOs). Producer re-anchors unit_base per extent, so per-unit
segment reads stay unit-aligned and decrypt correctly.
- Extent gains PartialEq/Eq for the read_plan tests.
Tests: read_plan non-forensic unchanged; forensic omits exactly the
alternate units, kept units match the decrypt gate unit-for-unit. All
2314 lib tests pass; precommit green on 1.86.
Pending: end-to-end ISO re-mux validation (concealed gaps 349 -> ~0).
Verified zero callers across all consumer crates AND libfreemkv integration tests:
MUX_APP, Disc::{aacs_disc_hash,encrypted_content_ranges,inject_unit_keys},
locate_ranges, mapfile::{MapEntry,entries}, diag::dump_mkv_track,
DiscStream::{errors,lost_bytes} (read via accessors). Removed the unused mux
DemuxSink/FviSink crate-root re-exports (constructed internally by output() via
the direct module path). Staged demux option variants marked allow(dead_code).
Removed the test-only unchecked wrapper and renamed decrypt_keys_for_title_checked
-> decrypt_keys_for_title (one method returning (keys, title_is_clear)). Production
already used the checked form; the _checked suffix had no counterpart. Callers +
docs updated; two tests take .0.
crack_key_halt had no caller except the crack_key wrapper — a needless middle
layer. crack_key now calls crack_key_scan directly; crack_key (Option) and
crack_key_outcome (full CrackOutcome + halt) remain as the two real entry points.
create_with_source (full SourceInfo) was only ever called by the create wrapper
with a partial SourceInfo — the extra capability was unused (YAGNI). Inlined the
constructor body into create and deleted the _with_source variant; callers
unchanged.
One method per action (CLAUDE.md), not a foo_with_X pair — create_at was the real
constructor and create was a None wrapper. Callers pass Option<&Path> directly.
Adds parse_hex_u16/u32/u8 and exposes strip_hex_prefix so callers stop hand-rolling
from_str_radix(trim_start_matches("0x")) — a case-sensitive strip that this module
exists to prevent. disc::aacs_disc_hash now uses strip_hex_prefix.
AacsState (public via Disc.aacs) and Key (the decrypt_with key-transport enum)
are crate-root re-exported and carried VUK/unit/read-data keys + volume id on
#[derive(Debug)]; HandshakeResult carried the VID + AACS 2.0 bus key. Manual
Debug impls print shape only, guarded by red->green tests.
These carry raw unit-key / VUK / processing-key bytes on their Debug; manual impls
print shape only (unit_keys_len, redacted markers). Each has a red→green test.
CssState is reachable via the public Disc.css field; #[derive(Debug)] leaked the
raw CSS title key on any {:?} of a Disc. Manual Debug prints crack_span only.
DeviceKey/HostCert/Vid/MediaKey/Vuk/ProcessingKey/UnitKey/DiscEntry carried key
material on #[derive(Debug)] — a stray {:?} would leak device/host-private/media/
volume/unit keys. Manual Debug impls print shape only. redaction_tests asserts no
key bytes appear and a redaction marker is present (fails if a derive returns).
FMTS (AACS 2.1) now decodes per (LBA, phase): Phase enum + AacsKeyMap::
from_ranges_phased, decrypt only the variant's parity half. resolve_fmts_key_map
does a 2-phase index-1 anchor then per-index phase probe, and sizes the forensic
set to whatever the source returns (no hardcoded 32). KeyFetch is now two explicit
operations (unit_keys / fmts_indexes) and KeySource splits get_uk into
get_unit_keys + get_fmts_indexes. BYPASS_FMTS_KEY gate removed (first-class format).
Teed up for 1.4.5. Local WIP baseline.
Matthew Jackson