Disc::read_aacs_inputs opened the ISO via FileSectorSource::open and
mapped any failure to Error::AacsNoKeys (E7000), discarding the real
Error::IoError (E5000) and its OS errno. A missing or unreadable ISO
(ENOENT/EPERM) is an I/O fault, not a key-resolution failure; callers
that dispatch on the error code would wrongly tell the user to check
their keys when the ISO simply does not exist.
Propagate the open error unchanged and add a regression test asserting
a nonexistent ISO yields E_IO_ERROR, not E_AACS_NO_KEYS.
detect_max_batch_sectors() is a Linux-sysfs probe with no platform
gate. It derived the device name with rsplit('/'), which never splits a
Windows \.\CdRom0 / \.\D: path, so the whole path became the device
name, no /sys node matched, is_optical fell to false, and the function
returned the 8192-sector block default (16 MiB/request) instead of the
60-sector optical default. That value then took the Some(b) arm in
Disc::copy and bypassed the 510-sector optical clamp that lives only in
the sysfs branch, leaving every Windows rip/verify running ~16x over the
optical cap (coarser bad-sector recovery, 16 MiB UDF reads).
Gate the sysfs probe behind a new sysfs_batch_probe_supported() helper
(Linux-only, requires a '/'-delimited path) and return the optical
default for any path the probe can't handle. Add regression tests for
the \.\ device-path forms.
STORAGE_ADAPTER_DESCRIPTOR.BusType is STORAGE_BUS_TYPE, an int-sized
(4-byte) enum, but the repr(C) struct declared it as u8. The total
size stayed 32 bytes by coincidence of alignment padding, and the two
fields actually read today (MaximumTransferLength at offset 8 and
AlignmentMask at offset 16) sit ahead of BusType, so there was no
runtime impact. But BusMajorVersion and BusMinorVersion landed at
offsets 26 and 28 instead of the SDK's 28 and 30, so any future reader
of those fields would have gotten wrong values.
Widen BusType to u32 so BusMajorVersion/BusMinorVersion fall at the
correct offsets, and add a layout regression test asserting every
field offset and the 32-byte total against the winioctl.h layout.
The doc comment claimed the version was a BE u32 at offset 8 of the
record body (offset 12 from pos), but the code correctly reads pos+8
(body offset 4): a 4-byte record header at pos, the Type field at body
offset 0, then the version at body offset 4. Rewrite the comment to
match the actual read so a maintainer does not 'correct' the offset and
break MKB version parsing. Clarify the matching test comment too.
aligned_unit_keys_validate accepted a unit-key set as soon as ONE
scrambled sample decrypted. On a multi-CPS-unit disc a set covering
CPS unit 0 but not CPS unit 1 therefore passed: decrypt_with committed
it, the sweep proceeded, and CPS-unit-1 sectors passed through as raw
encrypted bytes into the ISO/MKV with no error surfaced anywhere.
Require every scrambled sample to be descrambled by some unit key.
A sample no key covers now fails the gate, so an incomplete set is
rejected (AacsKeyRejected) and the caller falls through to the next
candidate, ultimately surfacing a key error instead of silently
writing ciphertext. Wholly-wrong-key rejection is unchanged.
Add a regression test for the partial-coverage case.
The comment on the bridge-degradation branch in handle_read_error
claimed it matched the NOT_READY 04/3E sense signature, but
is_bridge_degradation() keys solely on a non-standard SCSI status byte
(anything that is not GOOD/CHECK CONDITION/TRANSPORT FAILURE) and
ignores sense_key/ASC/ASCQ. A real 04/3E bad-sector error arrives as
CHECK CONDITION (0x02), so it never took this branch — it falls through
to the generic NOT_READY retry. Rewrite the comment to describe the
status-byte condition the predicate actually detects, and drop the
parallel misleading note in the not_ready_err test helper.
Add a regression test asserting a NOT_READY 04/3E error is not
classified as bridge degradation and routes to the NOT_READY retry
(3 s pause) rather than the bridge cooldown (15 s pause).
SptiTransport::reset() unconditionally slept 2 seconds after sending
IOCTL_STORAGE_RESET_DEVICE, even when the IOCTL failed (e.g.
ERROR_INVALID_FUNCTION on a driver that does not support the reset).
On failure no reset occurred, so there is nothing to settle and the
2-second penalty was pure waste. Gate the settle sleep on the IOCTL
return so it only fires when the drive was actually reset.
keydb::save() overwrote the live keydb.cfg with a bare in-place
std::fs::write. keydb.cfg is the single source of AACS truth and this
path runs unattended (first-boot download + daily-refresh thread, with
a container restart on every release), so a SIGKILL, OOM-kill, power
loss, or ENOSPC mid-write could leave the file truncated with the prior
good copy already gone. A truncated keydb does not error at write time;
it surfaces later as failed key resolution on every AACS rip.
Factor the write into write_atomic(): create the parent dir, write a
unique sibling temp file, fsync, then rename (atomic within a
filesystem). On any write/fsync/rename failure the temp is removed and
the existing keydb is left untouched. Same pattern already used by the
settings and mover write paths. Add regression tests covering in-place
replacement (no stray temp) and prior-copy preservation on failure.
DiscStream skips a whole AACS unit (3 sectors = 6144 bytes) per
read-error event, but only the skip-event count was exposed. Loss
estimates built from errors*2048 therefore undercounted AACS loss ~3x.
Add a lost_bytes field that accumulates the actual zero-filled byte
count at each skip, expose it via a new Stream::lost_bytes() accessor
(default 0; DiscStream and CountingStream override), so consumers can
scale lost-video time by real bytes lost rather than the event count.
Regression tests assert the AACS path records 6144 B/event (and
exceeds the errors*2048 undercount) while the align=1 path records
2048 B/event.
When finish_with_halt's grace period expires it detaches from the
consumer thread and returns an error to the caller, but the leaked
consumer kept running to completion: once its wedged write syscall
returned it would fall through to sink.close(). For the mux writer
close() finalises the MKV (Cues block + segment-header patch), so a
leaked consumer could finalise — and keep writing to — an output file
the caller had already reported as failed, racing a fresh rip for the
same device over the same path.
Add a shared abandonment flag the consumer polls in its drain loop and
again before close(). finish_with_grace sets it before dropping the
JoinHandle, so the moment the wedged syscall returns the consumer skips
any further apply and skips close() entirely, then exits. This does not
interrupt the in-flight syscall (only its return or process exit can),
but it bounds the damage to the write already in flight instead of a
full finalise of an abandoned file.
Regression tests cover both sides: a consumer leaked past the grace
period skips close(), while one that finishes inside the grace window
still calls close() and finalises normally.
SptiTransport::reset() discarded the DeviceIoControl return value, so a
wrong or unsupported reset IOCTL would fail with ERROR_INVALID_FUNCTION
and silently no-op while the unconditional 2s settle sleep made it look
like a reset happened. That is exactly the regression class the doc block
records for the two earlier (incorrect) code values.
Bind the result and warn (with GetLastError) when the reset fails, debug
on success. Lift IOCTL_STORAGE_RESET_DEVICE to module scope and add a
test recomputing it from the CTL_CODE formula so a wrong value can't slip
back in unnoticed.
Mapfile::flush() wrote the new state to a .tmp sibling, sync_all()'d the
temp file, then rename(2)'d it over the final mapfile path — but never
fsynced the parent directory. After the rename the new dirent lives only
in the directory's page cache, so a crash or power loss in the
rename-commit window (the wide window on NFS, the very case the temp
fsync guards) can lose it: resume then reads a stale or absent mapfile
even though the data bytes were durable, silently discarding multi-pass
recovery progress.
Add a best-effort fsync_dir() on the path's parent after the rename,
mirroring the established dirent-durability pattern in autorip's mover.
A directory that can't be opened or synced is logged and ignored rather
than failing the write, since the file bytes are already durable.
Adds a regression test exercising the parent-fsync branch against a real
subdirectory and asserting the helper is a no-op on a missing directory.
Tests, crates.io publish, and the GitHub Release no longer serialize
behind each other: each job only needs the version-check (verify) gate.
Binary consumers git-tag-pin libfreemkv, so they start building the
instant the tag exists rather than waiting on the crates.io publish.
crates.io publish runs as an independent job for external consumers and
uses --no-verify (CI already compiled this commit in the test job).
Drive::read fires hundreds of thousands of times per rip. Logging its
entry at DEBUG floods a diagnostic log and buries the events that
actually matter. Move it to TRACE so a level-3 (debug) bug-report log
stays readable; level-4 (trace) still captures it for deep dives.
The docs still documented the old in-tree firmware unlocker: the MediaTek
MT1959 variant table, the READ BUFFER unlock CDB bytes, the profiles.json
schema (unlock_mode/unlock_buf_id/unlock_cdb), the platform/mt1959 driver
listings, and the 'why unlock is needed' handshake mechanism. None of that
lives in libfreemkv anymore — the core is firmware-clean and ships only the
pluggable Unlocker trait + registry (src/unlock.rs).
Rewrite drive-access, architecture, api-design, disc-to-rip, and the README
to describe only the generic Unlocker seam: the trait, register_unlocker, the
registry routing, and the host-cert fallback when no unlocker matches. Point
readers to the freemkv-unlock repo for concrete unlockers. No source change.
Three fixes for a Windows ASUS Blu-ray drive that failed/spammed errors:
- resolve.rs: accept disk:// as an alias for disc:// (identical behavior;
empty = auto-detect, path = device). Windows users commonly type
disk://i: after the drive-letter convention.
- drive::find_drive: prefer a drive that reports media present. Enumerate
all optical drives, query Drive::drive_status() (GET EVENT STATUS, works
regardless of firmware), and return the first reporting DiscPresent;
fall back to the first enumerated drive when none report a disc so
single-drive / quirky setups don't regress. Selection policy split into
select_drive_with_media() for unit testing.
- READ chunking: add ScsiTransport::max_transfer_bytes() (default 1 MiB).
Windows SPTI overrides it with the adapter MaximumTransferLength queried
via IOCTL_STORAGE_QUERY_PROPERTY / StorageAdapterProperty, clamped to a
64 KiB floor (fallback on query failure). Drive::read now caps each
READ(10) to that limit: small reads take the unchanged single-CDB path,
larger reads loop over read_one() chunks, reporting the failing chunk's
LBA on error. This stops the 16 MiB single read that exceeded the
adapter limit, made DeviceIoControl fail, and spammed transport-failure
warnings with slow tiny-read fallbacks.
Tests added for the disk:// alias, media-preference selection, and READ
chunk decomposition / per-chunk error LBA.
Complete the OEM/AACS cert baseline so host certs are a KeySource output,
never compiled in. With an unlocker present the OEM route is unused
(unlocker_read_volume_id short-circuits); without one, the cert handshake
runs when a keysource supplies a host cert and fails gracefully when none
does.
- KeySource trait gains host_certs() (default empty), reusing the existing
aacs::HostCert type. A source holds certs as its second kind of AACS
material alongside decryption keys.
- ScanOptions gains key_sources so the handshake can collect certs across
the app's keysource layer, unioned with DriveCredentials.
- do_handshake_cert collects certs via collect_host_certs (credentials +
every key source). Zero certs from any source now returns the new
graceful Error::AacsNoHostCert (code 7024, sentinel <no host cert>)
instead of silently skipping; resolution still falls back to the
path-1 disc-hash -> VUK lookup, which drops the error on a hit.
- error.rs: add E_AACS_NO_HOST_CERT / Error::AacsNoHostCert, wired into
code(), Display, and the round-trip + sentinel tests.
HandshakeResult { volume_id, read_data_key } unchanged: the cert path
still yields both the VID and the bus key.
Rename the trait to a generic, drive-neutral capability contract so future
unlockers don't conform to LibreDrive specifics:
- unlock(...) -> unlock_drive(...) (the one required capability)
- read_vid(...) -> read_volume_id(...) (no-op default)
- add set_max_read_speed(...) (no-op default)
The trait doc now states the contract in one place: unlockers are optional
drive-capability providers; the AACS layer is the always-present baseline and
falls back to the full cert handshake when no unlocker matches. Implement only
the capabilities your drive supports.
Registry: route_unlock now calls unlock_drive; unlocker_read_vid renamed to
unlocker_read_volume_id; add unlocker_set_max_read_speed (mirrors route_unlock
resolution, first matching unlocker, no-op if none match). drive::init calls
it on a matched drive in the post-unlock path; a speed-set failure is logged
and does not fail the rip. encrypt.rs handshake updated to the new VID helper.
Tests updated for the renames; added a set_max_read_speed routing test
(match invokes, no-match is a safe no-op).
An Unlocker unlocks drive functionality, not just the disc: unlock() is
one capability, OEM VID retrieval is another. Widen the Unlocker trait
with a default-no-op read_vid(), add an unlocker_read_vid registry helper
that mirrors route_unlock resolution, and consult it in do_handshake_cert
before the cert-based VID read. A matching unlocker that serves a VID via
its OEM path short-circuits the cert handshake — VID is obtained without
the host certificate + HRL (restoring the pre-refactor decoupled OEM VID
path, now living inside the unlocker). Non-matching drives, and unlockers
without an OEM VID path, fall through to cert auth unchanged.
is_unlocked() now reports the honest signal (a registered unlocker matched
this drive) instead of const false.
libfreemkv must stay firmware-clean for crates.io. Move ALL drive-unlock
knowledge — firmware blobs, WRITE_BUFFER/MODE SELECT upload, unlock CDBs,
the MT1959 variant-A/B handshake, the 800 KB profiles.json database, and
the DriveProfile parsing — out into the freemkv-unlock-ld crate.
libfreemkv now keeps only the seam:
- Unlocker trait (name/matches/unlock) + a process-wide ordered registry
(register_unlocker / route_unlock) in src/unlock.rs
- Drive::init() walks the registry; the first unlocker whose matches(id)
is true runs unlock(scsi, id); if none match the drive is left in
stock mode and the host-cert AACS handshake (the OEM route) carries
the disc.
The unlocker issues its own CDBs through the public ScsiTransport::execute,
so libfreemkv knows nothing about how unlocking happens.
Removed:
- profiles.json
- src/platform/mt1959/{mod,variant_a,variant_b}.rs
- src/profile.rs (DriveProfile, ProfilesFile, find_by_drive_id, ...)
- the PlatformDriver trait
Because the Unlocker seam reports only success/failure (no extended-access
marker), VID acquisition is now always via the cert-based handshake; the
per-drive OEM-VID-CDB shortcut and Drive::is_unlocked() (now const false)
are removed/neutralized. Disc-speed calibration moved into the unlocker's
unlock(); Drive::probe_disc() is a no-op.
git grep over src/ is firmware-blob/profiles/WRITE_BUFFER/mt1959-free.
All tests pass on Rust 1.86 (precommit green).
Fixes corrupt MKV seek index on single-clip titles with many
interleaved tracks (Top Gun UHD: 2 video, 11 audio, 32 PGS).
TimelineContinuity previously shared one high_ns frontier + offset_ns
across ALL tracks. A sparse, lagging non-video frame (subtitle/audio)
ratcheted the frontier up; the next normal video frame then sat >3s
below it and was misread as a clip-boundary discontinuity, permanently
bumping offset_ns. On a one-clip title this fired thousands of times
and inflated Cue/cluster timestamps into the billions of ms, destroying
the seek index (ffmpeg then seeked to wrong positions and emitted
spurious 'Could not find ref with POC N' errors).
Now only the VIDEO track drives epoch decisions: video alone advances
the frontier and opens a new epoch on a real backward PTS jump.
Non-video tracks are remapped under the current offset and never touch
the frontier or offset. A lagging non-video tail straggler at a genuine
multi-clip boundary (old-epoch raw PTS under the new offset) is
recognised via the previous offset and remapped to the seam, so it
neither flies forward nor forces a back/forward-dated split cluster.
Genuine multi-clip seamless rebasing is preserved.
Also drop TimestampScale from 1ms to 0.1ms (100_000 ns/tick) so
23.976fps frames and 0.833ms TrueHD AUs stop colliding on a single
tick (the source of the non-monotonic-DTS warnings and the audio
cadence flattening). The finer scale shrinks the i16 block-relative
span to ~3.27s, so: cluster duration is set to 2s nominal (keeps
keyframe-driven clusters within the i16 range for typical GOPs), and
the i16-overflow cluster-split path now emits a Cue for the split
cluster so the seek index has no gaps.
Regression tests: single-clip late-subtitle must not inflate offset_ns;
non-video must not advance the frontier; non-video straggler remapped
to seam at a real boundary; every cluster (incl. i16-split) carries a
Cue. Existing tick/duration assertions updated for the new scale.
The subset-difference slots are independent, so rayon find_map_any scans them
in parallel and cancels on first match. UHD MKB no-match scan ~26s -> ~4.6s on
8 cores. Bit-identical result; 58 aacs tests pass.
calc_pk_from_dk derived all three children (left/pk/right) at every tree
level but used only the one it descended into; the Processing Key only
matters at the final node. Derive just the descended child per level + the
PK once at the end. Bit-for-bit identical; speeds every DK->MK derivation
(disc decryption + unpositioned-DK recovery). 60.8s -> 22.9s on a UHD
worst-case recovery scan.
Add recover_dk_position: boil a position-less device key down against a
disc MKB to its invariant subset-difference position (node/uv/u_mask_shift)
— zero-descent probe + ancestor walk-up, hoisted verify. Consolidate the
SD-walk surface (drop the research-only probe::walk_pk_against_tables;
make derive_media_key_from_pk_walked internal).
Two byte-identical copies of the subset-difference walk lived in keys.rs
and variants.rs. Consolidate the pure helpers (aesg3, calc_v_mask,
calc_pk_from_dk) into keys.rs (pub(super)); variants.rs imports them.
Add derive_media_key_and_pk_from_dk(mkb, dks) -> Option<(mk, pk)>, which
returns the intermediate Processing Key the walk already computes;
derive_media_key_from_dk becomes a thin wrapper. This lets callers bank
the PK on a DK boil instead of re-deriving it via a second, divergent
walk (the classical-vs-variant cvalues order made that miss silently).
216 AACS tests pass (incl. a new (mk,pk) regression); precommit (1.86) green.
read_file rejected the padded ~128 MiB MKB_RO.inf via the 0.31.0
MAX_FILE_BYTES cap, so read_aacs_inputs failed and the online
key-resolve path never contacted the keyserver. Read the MKB's real
record length from its header and read exactly that. Also honor
inline/embedded (AD type 3) files so small AACS .inf files read from
the ICB payload instead of being misparsed as allocation descriptors.
Release 0.31.6.
B-frame video PTS is legitimately non-monotonic in decode/storage order; the
audio-oriented monotonic nudge was clobbering it to prev+1ms, which decoders
flagged as non-monotonic DTS (thousands per title). Apply the nudge to
audio/subtitle only; video keeps its true PES PTS. + regression test.
Test-hardening release, no runtime changes. Adds spec-grounded unit tests
across the silent-corruption surfaces — UDF/MPLS/CLPI/IFO parsing, BD/DVD
title + extent assembly, AACS/CSS key handling, TS/PS demux + codec parsers,
MKV/EBML container output, the mux pipeline, sector prefetch + decrypt
decorator, drive/SCSI sense decoding, label extraction, and core I/O. Each
test is grounded in the format spec or real on-disc behavior and verified to
fail under a targeted source mutation. No behavior changed.
Add a precommit fixture proving Disc::read_aacs_inputs reads a Long-AD,
multi-extent /AACS/Unit_Key_RO.inf in full — the exact input the online
key-request path depends on (no disc/deploy needed). Make
read_aacs_inputs_from_reader pub(crate) for the test.
Reword internal comments/doc examples to generic descriptions instead of
specific media titles.
Spec-grounded unit tests for the silent-corruption surfaces, each verified to
fail under a targeted source mutation (no vacuous tests).
udf (10): Extended-AD 20-byte stride + extent LBA at off+12, type-1 sparse
extents skipped not emitted, zero-length type-0 terminator, continuation-loop
bound (anti-hang), UTF-16BE and 8-bit name decoding, FID L_IU offset, parent
(..) FID skip, d-string length-byte cap. Locks the spec branches a future
allocation-descriptor refactor must not silently break.
recovery (9): Pass-N damage-skip range bounds (forward/reverse cursor stays in
range), one-quarter-of-remaining skip cap, below-threshold no-op, work-done
accounting, and bridge-degradation retry-to-budget fall-through.
read_icb_extents hardcoded an 8-byte Short-AD stride for every file. Large
BD-ROM .m2ts streams use 16-byte Long ADs; striding them as Short ADs reads
descriptor #0 correctly (length+lba align) but lands #1 in the middle of the
first Long AD (its zero impl_use bytes). The AD-list terminator (data_len==0
=> break) then fired on that zero and stopped after the first extent, so every
multi-extent title truncated at ~1 GiB. The same reader backs read_file, so
disc AACS-input files (/AACS/*.inf) and the m2ts mux extents were both
affected.
Read the ICB Tag flags (AD type) and stride 8/16/20 bytes for Short/Long/
Extended ADs accordingly; Extended ADs carry the lba at off+12.
aacs: extract trim_mkb and restore its guard so an MKB whose content length
the parser cannot determine (mkb_content_len == 0) is returned intact instead
of truncated to empty.
Regression tests: Long-AD read_icb_extents returns all extents; Long-AD
read_file returns full content; trim_mkb never zeroes an unrecognised MKB.
Library-wide review-and-fix pass: tightened AACS keydb/handshake/variant
handling and trailing-partial-unit policy, corrected MPLS mark offset and
added UDF allocation bounds, hardened the mux/codec framing and M2TS paths,
guarded SCSI READ CAPACITY short transfers and unified error mapping, added
overflow guards on untrusted disc input, and made prefetch shutdown
deterministic. Release profile now builds with thin LTO + single codegen unit.
Subtitle/DVD output-corruption + stream-mapping coverage fixes.
1. DVD subtitle/audio track-mapping collision (CRITICAL). The PS path
routed 0xBD private-stream packets to a track via (sub_id & 0x1F)+1,
so VobSub subtitle sub-id 0x20+j aliased audio track j+1: subtitle
PES was fed to the AC-3 parser and the real subtitle track got
nothing. Route by the canonical DVD PID instead via a new
PsPacket::dvd_pid() that mirrors scan_dvd_titles' PID assignment
(video 0xE0, audio 0xBD00+i, subtitle 0x20+j), then look up the
track in pid_to_track. Fixed identically at all three sites
(pipelined_stream consume_ps, disc.rs live feed, disc.rs EOF flush).
Unmappable/unmapped packets now WARN instead of silently dropping.
2. PGS flush() missing. PgsParser inherited the no-op default flush, so
the last subtitle of every PGS track (emitted only when a following
PCS arrives) was dropped at EOF. Implemented flush() to drain the
pending display set (duration_ns: None for the trailing block).
3. DVD VobSub multi-PES SPU not reassembled. A subpicture unit larger
than one PES spans multiple PES (only the head carries a PTS).
DvdSubParser is now stateful: it buffers per sub-stream until the
leading 2-byte SPU_size is satisfied, inherits the head PTS, and
emits one Frame. flush() drains a truncated trailing SPU at EOF.
4. One-table hygiene. scan_streams had a duplicate stream_type->Codec
table that had drifted from Codec::from_coding_type (missing 0x80
LPCM, 0x85 mapped to DTS-HD MA vs HR, etc.). scan_streams now uses
from_coding_type plus a new Codec::kind()/CodecKind category split,
so the two mappings can never diverge. Silent drops in
scan_streams and bluray STN parsing now WARN with PID + type.
Tests: dvd_pid mapping + subtitle/audio collision regression, PGS
final-subtitle flush, VobSub multi-PES reassembly + EOF flush,
scan_streams 0x80 LPCM via from_coding_type.
H.264 stored SPS/PPS in single Option slots and always stripped them
from frame data, so a mid-title parameter-set redefinition (same id,
different body) was lost — those frames decoded against the stale avcC
copy the player re-applies at each keyframe. Same defect class as the
HEVC PPS-redefinition bug. Mirror that fix: emit a changed SPS/PPS
in-band at every occurrence; strip only the first-seen / identical ones.
Also guard avcC's 16-bit NAL length fields: a param set > 65535 bytes
truncated the length while appending all bytes. Return None instead.
codec_private() hardcoded 8-bit 4:2:0 in the hvcC fixed header, wrong for
10-bit Main 10 UHD (essentially all UHD). Parse chroma_format_idc and
bit_depth_luma/chroma_minus8 from the SPS RBSP (with emulation-prevention
removal and sub-layer profile_tier_level handling) and emit the real
values; fall back to 8-bit 4:2:0 only if the SPS can't be parsed.
Also guard the 16-bit NAL length fields: a param set larger than 65535
bytes would truncate the length while appending all bytes, mis-framing
the record. Return None instead of emitting a corrupt hvcC.
A continuation PES (one that merely extends a TrueHD AU spanning PES
packets) carries its own later PTS, which the parser was adopting
mid-assembly, snapping the AU's timestamp and breaking the monotonic
per-AU cadence (A/V drift). Capture the PTS base only when the
reassembly buffer is empty, i.e. when a PES actually begins a new AU.
When two access units flushed in one parse() call (a core arriving in an
earlier PES than the PES that closes the unit, the standard DTS-HD MA
core+extension-as-separate-PES layout), the second AU inherited the
latest PES's PTS instead of its own core's.
Track per-PES PTS markers keyed by buffer offset and stamp each emitted
AU with the PTS of the PES covering its first byte, rebasing markers on
every front drain. Each AU now keeps its own core's timestamp.
Extends two_cores_back_to_back to assert PTS; adds cross-PES coverage.
Two defects:
- Ac3Parser inherited the no-op default flush(), so a complete final
frame still buffered at end-of-stream was dropped (~32 ms of audio
lost). Add a flush() that drains a complete buffered frame, mirroring
dts.rs.
- Every frame in one parse() call was stamped with the single PES PTS,
collapsing their timecodes and drifting A/V. Compute a base PTS once
per call, then advance per frame by the frame's own duration (AC-3 =
1536 samples; E-AC-3 from numblkscod), converting samples->ns at the
stream sample rate (fscod). Each Frame now carries duration_ns.
MetaStream::Video dropped color_space on from_title/to_title, hardcoding
BT.709 on the way back. HDR titles (BT.2020) lost their color metadata.
Add a color_space field, populate it in from_title, and use it in
to_title. For pre-0.30.7 metadata that has no color_space, derive it from
the preserved hdr field (all HDR formats are BT.2020, SDR is BT.709).
Adds ColorSpace::id() + FromStr for serialization round-trip.
Fight Club redefines PPS id 0 mid-title; the parser froze the first PPS into
codecPrivate and stripped the rest, so the redefined segment decoded against
the wrong PPS (CABAC/cu_qp_delta desync, intact framing). Now any VPS/SPS/PPS
whose body differs from the codecPrivate copy is emitted in-band at every
occurrence, overriding the hvcC copy a player re-applies per keyframe. Proven:
Fight Club re-mux decode errors 320+ -> 0 across all corrupt regions.
Also adds aacs::unit_key_validates (1-block early-reject UK validation) and
ts_sync_count/ts_packet_total helpers.
Fixes the non-monotonic DTS ffmpeg/players reject (observed on Fight Club audio
streams 16/18: "14061 >= 14060"). Some audio PES PTS truncate to the same
millisecond as the prior frame, or tick back 1ms from rounding. The MKV writer
now tracks the last block timecode per track and nudges a non-increasing one to
prev+1ms (sub-frame, inaudible, A/V sync unaffected at ms granularity), recorded
only for frames actually written. New helper monotonic_ts() + unit test.
Independent of the separate HEVC slice-payload corruption investigation.
Add `DiscInputs.volume_label: Option<String>` — the disc's human title (UDF/ISO
volume identifier, else BDMV <di:name>), populated by Disc::inputs() from the
scan. Identity only, no secret, not used in any AACS derivation; lets a key
source forward the title so a key service can catalog disc_hash → title.
The video codec parsers (HEVC, H.264, VC-1, MPEG-2) used
pes.dts.or(pes.pts) as each frame's timestamp. MKV block timecodes
are presentation timestamps; frames are stored in decode order and
the player reorders for display by timecode. Using DTS makes the
timecode monotonic in storage order, presenting B-frames in decode
order — visible motion judder / wrong frames on playback, and
PTS-based seeking lands on the wrong frame.
The compressed video was always byte-correct (verified by NAL-level
diff against a known-good demux); this was purely a timestamp defect
affecting every B-frame title. Fix: prefer PTS (pes.pts.or(pes.dts)).
Verified on a real UHD iso->mkv: emitted PTS now reorders for
B-frames identically to a reference muxer.
Update the two tests that asserted the old DTS-preferred behavior and
add an HEVC regression test pinning PTS as the block timecode.
Audit-driven fixes (rounds 1–3):
- hevc: correct hvcC profile/level SPS offsets (HEVC has a 2-byte NAL header)
- mkv: map all DTS variants to the registered A_DTS codec id; force a new
cluster before the i16 cluster-relative timestamp can overflow
- ebml/mkvstream: bound untrusted EBML sizes (no multi-GB allocs); reject
uint>8 (was an OOB panic) and non-{0,4,8} float widths (were a desync)
- ts: skip PES-header bytes that span a TS packet boundary; add the PMT
section_len/prog_info_len bounds the PAT parser already had
- ac3: preserve a 0x0B77 syncword split across a PES boundary; cap buffer
- dts: validate each next-core boundary by decoded core size (a 0x7FFE8001
pattern inside XLL payload no longer false-splits/drops the lossless
extension); reject sub-minimum core frames; fix forced-emit PTS base
- lpcm: DVD program-stream PCM no longer double-strips the BD LPCM header
- vc1/mpeg2: do not emit a parameter-set-only PES as a standalone frame
- pgs/truehd: cap the pending reassembly buffer (parity with ac3/dts)
- aacs: ts_syncs_intact uses the exact packet count
- prefetched: capacity-guard the recycled-buffer set_len
- Cargo.toml: exclude project docs from the published crate
Convergence: a third independent audit pass found no remaining material
(CRITICAL/HIGH/MEDIUM) issues. Full precommit (fmt + clippy -D + tests,
Rust 1.86) green.
decrypt_with now takes the disc's encrypted content samples and, after
deriving the candidate unit keys, confirms at least one de-scrambles a real
aligned unit before committing them. A wrong key (a keydb VK that doesn't
match the disc, a stale UK) is rejected with AacsKeyRejected instead of
silently applying garbage unit keys. Conservative by design: with no samples
(resume / mapfile cache) it accepts as before, leaving those paths unchanged.
The KeySource trait becomes a stateful provider — next_key hands one candidate
at a time (the source owns the order) and reports exhaustion, replacing the
all-at-once resolve; errored() distinguishes a failed source from a clean
no-key.
DTS-HD MA/HRA access units on Blu-ray are a DTS core frame (sync
0x7FFE8001) followed by one or more DTS extension substreams (sync
0x64582025) carrying the lossless audio. Ground-truthing the Dunkirk
ISO showed the m2ts demuxer hands these out as SEPARATE PES packets on
the same PID: one core PES (exactly core-sized, nothing trailing), then
the extension substreams in following PES packets with their own later
PTS.
The old DtsParser emitted one frame per PES the moment a core frame was
complete, and dropped any PES with no core sync. So every core became a
core-only (lossy) frame and the extension PES packets were discarded as
junk -- silently downgrading the track to lossy DTS core (1557 kb/s CBR,
16-bit) instead of DTS-HD MA (VBR, 24-bit lossless).
Rewrite the parser to assemble across PES boundaries: an access unit
runs from its core sync up to (but not including) the NEXT core sync, so
the core plus every following extension substream stays together. Add a
CodecParser::flush() (default empty) called at end-of-stream by both the
pipelined and inline DiscStream mux paths to drain the final buffered
unit. A 64 KiB cap guarantees forward progress and never stalls if a
boundary can't be found.
Validated on the rip1 testbed: Dunkirk eng+ger and Fight Club eng main
audio now ffprobe as profile=DTS-HD MA (Fight Club eng at 24-bit), with
VBR packet sizes (~2716-2788 B) well above the old fixed 2012 B lossy
core. Genuinely-lossy DTS dub tracks are left untouched.
When an AACS-encrypted ISO is muxed with decryption requested (not --raw)
but key resolution yielded no usable key, input() proceeded to mux the
still-encrypted stream — emitting ~100 MB of garbage (no TS syncs, demuxer
emits nothing) and sometimes spinning for tens of minutes.
Add a cheap result-check in resolve::input()'s Iso branch via the pure
predicate aacs_key_missing(raw, has_aacs, keys): when decryption is
requested AND the disc carries AACS state AND decrypt_keys() is None,
return new Error::NoDiscKey { disc_hash } (E7022) before muxing. The
40-hex disc hash is sourced from AacsState::disc_hash. --raw and
non-AACS (unencrypted / CSS) discs are unaffected. Unit-tested.
correct_truehd_channels() probes the first DECRYPTED access units of the
chosen title (TrueHD PIDs, bounded 8 MiB read of the first extent) and
sets AudioStream.channels from the MLP major sync — fixing the MPLS
audio_format understatement (5.1 declared on a 7.1/Atmos TrueHD track).
Regenerates the basic codec label for the corrected count; richer
editorial labels are left untouched. Wired in resolve.rs input() for
iso:// after decrypt_with (the m2ts is only decryptable post-key), using
a fresh reader so the mux reader is undisturbed. generate_audio_label
made pub(crate).
Adds truehd_channels(format_info) + truehd_channels_from_stream(): the
real presentation channel count (8ch/7.1 when present, else 6ch/5.1)
from the TrueHD major sync, which the MPLS audio_format base field
understates (declares 5.1 even on a 7.1/Atmos track — the Dune case).
Per the MLP spec presentation channel-assignment masks. 3 unit tests.
This is the decoder; wiring it to correct AudioStream.channels requires
a mux-time probe of the DECRYPTED audio payload (the channel truth isn't
readable at scan — the m2ts is AACS-encrypted until the key is applied
at mux), to be added + validated on a real rip.
With the EL now carried (mpls PID fix), the MKV writer tags the Dolby
Vision layer so players/mediainfo recognise it: a BlockAdditionMapping
(0x41E4) with BlockAddIDType 'dvcC' and a DOVIDecoderConfigurationRecord
(profile 7, bl+el+rpu present) on the DV-layer video track. Disc
Profile 7 dual-layer is preserved losslessly as a second video track
(archival-correct, MakeMKV-equivalent); M2TS carries the EL as a native
second video PID with no extra signaling needed.
Adds dolby_vision_config() + a profile-7 dvcC unit test. End-to-end DV
recognition (mediainfo / playback) to be confirmed on a full rip.
parse_stream_entry only read the PID for stream-entry type 1 (a stream
in the PlayItem's own clip), returning PID 0 for every other type. The
Dolby Vision enhancement layer uses type 4 (verified on Wicked + Dune:
se_len=9 type=0x04 [00 10 15] -> PID 0x1015), so the EL fell through to
PID 0x0000 and was silently dropped by the demux/mux — Dolby Vision lost.
Now the PID offset is keyed off the entry type per the BD stream_entry()
layout: type 1 -> +2, type 2 -> +4, type 3/4 -> +3. The DV EL now
resolves to its real PID (0x1015) so it is demuxed and carried as a
stream through the PES layer to every writer (M2TS carries dual-PID DV
natively; the MKV writer's DV signaling is the format-specific piece).
The DTS parser emitted a core-only frame and discarded the trailing
DTS-HD extension substream whenever the extension straddled a PES
boundary (it advanced past only the core, then re-synced on the core
syncword, skipping the leftover extension bytes). That silently
downgrades DTS-HD MA / HRA to lossy DTS core.
Now: when an extension sync is visible after the core — full, or a
partial prefix at the buffer edge — wait for the full extension instead
of splitting it off; only 'nothing after the core' (final unit / EOF,
no parser flush) is taken as a genuine lossy core-only unit. Adds a
boundary-split regression test asserting core+extension is preserved.
Two validated audio-correctness fixes (proven on real discs Wicked/
Paddington/Dune/Fight Club via head-captures):
labels: apply_labels now derives the codec/channel descriptor from the
stream's OWN codec/channels unless the parser's codec_hint is BOTH
consistent with it AND richer (e.g. "Dolby Atmos" on a TrueHD stream).
A mis-bound hint ("AC-3 2.0" on a TrueHD track) is rejected and the
stream's own codec used — killing the cross-labeled shuffle (Wicked) and
the compat-core mislabel (Paddington), while keeping rich hints and
normalizing plain ones to uniform marketing names. (codec_hint_consistent
+ codec_hint_adds_detail, 5 tests.)
disc: canonical_title_order gains an audio-richness tiebreak
(lossless > channels > track-count) for titles that tie on
duration+clips — so a movie authored as a full-audio playlist plus a
stereo-only twin (Fight Club 00800 vs 00004) picks the full-audio one
instead of falling to array order.
- mux input(): propagate a failed decrypt_with rather than muxing an
undecryptable stream silently (review M6).
- Scrub synthetic but key-SHAPED hex (HOST_PRIV_KEY/DEVICE_KEY/disc-entry) from
the keydb parser tests to obvious zero/repeated-byte placeholders. No real or
real-looking key material in code.
The library no longer loads a keydb anywhere. The scan path always captures the
disc's AACS inputs (MKB, VID, Unit_Key_RO.inf) and resolves NO key; a caller
resolves a Key from a key source and applies it via Disc::decrypt_with.
- ScanOptions loses keydb_path / unit_key / disable_keydb (and the path search);
it now carries only optional DriveCredentials (host certs) for the live-drive
AACS handshake. do_handshake_cert uses those instead of loading the keydb.
An unlocked / LibreDrive drive takes the OEM Volume-ID path and needs none.
- The mux input() path takes caller-resolved unit_keys instead of a keydb_path,
and applies them via decrypt_with.
- Deleted the now-dead inline resolve_encryption / resolve_encryption_static.
All 700+ lib tests pass.
The VID-only scan path stashed MKB_RO/RW raw — those files are allocated to a
fixed ~128 MiB and zero-padded, so the MKB on AacsState (consumed by
Disc::inputs() and the device/processing-key decrypt_with derivation) was the
full pad, not the ~few-MB record stream. Trim to mkb_content_len, matching
read_aacs_inputs.
A source that validates server-side against ciphertext (an online key service)
needs encrypted content samples; one that keys on disc identity (keydb, mapfile)
does not. needs_samples() lets the caller skip the extra disc read unless a
configured source actually needs it. Defaults false.
An online key service validates a candidate against real ciphertext, so it
needs a few encrypted content sample units. Add a samples field to DiscInputs;
Disc::inputs() leaves it empty (reading content needs the disc reader, which
scan does not retain) for the application to fill.
The library now describes the key-source abstraction it consumes, while the
concrete sources live in a companion crate. A KeySource looks a disc up from
its DiscInputs (disc hash, volume id, MKB, Unit_Key_RO.inf) and returns a Key;
the library derives down and decrypts. Sources do no derivation.
Disc::inputs() exposes those inputs from a scanned disc.
Key::Media now carries a Vec of candidate media keys rather than one: a media
key is MKB-scoped, so a source hands its whole pool and the library picks the
one that verifies against the disc's MKB. That mirrors Device/Processing and
keeps key selection (a derivation step) inside the library.
Disc::decrypt_with now takes Device / Processing / Media / Volume keys in
addition to Unit. A caller hands in whatever level it resolved and the
library derives down the AACS chain to the per-CPS-unit keys, then
decrypts:
Device -> MKB walk -> media key -> VUK -> per-CPS unit keys
Processing -> MKB -> media key -> VUK -> per-CPS unit keys
Media -> Volume ID -> VUK -> per-CPS unit keys
Volume -> Unit_Key_RO.inf, one unit key per CPS unit
Unit -> used directly (terminal)
Derivation stays centralized in the version-dispatched resolver
(1.0 / 2.0 / 2.1), fed by a single-key provider built from the supplied
key — no new crypto. Volume notably does NOT stop at the volume key: it
decrypts every CPS unit's key.
Scan stashes the AACS inputs (Unit_Key_RO.inf and MKB) on AacsState so an
out-of-band decrypt_with can derive without re-reading the disc.
Non-breaking: Key is #[non_exhaustive] and the existing Unit path is
unchanged. New tests cover the Volume -> per-CPS derive-down, the
missing-inputs error, and the no-units rejection.
scan now stashes the raw Unit_Key_RO.inf + MKB bytes on AacsState (via
resolve_vid_only, the disable_keydb path), so an external key-resolver can
derive unit keys from a resolved VUK without re-reading the disc — the
foundation for moving lookup/derivation out of libfreemkv. Additive: the keydb
path is untouched, all existing constructors default the new fields empty.
584 lib tests green. Builds on the KeyOrigin rename + the Key/decrypt_with API.
Disambiguates the key vocabulary: Key (the input handed to decrypt_with),
key sources (the resolver's lookup list), and KeyOrigin (how a key was
resolved). Internal-only rename — no dependents import it.
Add libfreemkv::Key (non_exhaustive; Unit wired) + Disc::decrypt_with(Key) — the
caller resolves a key from any source and hands it in; the library decrypts with
no keydb/network of its own. inject_unit_keys now synthesizes a minimal
ExternalUk AACS state when a scan built none (no keydb at scan time), so a disc
keyed at sweep but scanned without a keydb still decrypts from its mapfile UK
instead of reporting E8005 and deferring the mux forever. Tests cover the
synthesize, replace, empty->None, and decrypt_with paths.
A keyed disc now writes its decrypted AACS unit keys to the mapfile header
(# freemkv-uk: <cps>:<hex>); an unresolved disc writes only the VID. The two are
mutually exclusive (set_unit_keys clears the VID) — unit keys are the final
answer, so deferred-mux / resume decrypts directly with no key lookup, while the
VID alone is the 'still unresolved, retry' marker. CopyOptions/SweepOptions carry
the keys (written when present, else the VID); Disc::inject_unit_keys applies
mapfile-recovered keys to a scanned disc. Round-trip test added.
Rename is_unit_encrypted -> is_aacs_scrambled and decide encryption from the unit's MPEG-TS sync bytes (destroyed by the encrypted body) instead of the TP_extra copy-control (byte 0) or TS scrambling-control (byte 7) flags, which discs do not set reliably. One shared predicate now backs the decrypt gate and out-of-band key validation, so callers agree on what 'encrypted' means. Decryption restores the syncs, so a decrypted unit reads as clear and there is no flag to clear.
is_unit_encrypted read the TP_extra copy-control bits (byte 0), which are a
copy-permission flag, not an encryption flag. On discs whose sampled units are
clear navigation packets (PAT/PMT) those bits can be set while the unit is not
scrambled, so a correct Unit Key was used to 'decrypt' already-plaintext data,
produced garbage, and the key was wrongly treated as failing.
Read the actual flag instead: the TS transport_scrambling_control bits (top two
of TS-header byte 3 = byte 7 of the aligned unit, inside the clear seed). AACS
encrypts whole aligned units, so this one packet's TSC reflects the unit.
decrypt_unit now clears the TSC bits of every packet on the way out so the
result is valid unscrambled TS. Tests updated to the TSC flag.
keydb stores Media Keys per-disc, but an MK is MKB-scoped (shared across a
pressing/MKB-family). A disc whose own hash/VID isn't keyed can still resolve
if any stored MK verifies against its MKB. New path 2.5 (between PK and the
VID lookup) collects the distinct MK pool from the providers, km_verifies each
against the disc MKB, and on a UNIQUE pass derives VUK (with the disc VID) then
the UK — matching the online resolver's behavior so local keydb mode resolves
the same discs (e.g. an MK present in keydb under a sibling pressing).
km_verifies is one AES-D + magic check per candidate (cheap). Adds
KeyProvider::media_keys() + a path-2.5 unit test.
- mkb_content_len walks the MKB record stream and returns where it ends,
so callers can drop the trailing zero padding. MKB files are allocated
to a fixed size (~128 MiB) with records at the front; both MKB_RO and
MKB_RW can be padded. read_aacs_inputs* now trim to the real records
(~few MB) instead of shipping ~128 MiB of zeros.
- decrypt_keys() returns None when unit_keys is empty: an AACS state that
carries only a VID (out-of-band resolution, pre-key) is 'encrypted, no
keys', not a usable empty key set.
- read_aacs_inputs / read_aacs_inputs_from_drive now read MKB_RO.inf first.
MKB_RW.inf is a fixed ~128 MiB rewritable region that is mostly zero
padding; reading it shipped 124 MiB of nothing. MKB_RO is the real,
correctly-sized MKB (a few MB). Fall back to RW only if RO is absent.
- disable_keydb no longer drops the Volume ID. A caller resolving Unit Keys
out-of-band needs the VID (on-disc content read during the handshake).
New resolve_vid_only builds a keys-free AacsState carrying just the VID +
version metadata, so the disc reports 'encrypted, no keys' (resolved
out-of-band) instead of discarding the VID.
A caller that resolves Unit Keys out-of-band can now set disable_keydb
so the scan consults no keydb at all — neither an explicit keydb_path
nor the standard search locations. Without it, a keydb that merely sits
in a default location ('~/.config/...') silently shadows the out-of-band
path. unit_key still takes precedence over everything.
The ScanOptions.unit_key path is a generic primitive — a caller-supplied Unit
Key that bypasses keydb lookup. Doc comments + a tracing log named a specific
external source; reworded to neutral 'out-of-band / external key service' so
the library makes no assumptions about where the key came from.
Add an optional 16-byte AACS Volume ID to the Mapfile, persisted as a
'# freemkv-vid: <32-hex>' header comment so it survives to deferred-mux
and resume without touching the ISO payload. ddrescue treats the line
as a comment, so the pos/size/status data parser is unaffected.
- Mapfile gains set_vid/vid accessors; save() emits the comment in the
header block, load() parses it back (malformed/absent -> None, never
errors the load). Load->save round-trip preserves it (patch pass).
- Thread vid through CopyOptions and SweepOptions; sweep persists it
when creating/opening the mapfile.
- Unit test: round-trips the VID, asserts the saved comment, and
confirms data ranges parse identically with and without the comment.
Matthew Jackson