is_unit_encrypted read the TP_extra copy-control bits (byte 0), which are a
copy-permission flag, not an encryption flag. On discs whose sampled units are
clear navigation packets (PAT/PMT) those bits can be set while the unit is not
scrambled, so a correct Unit Key was used to 'decrypt' already-plaintext data,
produced garbage, and the key was wrongly treated as failing.
Read the actual flag instead: the TS transport_scrambling_control bits (top two
of TS-header byte 3 = byte 7 of the aligned unit, inside the clear seed). AACS
encrypts whole aligned units, so this one packet's TSC reflects the unit.
decrypt_unit now clears the TSC bits of every packet on the way out so the
result is valid unscrambled TS. Tests updated to the TSC flag.
keydb stores Media Keys per-disc, but an MK is MKB-scoped (shared across a
pressing/MKB-family). A disc whose own hash/VID isn't keyed can still resolve
if any stored MK verifies against its MKB. New path 2.5 (between PK and the
VID lookup) collects the distinct MK pool from the providers, km_verifies each
against the disc MKB, and on a UNIQUE pass derives VUK (with the disc VID) then
the UK — matching the online resolver's behavior so local keydb mode resolves
the same discs (e.g. an MK present in keydb under a sibling pressing).
km_verifies is one AES-D + magic check per candidate (cheap). Adds
KeyProvider::media_keys() + a path-2.5 unit test.
- mkb_content_len walks the MKB record stream and returns where it ends,
so callers can drop the trailing zero padding. MKB files are allocated
to a fixed size (~128 MiB) with records at the front; both MKB_RO and
MKB_RW can be padded. read_aacs_inputs* now trim to the real records
(~few MB) instead of shipping ~128 MiB of zeros.
- decrypt_keys() returns None when unit_keys is empty: an AACS state that
carries only a VID (out-of-band resolution, pre-key) is 'encrypted, no
keys', not a usable empty key set.
- read_aacs_inputs / read_aacs_inputs_from_drive now read MKB_RO.inf first.
MKB_RW.inf is a fixed ~128 MiB rewritable region that is mostly zero
padding; reading it shipped 124 MiB of nothing. MKB_RO is the real,
correctly-sized MKB (a few MB). Fall back to RW only if RO is absent.
- disable_keydb no longer drops the Volume ID. A caller resolving Unit Keys
out-of-band needs the VID (on-disc content read during the handshake).
New resolve_vid_only builds a keys-free AacsState carrying just the VID +
version metadata, so the disc reports 'encrypted, no keys' (resolved
out-of-band) instead of discarding the VID.
A caller that resolves Unit Keys out-of-band can now set disable_keydb
so the scan consults no keydb at all — neither an explicit keydb_path
nor the standard search locations. Without it, a keydb that merely sits
in a default location ('~/.config/...') silently shadows the out-of-band
path. unit_key still takes precedence over everything.
The ScanOptions.unit_key path is a generic primitive — a caller-supplied Unit
Key that bypasses keydb lookup. Doc comments + a tracing log named a specific
external source; reworded to neutral 'out-of-band / external key service' so
the library makes no assumptions about where the key came from.
Add an optional 16-byte AACS Volume ID to the Mapfile, persisted as a
'# freemkv-vid: <32-hex>' header comment so it survives to deferred-mux
and resume without touching the ISO payload. ddrescue treats the line
as a comment, so the pos/size/status data parser is unaffected.
- Mapfile gains set_vid/vid accessors; save() emits the comment in the
header block, load() parses it back (malformed/absent -> None, never
errors the load). Load->save round-trip preserves it (patch pass).
- Thread vid through CopyOptions and SweepOptions; sweep persists it
when creating/opening the mapfile.
- Unit test: round-trips the VID, asserts the saved comment, and
confirms data ranges parse identically with and without the comment.
Two coherent additions to the AACS resolver:
KeyProvider abstraction (provider.rs) — key material comes from pluggable
backends; KeyDb implements it (device/processing keys, host certs,
disc-by-hash / disc-by-vid lookup) plus orphan-DK parsing. ResolveContext
takes a provider array. Adds the SD-tree PK walker
(derive_media_key_from_pk_walked) and a `probe` module (km_verifies MK
oracle, mkb_* record parsers) used for offline key verification. Cvalue
record selection prefers 0x05, falls back to 0x07.
External-UK key source — the second, mutually-exclusive key source for the
keyserver path. ScanOptions/InputOptions gain `unit_key`; when set,
resolve_encryption_static skips keydb entirely and uses the caller-supplied
Unit Key directly (KeySource::ExternalUk). Disc::read_aacs_inputs exposes a
disc's Unit_Key_RO.inf + MKB so a caller can fetch the UK out-of-band; the
library makes no network call itself.
CHANGELOG: redact test-disc title in historical notes.
Pixelogic discs that authored a SEG_MainFeature menu/preview segment
containing a stray commentary token made the parser anchor on that
segment instead of the real FPL_MainFeature playlist. Combined with
'Audio Stream N' placeholder tokens not advancing the audio counter, a
lone eng_ACOM_ commentary at STN slot 4 collapsed onto slot 1 — tagging
the main feature track as Commentary (seen on Wicked: For Good).
Fix:
- Anchor the feature section on the FPL_ playlist when present; only
fall back to SEG_MainFeature on discs with no FPL_ section.
- Count 'Audio Stream N' placeholders so editorial audio tokens get
their true STN ordinal. Subtitle (PG Stream N) numbering is left
unchanged — counting it regressed several corpus discs.
Extract the section/numbering loop into assign_labels with unit tests.
Verified: 11/11 labels-corpus discs still PASS; Wicked commentary now
lands on the AC-3 2.0 stereo track instead of the main TrueHD.
Adds a 5th key-resolution path that consumes pre-decrypted unit keys
directly from KEYDB when the entry has no VUK field. Covers ~4,572
entries in the public keydb (~2.5%), heavily skewed toward MKBv76+ UHD
discs where DVDFab/FindVUK can no longer extract a VUK but does extract
unit keys. Partial CPS-unit coverage is rejected so a disc is never
half-decrypted.
Resolver path order reordered root-to-leaf: DK (1) → PK (2) →
KEYDB-derived MK+VID (3) → KEYDB VUK (4) → KEYDB unit keys (5).
Previous order was leaf-first.
API:
- AacsState::vuk is now Option<[u8; 16]> (was [u8; 16])
- ResolvedKeys::vuk is now Option<[u8; 16]> (was [u8; 16])
- KeySource variants reordered + new KeyDbUnitKeys variant
3 new resolver tests (path 4 still works without VID; path 5 succeeds
with pre-decrypted unit keys; path 5 rejects partial CPS coverage).
When the drive is in extended-access state (unlocked), retrieve VID via
the per-drive `read_vid_cdb` from the bundled profile instead of the
cert-based AACS REPORT_KEY handshake. Cert handshake remains the
fallback for drives that don't enter extended-access state, or whose
profile lacks the required CDB.
Empirically verified on the BU40N (signature 999ec375) against
Barbie UHD: drive returns 36 bytes from buffer 0x44 at offset
0x10E291, VID at response[4..20]. The 16 bytes match Dune Part Two's
known VID in keydb.cfg byte-for-byte, cross-validating the path
against an independent oracle.
Architectural impact:
- Renames `Drive::is_libredrive_active()` → `Drive::is_unlocked()`.
Internal `Mt1959::libredrive_active` becomes `Mt1959::unlocked`;
the prior `unlocked` (init-success flag) becomes `init_complete`
to avoid the name collision.
- `disc/encrypt.rs::Disc::read_vid` is the single entry point.
When `is_unlocked()` is true, calls `read_vid_oem` (issues the
per-drive CDB, validates the response signature high-3-bytes
`00 22 00`, returns bytes [4..20]). Otherwise delegates to
`read_vid_cert` (the existing AACS REPORT_KEY format 0x80 path).
- `DriveProfile` gains the per-drive CDB templates and identifier
blocks extracted from each per-drive firmware payload — including
`read_vid_cdb`, `read_disc_keys_cdb`, `drive_nominal_speed_cdb`,
`set_speed_max_cdb`, two cache-prime canary CDBs, the buffer-0x45
verify CDB, the firmware-upload CDB, and the unlock probe CDB.
Variants A and B differ in which fields are populated. All optional;
consumers fall back to the cert/handshake path when fields are
absent.
- New error variants `Error::DriveProfileMissing` (E7020) and
`Error::VidCdbUnavailable` (E7021). Both treated as
"OEM unavailable → try cert path" by `read_vid`, not terminal.
Closes the v0.25.x gap where HRL-burned host certs (the public
libaacs leaked cert is on every recent drive's HRL) blocked all
post-handshake VID retrieval. With OEM-driven VID:
- AACS 1.0 BD on supported drives: rips end-to-end with our existing
DKs walking the MKB.
- AACS 2.x UHD: fails honestly at the DK wall (E7018 "No usable DK"
for v77+ MKBs) instead of the misleading E7017 "No Volume ID"
the prior code surfaced. We have VID; we just don't have v77+ DK
material — that gap is a key-acquisition problem, not a code
problem.
Empirically verified on rip1 (BU40N + Barbie UHD, MKB v77,
2026-05-21): error code flipped from E7017 to E7018 as predicted.
The DK wall is now correctly the proximate failure for unrippable
modern UHD discs, instead of the indirect VID-retrieval wall the
v0.25.x cert-only path produced.
Renames and comment scrubs eliminate upstream-RE-vocabulary
references in the public crate per `feedback_no_breadcrumbs.md`.
674 tests pass (565 lib + 109 integration). No tradename leaks in
any modified file.
- Introduce DrmScheme enum (Css/Aacs10/Aacs20/Aacs21) + drm module with
uniform detect/load dispatch across all four protection schemes.
- Land AACS 2.1 Media Key Variant framework in aacs::variants: chain
derivation, MKB record types 0x82/0x83, bit-0x02 SoftKCD and bit-0x04
online-challenge detection. Aacs21 dispatcher arm wired but commented
out pending validation against a Variant-scheme disc.
- Replace aacs2: bool with AacsVersion enum across ContentCertificate,
UnitKeyFile, ResolvedKeys. resolve_keys splits into _v1/_v2/_v21.
- Delete the libredrive raw-read VID shortcut from do_handshake; the
drive enforces the AGID requirement regardless of firmware-upload
state, so the shortcut spuriously dispatched E7017 instead of
surfacing the real downstream walls.
Three coherent threads landing for v0.25.11:
1. Libredrive raw-read VID path. When Mt1959::do_unlock sees both the
MMkv active-mode marker at [12..16] and the LbDr mode-ID marker at
[16..20], Drive::is_libredrive_active() returns true and
do_handshake skips the AACS cert dance — VID is retrieved via
READ_DISC_STRUCTURE format 0x80 with AGID=0 and bus encryption is
already off. This unblocks UHD ripping on drives whose leaked host
cert is on the AACS HRL.
- platform/mt1959/mod.rs: detection + active flag + 4 unit tests.
- platform/mod.rs: PlatformDriver::is_libredrive_active trait method.
- drive/mod.rs: Drive::is_libredrive_active accessor.
- disc/encrypt.rs: do_handshake branches on the flag; new
read_volume_id_libredrive helper. Return type widened to
(Option<HandshakeResult>, Option<Error>) so callers see which
specific failure happened.
- disc/mod.rs: scan_with plumbs the new tuple through and preserves
handshake errors as disc.aacs_error.
2. Revert v0.25.9 built-in AACS keys + plugin slot. Single source of
AACS truth: keydb.cfg. The compiled-in DKs/PKs were a slim
convenience that didn't move the hard problem (no v77+ DKs) and
added a maintenance surface. Plugin slot was overlapping
functionality with the main keydb.
- Deleted src/aacs/builtin_keys.rs (4 DKs + 3 PKs).
- Removed KeyDb::with_builtins, load_or_builtins, merge_from,
merge_local_plugin, local_plugin_path, internal dedup helpers.
KeyDb::empty kept for unit-test use.
- KeyDb::load reverts to pre-0.25.9 form: read file or return I/O
error; no fallback.
- disc::encrypt::resolve_encryption keydb_path back to required
(&Path), not Option<&Path>.
- disc::scan_with surfaces KeydbLoad { path: "<no keydb in search
paths>" } sentinel when encrypted + no keydb — same sentinel
autorip's message switch already handles.
- CSS player keys in src/css/auth.rs stay compiled in; they're
1999-era public inputs separate from AACS and pre-date the 0.25.9
additions.
3. Walker fix follow-through (libaacs-parity validate_processing_key,
cvalues 0x07-then-0x05 preference, path-2/3/4 short-circuit on
zero VID) + NIST AES-CMAC KAT + VID MAC round-trip / mutation /
zero-rejection tests.
5 new Error variants for finer-grained AACS failure reporting:
AacsHostCertRejected (E7015), AacsLibredriveUnsupported (E7016),
AacsVidUnavailable (E7017), AacsMkUnavailable (E7018),
AacsVukNotInKeydb (E7019). Lets CLIs/UIs render which piece of the
AACS chain failed instead of always saying "no keys."
Two changes that make AACS 1.0 / DVD self-sufficient:
1. MKB record-type identification bug fix. `mkb_find_mk_dv` was
searching for type 0x10 (which is Type-and-Version, 12 bytes)
when the Verify Media Key Record is actually type 0x81 for
AACS 1.0 or type 0x86 for AACS 2.0/2.1. `mkb_version` had the
inverse bug. PK and DK derivation paths therefore silently
failed on every disc, masking how often the fallback paths
could have worked. Fix searches the correct types; tests added
covering both the 0x81 and 0x86 verify-record forms and the
0x10 version record at offset 8 of the body.
2. Built-in AACS keys + operator plugin slot. Four device keys
(covering MKB v01-v82+) and three processing keys (covering
v63-v68) compiled directly into the library. Combined with the
31 CSS player keys already in css/auth.rs, DVDs and Blu-rays
(AACS 1.0) now decrypt with zero external files. New plugin
path at ~/.config/freemkv/local_keys.cfg (same syntax as
keydb.cfg) layered additively on top of built-ins and main
keydb. `Disc::scan` no longer errors when keydb.cfg is absent;
AACS 2.0 / UHD still surfaces a specific error when the disc
needs keys none of the layers provide.
Public docstrings in project docs + README updated to describe the
three additive layers (built-ins → keydb.cfg → local_keys.cfg).
Pre-0.25.7 the AACS authenticate loop fired up to 16 host-cert
attempts back-to-back with no pause. Each attempt is 5-10 SCSI
REPORT_KEY/SEND_KEY exchanges, so on a disc whose host cert isn't
in our KEYDB (or one the drive rejects), the drive saw 80-160 SCSI
commands in a few hundred ms and entered a fast-fail firmware
wedge state where every subsequent CDB returns sense 05/24 until
power-cycled.
Three defences:
- MAX_CERT_ATTEMPTS capped at 3 (was 16)
- 1-second sleep between attempts
- Bail immediately on any sense_key == 0x05 (ILLEGAL_REQUEST) so
the loop can't deepen the wedge if a regression undoes the
attempt cap.
- MkvTrack::audio emits A_DTS/MA, A_DTS/HR, A_DTS per the DTS family
instead of mislabelling everything as A_DTS. Plex transcoder and
strict hardware decoders reject DTS-HD MA payload under a plain
A_DTS track.
- PgsParser is now stateful: pairs display PCS with the following
empty PCS to compute a duration. Frame::duration_ns + PesFrame::duration_ns
carry it through; MkvMuxer::write_frame gains a final Option<u64>
parameter that emits BlockGroup + BlockDuration when set. Fixes
subtitle bitmaps lingering past their intended end-time.
* `PrefetchedSectorSource::new_with_events` adds an optional
`event_fn` callback that fires `BytesRead` after every successful
batch from the producer thread. The original `new()` becomes a
thin no-events wrapper. Lets autorip wire the highway and still
get UI progress events without polling the consumer side.
* `build_iso_pipeline` grows an `event_fn` arg so the autorip
multipass mux can pipe BytesRead straight through to its progress
UI.
* Stream trait gains a default `errors() -> u64` method (= 0) so
Box<dyn Stream> callers (autorip's mux loop) can read the
skip-on-error counter without downcasting. `DiscStream` overrides
to return its `errors` field.
* Delete `DiscStream::new_pipeline` and the pipeline-mode fields
(`demux_thread`, `demux_rx`) plus the `read_pipeline` helper.
All pipeline construction now goes through
`PipelinedPesStream` via `build_iso_pipeline`; `DiscStream`
becomes the single-thread-only inline path used by the drive
single-pass read.
* `lib.rs` re-exports `build_iso_pipeline`.
Introduces the freemkv mux throughput highway: a three-stage thread
pipeline that replaces the inline single-thread read path for any
file-backed source (ISO and m2ts file URLs both route through it).
Thread A: read + decrypt (PrefetchedSectorSource / BytePrefetcher)
Thread B: M2TS demux (DemuxThread)
Thread C: codec parse (PipelinedPesStream, on caller thread)
Each handoff uses a bounded crossbeam channel with a recycled buffer
pool — no allocations or memcpys in the steady-state hot loop.
Component map:
* io/byte_prefetcher.rs (new) — std::io::Read producer thread with
recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector
side) so demux_thread::spawn_zero_copy can wire either upstream.
* sector/prefetched.rs — recycled buffer pool added; into_channels()
peels off the rx/recycle_tx/shell triple for zero-copy demux.
* mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs
feed() on its thread, ships Vec<PesPacket> batches.
* mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls
packets from the demux thread and runs codec parse on the caller.
* mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline
(private) assemble the three stages; iso:// and m2ts:// both
return PipelinedPesStream.
* mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted;
the read direction lives on the highway now).
* mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find.
* mux/codec/hevc.rs — tightened frame_data initial capacity.
* mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB
remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page
first-touch fault tax that the previous 256 KiB pre-alloc paid on
every PES boundary.
* mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as
the legacy autorip ingress (drive + multipass paths still need
on_event / skip_errors before they migrate to the highway).
* io/file_sector_source/* — per-OS prefetch() syscall hook
(Linux readahead, macOS F_RDADVISE, Windows/other no-op).
* decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS;
pool sized to all cores by default.
Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://):
60 → 322 MB/s warm cache (old new_pipeline path)
60 → 660 MB/s warm cache (highway path, this commit)
60 → 126 MB/s sustained disk-bound
The IsoSectorReader baseline reader was deleted in favour of
FileSectorSource so the freemkv CLI and autorip exercise the same
read path.
MKV: cluster boundaries now require a video keyframe in addition to the
5s minimum, so every cluster has a CuePoint at its start. Pre-first-
keyframe frames are dropped. Adds a SeekHead at Segment start with
fixed-width back-patched SeekPositions for Info/Tracks/Chapters/Cues.
Before this change a 2h26m UHD rip had 52 CuePoints across ~1750
clusters and a 16.5-minute gap between adjacent seek entries; players
seeking inside that gap had to scan from the prior cue. After, one
CuePoint per cluster.
TS (tsmux production path + m2ts_mux): PesFrame.keyframe is plumbed
end-to-end. Codec-private parameter sets are prepended on the first
keyframe (not the first frame); non-key video before any keyframe is
dropped. The first TS packet of a keyframe video PES carries an
adaptation field with random_access_indicator=1. m2ts_mux previously
hardcoded RAI=1 on every PCR packet; that is now gated on the current
PES being a keyframe video PES, combining correctly with PCR when both
land on the same packet.
Adds 17 tests covering keyframe alignment, cue count/position/timing,
SeekHead correctness, RAI set/clear, codec_private gating, non-key drop,
and PCR+RAI combination.
iter13 strace finding: producer thread (mux ISO reader) spends 80%
of wall-clock in state D (NFS RPC wait), doing 207 preads/sec at
103 us each. Pipe is 78 MB/s isolated read; we use ~25 MB/s.
FileSectorSource was using direct per-sector pread (0.21.3 bypass).
That bypass was justified under Phase 2.5 + 0.21.7 producer-poll cap
("32 MiB refill bursts the TCP connection enough to starve the
writer thread"). Both of those conditions are gone now (iter8
baseline: no Phase 2.5, no producer poll cap).
The 4 MiB buffer infrastructure was preserved with #[allow(dead_code)]
in case re-engagement was ever wanted. iter14 just routes the hot
path through buffer_covers + refill + memcpy. Net diff: ~10 LOC of
business logic, 5 unit tests already cover the contract.
Expected: producer's effective read rate jumps from 25 → 60+ MB/s
(matches dd ceiling for 1+ MiB block reads). If consumer side keeps
up, mean mux climbs from 28.7 → 35-50 MB/s. If consumer is now the
cap, we see a clear ceiling around 30-35 and we know where to look
next.
Critically: this change ONLY affects mux-from-ISO. Disc→ISO sweep,
Pass N bad-sector retry, AACS, drive ops, mapfile, recovery — all
untouched (they use DriveSectorSource which is a separate impl).
iter12 revert restored chunk to 128 (iter11 value) by way of the
revert chain. Setting it back explicitly to 32 for iter13. We're
back to iter8 baseline config (no Phase 2.5, 32 MiB chunks,
DONTNEED, no app-level FileSectorSource buffer, channel depth 32).
iter13 is instrumentation: strace the mux thread during a known-dip
to see where time is actually going.
iter11 (128 MiB) crashed to 16.6 mean — chunk-size sweet spot is
firmly 32 MiB. Locking in.
iter12 hypothesis: each NFS write past the file's EOF triggers a
server-side SETATTR (metadata commit) to update the file length.
With ~62 GiB of MKV output and writes at chunk boundaries, that's
thousands of SETATTRs per rip. By using plain fallocate (mode 0)
the file is pre-extended to size_bytes up front; subsequent writes
overwrite the pre-extended region with no metadata ops.
Adds high_water tracking + truncate_at_sync so we ftruncate down
to actual content size if the size hint was an overestimate.
iter9 (64 MiB on top of no-P2.5) regressed -1.2 from iter8. 32 MiB
remains best on the no-P2.5 path. Net iter10 = iter8 minus DONTNEED
(via the prior commit re-applying f4a881b).
iter8 (Phase 2.5 disabled, 32 MiB chunks) hit 28.7 MB/s mean
— 1.3 MB/s below R2 floor. Without Phase 2.5, WAIT_AFTER blocks the
mux thread directly once per chunk. Halving the WAIT_AFTER frequency
(32 → 64 MiB chunks) should raise the mean without changing the
underlying architecture.
iter7's 64 MiB chunks didn't help (23.1 vs iter4's 24.0). Reverting
to 32 MiB so iter8 differs from iter4 by exactly one variable:
Phase 2.5 disabled / direct passthrough mux thread.
Test question: is Phase 2.5 helping at all? If iter8 mean > iter4,
Phase 2.5 has been a net negative on this rig the whole time. If
iter8 mean < iter4, Phase 2.5 is doing what it claimed.
iter6 (8 MiB chunks) regressed mean -8.2 MB/s vs iter4 (32 MiB).
Trend: bigger = better in this workload, against the page-age
theory. Try 64 MiB to see if fewer/larger syncs raise mean further.
0.21.14 went to 128 MiB and was reverted; 64 is between.
iter5 (256-frame channel) regressed -1.8 MB/s vs iter4. Reverting to
32 frames.
iter4 sample pattern shows clear ~30 s oscillation (peak 45 MB/s →
dip 3 MB/s → recovery). Matches Linux vm.dirty_expire_centisecs
default (30 s). 32 MiB chunks at 25 MB/s issue WAIT_AFTER every
~1.3 s, which can't outrun the kernel's own page-age limit, so
pages buildup then flush in bursts. Smaller 8 MiB chunks
(WAIT_AFTER every ~0.33 s) should keep the dirty-page set young
and eliminate the periodic flush-burst dip.
iter4 measurement (Phase 2.5 + DONTNEED, no FileSectorSource buffer)
showed mean 24.0 MB/s but with dips to 2.8 MB/s. Channel at 32 frames
(~1.6 MiB at 50 KB/frame avg) drains in <100 ms on any producer pause,
starving the consumer.
256 frames = ~12 MiB ≈ 3-4 sec of consumer drain at the rolling-avg
floor we want to hit. Should coast through producer micro-stalls
without idle gaps.
iter1 baseline (Phase 2.5 + DONTNEED restored) measured at 18.4 MB/s
mean on Civil War remux — well below the rig's 37 MB/s concurrent-r+w
ceiling. Per-sector pread is the producer-side bottleneck: ~50us per
pread on NFS = ~19k preads/sec = effective ceiling near what we see.
The 0.21.3 bypass commit cited an A/B test showing the buffer hurt
throughput on NFS bidirectional workloads. That test was taken under
the 0.21.7 producer polling cap; once the cap is gone, the cap was
the bottleneck, not the buffer. Same invalidation pattern as the
0.21.5 Phase 2.5 revert.
Restored the buffered path. DONTNEED page-cache eviction from 0.21.6
stays intact.
Three coupled changes that together meet the "stable max throughput
on any medium" bar:
1. Remove `is_nfs` from `skip_wait`. WAIT_AFTER runs on every medium
now. The `bounded_syscall` 30s safety net (already in place) covers
the wedged-FS case generically — no need to predict NFS-hangs at
compile time. Re-applies the 0.21.12 fix that landed the flat
25 MB/s on NFS in the first place.
2. Drop the `posix_fadvise(DONTNEED)` calls after WAIT_AFTER. Bounded
*dirty* pages (the actual invariant) does not require evicting
*clean* pages. DONTNEED was forcing read-modify-write on any
in-window seek-then-write — exactly the pattern matroska
cluster-size backpatches produce. Empirical 2026-05-15: write side
of mux fed NFS at 43 MB/s while the file grew at 25 MB/s, an 18
MB/s overhead almost certainly composed of those RMW cycles. The
kernel reclaims clean pages under LRU when memory is actually
needed — we don't have to ask.
3. Replace `detect_nfs(fd) -> bool` with `detect_storage_class(fd)
-> (StorageClass, chunk_bytes_seed)`. Medium detection happens
*once* at construction and produces *one* output: the initial
`chunk_bytes` seed for the autotuner. NFS gets 64 MiB (commit ack
~10-30 ms; needs bigger chunks to amortize); other media use the
caller's hint. No hot-path branches on medium. The autotuner drives
all subsequent decisions from measured WAIT_AFTER p95 latency,
identically on every (OS, FS) combination.
Module-level doc rewritten to match: no more "NFS escape hatch", no
"is_nfs" framing. The whole writeback module is now medium-agnostic
except for one labelled detection point.
The pipeline's adaptive autotuner grows chunk_bytes only when the p95
WAIT_AFTER latency exceeds 200 ms. On NFS, sync_file_range(WAIT_AFTER)
translates to an NFS COMMIT RPC whose ack lands within ~10 ms — so
the autotuner never triggered and the pipeline stayed at the original
32 MiB initial value forever.
That capped sustained mux throughput by paying NFS COMMIT-RPC overhead
roughly once per second of writes. Bidirectional mountstats on rip1
2026-05-15: write side 29 MB/s (RTT 30 ms but exec_time 257 ms — 220 ms
queue/serial waiting) while concurrent dd on the same disk shows
~91 MB/s write + ~65 MB/s read available.
128 MiB initial drops COMMIT cadence 4x while keeping the bounded-
cache invariant intact (worst-case dirty pages ~2 x chunk = 256 MiB,
well under vm.dirty_ratio = 6.6 GB on the 32 GB rig). The adaptive
autotuner can still grow further (up to 256 MiB) or shrink if
WAIT_AFTER ever measures sub-20 ms p95 on faster media.
The WritebackPipeline's WAIT_AFTER + posix_fadvise(DONTNEED) dance
keeps dirty pages bounded at ~2 × chunk_bytes by waiting for each
chunk's writeback to commit before issuing the DONTNEED hint to drop
it from cache. The original 0.18-era design unconditionally skipped
this on NFS on the premise that "NFS clients have their own buffering
and commit semantics that handle dirty-page bounds without us forcing
the issue."
Empirically wrong. On unraid-1 NFS the kernel client buffers dirty
pages up to vm.dirty_ratio (default 20% of RAM = ~6.6 GB on the rip1
host) before the kernel forces writeback and throttles app writes.
Result on 0.21.11 mux measured 2026-05-15: mux throughput cycled
between ~45 MB/s (cache absorbing) and ~7 MB/s (cache draining under
throttle) on a ~100 s period — exactly the burst-flush pathology this
pipeline was built to fix, but disabled on the medium it actually
runs on. /proc/meminfo Dirty: column climbed lockstep with mux
write rate during the slow half of every cycle, confirming the cause.
The original safety concern — `sync_file_range(WAIT_AFTER)` hanging
indefinitely on a wedged NFS server — is already handled by
`wait_after_with_timeout`'s `bounded_syscall` wrapper (30 s deadline).
If a real WAIT_AFTER call exceeds the deadline the pipeline flips to
the `degraded` state and skips WAIT_AFTER + DONTNEED for the rest of
its life — same effect as the old NFS branch, but only triggered when
something is genuinely broken rather than as a blanket exception.
This change is medium-agnostic: every medium goes through the same
path now, every medium gets the same safety net, and the
ADAPTIVE_WINDOW chunk-size autotuner (lines 226-255 — measures p95 of
WAIT_AFTER and resizes between 4 MiB and 256 MiB) finally activates
on NFS where previously it was dead code. Slow medium auto-grows
chunks to amortise per-chunk overhead; fast medium auto-shrinks to
keep cache pressure tight; nothing in the code special-cases the
filesystem type.
`is_nfs` is still detected (for logging + observability) but no
longer keys `skip_wait`. Module doc + startup log line updated to
match.
Pre-coalescing the Phase-2.5 writer thread issued one `file.write_all`
syscall per `Cmd::Write` dequeued. The mux side calls
`WritebackFile::write_all(buf)` per PES frame, typically 30-200 KB.
On NFS that translates to one RPC per syscall, capping per-thread
throughput at `(wsize / rtt) × inflight` — well below what the same
disk delivers under a 1 MiB `dd oflag=direct` workload (empirical
2026-05-15: dd 71 MB/s vs mux ~25 MB/s sustained, with instantaneous
samples bursting 7→108 MB/s as the kernel page cache filled and
drained on its own cadence).
Coalesce instead: dequeue drains consecutive `Cmd::Write` items off
the ring up to a 1 MiB byte budget, returns them as
`DequeuedWork::Writes(Vec<Vec<u8>>)`, and the run loop concatenates
into one contiguous buffer and issues a single `file.write_all`.
Non-write commands (Seek, Flush, SyncAll, Finish) break the run and
are returned one at a time as `DequeuedWork::Other`, preserving their
ordering relative to the writes.
Single-buffer fast path avoids the concat allocation when only one
write is in the queue at dequeue time. A single oversize write (e.g.
the rare matroska cluster larger than 1 MiB) is admitted alone so it
still makes progress — the kernel splits internally.
This is generic across mediums: bigger app writes are at-least-as-
good on local SSD, HDD, or NFS. On fast storage the ring rarely fills
so coalescing is mostly a no-op; on slow storage with significant
per-RPC overhead it materially improves throughput.
Tests in `tests/` (write_then_drop_persists_bytes, sync_all_drains,
seek_then_patch_roundtrip, flush_is_observed_in_order) still pass —
ordering and durability semantics are unchanged.
The depth was conservative because pre-0.21.8 a full sync_file_range
stall on NFS could traverse this channel and pin the producer. With
0.21.8's restored Phase 2.5 writer thread + 128 MiB byte-bounded ring
inside WritebackFile, every blocking syscall happens downstream of
this channel — never on it. The original "smaller buffer reduces
backpressure risk" rationale no longer applies.
Empirical (2026-05-15 Civil War UHD remux on 0.21.8): 30.9 MB/s
sustained but instantaneous samples spanning 9-53 MB/s, stdev 9.3.
Distribution clusters 52% of samples in 25-35 MB/s but has a long
9-15 MB/s tail. The tail corresponds to brief stalls in the matroska
builder when the sink momentarily lags — exactly the case a deeper
inter-thread channel covers. 32 frames at PES-frame sizes is still
well under a megabyte of additional memory, so the cost is zero.
Reverts 523af46. That revert was made on the premise that Phase 2.5
caused a ~60% mux throughput regression on NFS bidirectional workloads.
The premise was wrong: at the time of measurement the producer was
capped at ~8 MB/s by a 50 ms thread::sleep poll in
Pipeline::send_with_halt (fixed in v0.21.7's io/pipeline change), so
the comparison was measuring the polling cap on both sides.
With the polling cap removed, direct passthrough exposes the kernel's
default dirty-page writeback pathology on NFS: writes accumulate, the
kernel periodically bursts a flush, app writes block for the burst.
Observed empirically on Civil War UHD remux 2026-05-14: 5-45 MB/s
spiking around a ~21 MB/s sustained mean, dominated by burst-flush
back-pressure cycles.
Phase 2.5 decouples the mux thread from the file syscall:
* mux writes complete instantly into a 128 MiB byte-bounded SPSC ring,
* a dedicated writer thread executes the real File writes, seeks,
and sync_file_range calls; can sit in a kernel burst without
blocking the mux pipeline,
* backpressure via Condvar notify/wait, no polling primitive,
* the ActiveClusterBuffer fast-path preserves the original MKV
cluster-backpatch optimisation so in-window seeks don't drain
the current writeback chunk.
Halt-safety is preserved: every blocking writeback syscall on the
writer thread still routes through bounded_syscall with a 60 s
deadline. A wedged NFS server cannot trap the writer indefinitely;
the muxer keeps queueing into the ring; the kernel page cache and
the ring together absorb the stall.
The halt-aware send loop polled try_send on a 50 ms sleep slice when
the channel was full. That capped producer throughput at 1 / 50 ms =
20 frames/sec ≈ 1 MB/s at typical PES frame sizes — way below NFS,
let alone local SSD/NVMe. Multi-day diagnostic 2026-05-13/14 surfaced
it as the root cause of the 18 → 3 MB/s mux throughput regression on
0.21.x.
Replaced std::sync::mpsc::sync_channel with crossbeam_channel::bounded
and switched send_with_halt to send_timeout. Producer now BLOCKS on
consumer drain (kernel-wakeup) instead of polling — the timeout slice
(250 ms) only fires when the producer is waiting for stop signal
observation, never on the happy path.
Result: no throughput cap from this primitive at any medium speed.
Mux is bounded by actual storage / network bandwidth, not by our
channel implementation.
Same fix needs to land in autorip's mux.rs producer-side
sync_channel (separate sibling commit).
Documented in (internal)/memory/
feedback_send_with_halt_poll_throttle.md.
Empirical: isolated NFS read 70 MB/s + write 93 MB/s on the rip1 setup
right now, but mux throughput pinned at 2.7 MB/s on 0.21.5. NOT
environmental — code regression.
Root cause: Phase 1 silently dropped the read-side
posix_fadvise(POSIX_FADV_DONTNEED) eviction that the pre-Phase-1 (0.20.7)
hot path had. Without it, an 85 GB streaming ISO read pins the entire
file in the kernel page cache, starving concurrent MKV writeback. 0.21.2
then also dropped the POSIX_FADV_SEQUENTIAL hint on the same theory,
compounding the regression.
Restored both, per-OS split:
- linux: posix_fadvise(SEQUENTIAL) at open + posix_fadvise(DONTNEED)
on consumed 32 MiB windows
- macos: F_RDADVISE hint at open (kept); drop_window no-op (macOS unified
buffer cache less prone to the pin pathology)
- windows / other: both no-op stubs
Target mux speed restored to 20+ MB/s (per concurrent-NFS math:
70/2 read × 0.73 MKV/ISO ratio ≈ 25 MB/s achievable).
The Phase 2.5 writer-thread + bounded ring + ActiveClusterBuffer
architecture introduced a ~60% mux throughput regression on NFS
bidirectional workloads (18 MB/s -> 7-8 MB/s in 0.21.x). Other
candidates (FileSectorSource readahead buffer, per-OS sink split)
ruled out empirically across 0.21.1-0.21.4.
Reverts WritebackFile's hot path to direct File passthrough matching
the 0.20.7 baseline:
* Write/write_all/flush call straight through to self.file.
* Seek calls through to self.file and notifies the pipeline.
* sync_all runs pipeline.finalize() then per-OS durable_sync.
Kept (intentional): the per-OS file split established in earlier work
(writeback_file/{linux,macos,windows,other}.rs). create_with_size_hint
still dispatches to platform::preallocate and sync_all still routes
through platform::durable_sync (bounded_syscall 60s deadline on
Linux/macOS), so halt-safety on a wedged NFS fsync is preserved.
Removed:
* Cmd / RingState / Shared / WriterState / writer_thread_main
* ActiveClusterBuffer (active-cluster window for in-window seeks)
* push_command / dequeue / publish_error / mark_writer_gone
* RING_CAPACITY_BYTES, ACTIVE_CLUSTER_WINDOW_BYTES,
MAX_WRITE_CHUNK_BYTES, WRITER_THREAD_NAME constants
* writer JoinHandle field + Drop join logic
* Tests that asserted internal writer-thread state
(backpressure_blocks_when_ring_full, in/out_of_window_seek_*,
active_cluster_buffer_*, writer_thread_panic_surfaces_on_drop)
Kept tests (correctness-of-output only):
* write_then_drop_persists_bytes
* sync_all_drains_and_flushes (renamed from sync_all_blocks_until_ring_drains)
* seek_then_patch_roundtrip (replaces in_window_seek_then_patch_roundtrip)
* flush_is_observed_in_order
mod.rs: 1026 -> 303 lines. linux/macos/windows/other.rs untouched.
The three tests (multi_sector_read_spanning_buffer_boundary,
backward_seek_rebuffers, partial_buffer_at_eof) were asserting
internal buf_start_lba / buf_len_sectors state. With 0.21.3's
read-path bypass, those fields are no longer mutated. The
byte-level contract assertions (read returns correct bytes for
every scenario the tests cover) remain intact.
The 32 MiB readahead window (0.21.0–0.21.1) regressed mux throughput
on NFS bidirectional workloads vs the pre-Phase-1 0.20.7 baseline
(18 -> 7-8 MB/s). The 0.21.2 4 MiB shrink made it worse (5-6 MB/s).
Both signs point at the application-level buffer itself, not the size.
This commit bypasses the buffer entirely on the read path — every
read_sectors call seeks and pread()s direct to the file. That matches
0.20.7's hot path. Kernel readahead handles the policy; on NFS that
interleaves naturally with concurrent writes on the same TCP
connection.
Buffer state fields and refill/buffer_covers are kept so the
structure is preserved for a future per-source-type policy (e.g. a
local-disk source where batched reads ARE beneficial), and so the
existing tests still exercise that machinery.
Empirical regression observed during 0.21.1 mux test on rip1/unraid-1:
historical 0.20.7 baseline averaged ~18 MB/s mux throughput; 0.21.1
dropped to ~7-8 MB/s flat. Same NFS source + destination, same disc.
Suspect: 32 MiB FileSectorSource readahead + posix_fadvise(SEQUENTIAL)
together saturate the TCP connection on read bursts, starving the
writer thread's concurrent NFS writes (mux reads the source ISO and
writes the MKV over the same connection).
- READAHEAD_BUF_BYTES: 32 MiB -> 4 MiB. Matches NFS rsize=1 MiB * 4
round-trips per refill, interleaves cleanly with writes.
- linux/hint_sequential: now no-op. Kernel's default ~128 KiB
readahead is what we want on NFS-backed ISOs (the dominant case).
Per-OS file stays so we can re-enable a hint cleanly later if a
different path benefits.
CI's lint workflow runs clippy on linux target where:
- platform/fs_type/linux.rs and io/writeback/linux.rs: the i64 cast
on buf.f_type / NFS_SUPER_MAGIC is unnecessary on glibc x86_64 (both
already i64) but required on musl (c_ulong); silence the lint via
inline allow with explanatory comment.
- mux/m2ts_mux/packet.rs: doc comment continuation across lines was
parsed as an unindented list item. Reworded to a single flowing
sentence.
- m2ts_mux: replace u64::is_multiple_of (stable in 1.87+) with %.
CI on the pinned 1.86 toolchain rejected the unstable feature use.
- mux/{fmp4,hevc,m2ts_mux}: rustfmt drift cleanup (test-code wrapping).
SocketSink + UdpSocketSink (`src/io/sink/socket.rs`) — sequential-only
TCP/UDP write destinations. SocketSink wraps BufWriter<TcpStream> with
1 MiB capacity, tunes SO_SNDBUF on construction, calls shutdown(Write)
on finish(). UdpSocketSink emits one datagram per write — caller
packetizes. Both impl Write+Send and thus satisfy SequentialSink via
the Phase 2 blanket; neither impls Seek, so RandomAccessSink is
correctly inaccessible (compile error to mux MKV onto a socket).
New sequential container muxers in src/mux/:
- hevc/ — raw HEVC Annex B elementary stream. Length-prefixed NALU
→ 00 00 00 01 NALU. hvcC parsing emits VPS/SPS/PPS once at stream
head. Fully ships.
- m2ts_mux/ — standard MPEG-TS (188-byte packets). Single program,
HEVC video on PID 0x100, optional AC3/TrueHD audio on PID 0x101.
PAT+PMT re-emitted every 250 packets; PCR stamped on video every
40 packets. Hand-rolled, no new deps. Distinct from the existing
BD-TS (192-byte) `mux::m2ts::M2tsStream` — that path stays as-is.
- fmp4/ — fragmented MP4. STUB: ftyp + minimal moov skeleton with
one HEVC video trak + mvex/trex. Media fragments (moof+mdat) are
TODO for v0.22.0 — write_video accumulates frames into a pending
buffer that finish() clears. Init segment is well-formed enough
that init_segment_starts_with_ftyp_then_moov asserts the box
chain.
17 new unit tests added (socket round-trip, HEVC Annex B conversion,
M2TS packet alignment + PAT/PMT cadence + per-PID CC, fMP4 box chain).
All 514 lib tests + 17 new = pass on Rust 1.86 (fmt + clippy + test
via (internal)/scripts/precommit.sh libfreemkv).
No new dependencies. No version bump. Don't-touch list clean.
WritebackFile now offloads all File I/O to a dedicated writer thread.
Muxer's Write/Seek/sync_all push into a bounded byte ring; the writer
drains the ring and runs the syscalls.
- Commands: Write(Vec<u8>), Seek(SeekFrom), SyncAll(oneshot),
Finish(oneshot). One ring carries all four; ordering preserved.
- ActiveClusterBuffer mirrors the last ACTIVE_CLUSTER_WINDOW_BYTES
written. Seek-back within window → in-memory patch + re-emit
(no forced drain). Seek-back outside window → real drain+seek.
Wins for MKV: cluster size patches almost always land inside the
active window; only the end-of-mux Cues + segment header backpatch
fall outside.
- Ring capacity RING_CAPACITY_BYTES = 128 MiB. Backpressure on full
blocks the muxer (correct semantics for archival workflows).
- All syscalls in writer thread wrapped in bounded_syscall(60s) so
a wedged NFS doesn't pin the thread forever.
- Drop drains via Finish + joins the writer thread; sync_all blocks
until ring drained AND underlying fsync completes.
Introduces the SequentialSink / RandomAccessSink trait pair under
io::sink and an open_for_mkv dispatch helper that picks WritebackFile
on Linux+NFS and LocalFileSink everywhere else. LocalFileSink wraps
BufWriter<File> with a 4 MiB buffer and exposes a per-OS preallocate
path (fallocate on Linux, F_PREALLOCATE on macOS, no-op fallback).
Adds platform::fs_type::detect with a per-OS split (statfs on Linux /
macOS, UNC heuristic on Windows, Unknown elsewhere) so construction-
site dispatch has a single primitive to call.
Blanket impls cover the common shapes: any Write+Send is a
SequentialSink, and any SequentialSink+Seek is a RandomAccessSink.
WritebackFile satisfies the random-access trait via the blanket impl
without needing an explicit per-type impl. No callers wired yet — the
mux::resolve construction sites stay on WritebackFile pending Phase 3.
Tests: 5 new sink/preallocate tests + 3 fs_type tests (1 ignored,
needs a real NFS mount). cargo +1.86 fmt + clippy + tests all green.
Three changes targeting 0.20.9's "muxer never read-stalls on NFS read
latency" invariant:
A. FileSectorSource gets a 32 MiB internal read-ahead buffer
(READAHEAD_BUF_BYTES). Splits out from src/sector/file.rs into
src/io/file_sector_source/ with per-OS open hints (Linux
posix_fadvise(SEQUENTIAL), macOS fcntl(F_RDADVISE) with 64 MiB
cap, Windows TODO stub, BSD/illumos no-op). Backward seeks
rebuffer; partial reads at EOF return only the bytes that exist;
oversize-request bypass for count > BUF_SECTORS.
B. WritebackFile inline #[cfg(target_os = "linux")] blocks split
into per-OS files under src/io/writeback_file/. Linux unchanged
(fallocate KEEP_SIZE, fsync via bounded_syscall). macOS gets a
real F_PREALLOCATE + F_FULLFSYNC impl (was a "skipped (non-linux)"
debug log before). Windows is a stub (FlushFileBuffers via
std sync_all; TODO for SetFileValidData). BSDs/illumos fall back
to std sync_all.
C. New byte_channel module — byte-bounded producer/consumer wrapping
std sync_channel with Mutex/Condvar byte accounting. Sender blocks
when used_bytes + item.byte_size() > capacity. HasByteSize impl
for PesFrame. Default cap BYTE_CHANNEL_DEFAULT_CAPACITY = 64 MiB,
sized to absorb worst-case NFS read p99 (~2 s × UHD peak compressed
~15 MB/s). The mux call site lives in autorip (out of scope here);
this lands the primitive in libfreemkv for autorip to adopt.
Test counts: byte_channel +6, file_sector_source +5, sector::file
round-trip suite (3) preserved. passn_handler_ab.rs A/B fixture
(8 profiles) still green.
precommit.sh libfreemkv: fmt + clippy + test all green on Rust 1.86.
No version bump; no Cargo.lock changes; no forbidden-file edits
(disc/patch.rs, disc/read_error.rs, io/pipeline.rs,
tests/passn_handler_ab.rs).
The 0.20.7 work lives downstream in autorip — process-level safety net
(hard watchdog 5-min mux escalation + std::process::exit, restart-loop
counter + auto-quarantine after 3 attempts, partial-state preservation,
failure_reason surfaced in /api/state).
Bumping libfreemkv to 0.20.7 keeps the 4 crates on the unified release
train per project docs.
Generalizes 0.20.5's hand-written wait_after_with_timeout into a
reusable primitive. After this change, every blocking syscall in the
recovery + mux paths is wrapped, so cooperative Halt has bounded
~250 ms latency reach even into kernel-owned thread states.
New module src/io/bounded.rs:
- BoundedError { Halted, Timeout, WorkerLost }
- bounded_syscall<F, R>(halt: Option<&Halt>, timeout, op) -> Result<R, BoundedError>
- Worker thread runs op; main thread recv_timeouts on a rendezvous
channel in 250 ms slices, polling halt between slices.
- Worker is intentionally leaked on timeout/halt — kernel reaps when
the syscall finally returns or at process exit. Calling thread is
NEVER trapped inside a kernel call.
- 6 unit tests cover the happy path + each error variant.
Refactored callsites:
- src/io/writeback/linux.rs::wait_after_with_timeout now delegates
to bounded_syscall. ~30 LOC of duplicated channel/thread plumbing
deleted. Same semantics, cleaner.
- src/io/writeback_file.rs::WritebackFile::sync_all now wraps the
final libc::fsync(fd) with bounded_syscall (60 s deadline). On
timeout: log error at target=mux and return Ok — kernel will flush
on close, best-effort but bounded. Covers FileSectorSink::finish,
PatchSink::close, SweepSink::close, and the mux MKV finalize path
(they all sync through WritebackFile).
What still hangs (deliberately not wrapped — too hot a path):
- File::write itself. Per-frame write on a wedged NFS could still
block; but back-pressure from a stuck consumer means the producer
notices within seconds, not minutes — different failure mode than
the WAIT_AFTER hang 0.20.5/0.20.6 fix.
Targets the recurring mux hang on NFS dest where the consumer thread
sits indefinitely inside libc::sync_file_range(SYNC_FILE_RANGE_WAIT_AFTER)
because the NFS server never returns a commit ack. The whole rip
wedges; halt is cooperative and can't reach inside a kernel syscall.
A. NFS detection at WritebackPipeline construction (fstatfs f_type ==
NFS_SUPER_MAGIC 0x6969). When NFS:
- Skip SYNC_FILE_RANGE_WAIT_AFTER entirely.
- Skip posix_fadvise(DONTNEED) — NFS client handles its own buffering.
- Still issue async SYNC_FILE_RANGE_WRITE (harmless hint).
Cannot hang on a syscall not made. fstatfs failure fails open (assume
local). Logged at info on construction so operators see which strategy
is active. The whole hang vector is removed for NFS deployments.
B. Hard timeout on WAIT_AFTER for non-NFS (defense in depth, since
even a degraded local disk could in principle hang the syscall).
Each WAIT_AFTER runs on a worker thread; main thread waits on a
sync_channel rendezvous with 30s deadline. On timeout: log error,
set per-pipeline 'degraded' Arc<AtomicBool>, downgrade to NFS-style
skip for the rest of the pipeline's life. Worker thread leaks
intentionally — it'll unwind when the syscall eventually returns or
the process exits. Converts indefinite freeze into 'log loud +
downgrade + keep ripping'.
C. Diagnostic logging for the 73%-of-this-movie reproduction:
- WritebackFile::seek logs every non-trivial seek (from, to, signed
delta) at target=mux so we can see if MkvMuxer seeks back before
a stall.
- WritebackPipeline::finalize logs the chunk being finalised before
any WAIT_AFTER call, so a hung chunk is identifiable by offset.
No new dependencies. macOS / Windows noop stubs unchanged. Net
+198 LOC libfreemkv (mostly writeback/linux.rs).
Four targeted changes to maximize mux throughput regardless of storage
backend (local SSD, local HDD, NFS, network share) and surface enough
log data to diagnose 'mux slow' reports without a re-rip:
1. POSIX_FADV_SEQUENTIAL on FileSectorSource::open (Linux only).
Widens the kernel readahead window for sequential ISO reads. One
syscall at open, free on every storage type.
2. POSIX_FADV_DONTNEED on the ISO read side after every 32 MiB chunk.
Mirrors the writeback DONTNEED that already runs on the write
side. Keeps the read-side page cache bounded during multi-GB ISO
reads — eliminates the OOM-pressure / eviction-storm risk on
long mux runs. Linux only; per-drop trace at target="mux".
3. WritebackFile::create_with_size_hint(path, size_bytes) calls
fallocate(FALLOC_FL_KEEP_SIZE) on Linux to pre-reserve extents
for the output. Reported file size stays 0 (writes grow it
naturally) but the on-disk extent allocation is contiguous —
reduces extent fragmentation for big sequential muxes. Wired
into mkv:// and m2ts:// output paths via DiscTitle::size_bytes.
No-op on macOS/Windows; old create() kept with #[allow(dead_code)]
for callers without a size hint.
4. Adaptive WRITEBACK_CHUNK_BYTES in the Linux writeback pipeline.
Tracks sync_file_range(WAIT_AFTER) elapsed_ms in a rolling
16-sample window. p95 > 200 ms → double chunk size (cap 256 MiB).
p95 < 20 ms → halve (floor 4 MiB). One algorithm, both
fast-storage (small chunks, responsive) and slow-storage (big
chunks, fewer commit round-trips) optimized. Per-chunk trace +
per-32-chunk debug snapshot + info-on-resize so an operator can
see where the autoscaler settled.
All four are universal — no storage-type detection, no env vars to
flip, no per-deploy tuning required. Total +201/-6 across four files.
WO-5 (partial): the patch backtrack inner loop ('while bt_pos <
backtrack_end' in disc/patch.rs) issues per-sector reads to fill the
gap created by a damage-window skip. A long backtrack span can run
minutes; without an inline halt poll, the outer halt only takes
effect when control returns to the per-range loop. Adds a halt poll
at the top of each iteration so cancellation propagates inside the
backtrack span.
Per-sector read failures inside the backtrack already drop through
to the main fail path; this only changes the cancellation latency
between an /api/stop call and the producer actually unwinding. Drops
worst-case unwind from 'whole backtrack span × per-sector recovery
timeout' (minutes) to 'one in-flight SCSI command' (seconds).
The broader Arc<AtomicBool> → Halt migration on CopyOptions /
SweepOptions / PatchOptions / Drive::halt and Pipeline::send halt-
awareness is deferred — separate cycle, larger API impact.
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.
WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.
Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
(write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
&mut dyn SectorSource. The trait method capacity() becomes
capacity_sectors() with a default of 0 (preserves SectorReader's
default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
the FileSectorReader type alias. Adds explicit forwarding impls
for Box<dyn SectorSource> and &mut dyn SectorSource so generic
decorators like DecryptingSectorSource<S: SectorSource> compose.
WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
behavior change — pure mechanical relocation. disc/mod.rs drops
from 3,945 to 2,714 LOC.
WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
detect() returning false, never wired into the PARSERS registry.
project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
capped at RANGE_BUDGET_CAP_SECS=1800.
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.
Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports
Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
concrete impl was already Send-compliant — Box<dyn Read + Send>
and Box<dyn Write + Send> were already in place on the trait
objects MkvStream / M2tsStream / etc hold internally. Promoting
Send into the trait makes Box<dyn Stream> Send too, which lets
autorip drop its SendStream unsafe newtype.
The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.
Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.
Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2
Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
scan_with() collapsed every failure path from resolve_encryption() into
None via .ok(), so callers couldn't tell the difference between "no
KEYDB found", "KEYDB failed to parse", "disc hash not in KEYDB and
fallback derivation failed", "AACS files unreadable on disc", and a
handshake that rejected every host cert. autorip's UI was stuck
printing "no decryption keys found (check KEYDB)" for all of them,
which is a particularly bad message when the user has actually loaded
a KEYDB and the real failure is something else.
Changes:
- New pub field Disc.aacs_error: Option<Error>. Populated by scan_with
whenever encrypted && aacs.is_none(). Sentinel KeydbLoad path
"<no keydb in search paths>" distinguishes the no-keydb case from
a real load failure without adding a new Error variant (which would
be a breaking change for downstream exhaustive matches).
- tracing::warn in scan_with at scan_aacs_resolve_failed and
scan_aacs_no_keydb, with error_code and keydb path for grepping.
- tracing in do_handshake: keydb load failure, host-cert exhaustion
(with cert count and last error code), VID read failure post-auth,
and a debug-level success log. Lets us see whether handshake even
got off the ground for a given disc.
Test fixtures updated to set aacs_error: None.
The 'All probes failed — possible wedge condition' log fired during patch
probing whenever 10+ consecutive failures hit AND a probe sweep at the
local zone returned 0 successes. This was distinct from the read_error.rs
'wedge_transition' log that fires when the SCSI sense family ACTUALLY
flips into Hardware/IllegalRequest fast-fail mode.
Two logs both saying 'wedge' caused operator confusion during the
2026-05-11 Dune Pt 2 wedge investigation — was the drive wedged, or was
it just a zone of fully-bad sectors? They mean different things.
Relabel to 'patch_zone_fully_bad' with explicit pointer to read_error.rs
for the canonical wedge detection. Same triggering condition; just clearer
wording in the log stream.
Disc-04 (Top Gun: Maverick) re-test 2026-05-11 surfaced a real-world
bdmt_eng.xml where <di:description> contained no prose, only nested
<di:thumbnail href="…"/> elements. The previous parser surfaced
the raw XML fragment as the description string ("<di:thumbnail
href=\"tgm_meta_sm.jpg\" />\\r\\n <di:thumbnail
href=\"tgm_meta_lg.jpg\" />"). Worse than no description.
Fix: filter description candidates that begin with `<` after
trimming. Real prose never starts with an angle bracket; XML-only
content always does. Net: title extraction unaffected (it uses its
own element-priority path); description field drops when it would
otherwise carry XML noise.
Two new bdmt tests, 12 of 12 passing.
Three layered sources of stream labels now, in precedence order:
1. **Framework parser** (paramount/criterion/pixelogic/ctrm/dbp/deluxe)
— editorial labels with purpose/qualifier ("English Atmos",
"Director's Commentary", "English SDH"). High or Medium confidence.
2. **MPLS gap-fill** (`fill_gaps_from_mpls`) — every stream the
playlist references gets at least a basic lang+codec label, even
when the framework parser missed it.
3. **CLPI orphan append** (`append_clpi_orphans`) — streams in
/BDMV/CLIPINF/*.clpi ProgramInfo that NO MPLS playlist references.
Empirical (2026-05-11): ~5% of streams across the 11-disc corpus,
most dramatic on disc-02 (HDMV-only) at 40% CLPI-only.
Orphan numbering: each appended orphan gets
`stream_number = max(existing per type) + N` so playlist-reachable
streams keep their original positions and orphans sort cleanly at
the tail.
Orphan dedup: (stream_type, language, codec_hint) tuple — fuzzier
than PID matching (PIDs aren't carried on StreamLabel) but it's the
only signal available downstream of the gap-fill. False positives
(genuine orphan that happens to share lang+codec with an existing
entry) silently drop, which is the conservative failure mode — the
user-facing display would just see a confusing duplicate otherwise.
`mpls_universal::language_display_name` and `::codec_name` promoted
from private fn to pub(crate) so this module can build orphan labels
with consistent naming.
Tests: 2 new in gap_fill_tests — synthetic-input verification of the
dedup tuple logic and the stream_number assignment. 6/6 tests in the
gap-fill module now passing.
Rewrites the Pass 1 wedge handling from "slow skip after the drive
has already wedged" to "prevent the wedge transition in the first
place." Driven by 2026-05-11 empirical data: the BU40N transitioned
into IllegalRequest fast-fail mode at exactly 7 medium errors in
6.5 seconds (~1 read/sec retry cadence). Once there, only physical
eject + reload clears it — 30s pauses + 1 GB jumps do not.
The fix is the user's mental model from that session:
"We can detect bad reads, failed reads, and asking to read again
fast after causes a wedge. We need to prevent the wedge in the
first place."
Two changes to the centralized error handler:
1. **`for_sweep().fast_jump_threshold = 1`** (was 4). Pass 1 now
JumpAheads on the FIRST outer-batch failure, not the 4th. The
drive never gets back-to-back retries at the same LBA in Pass 1
— every error → jump 64 MB forward + long cooldown. Pass N keeps
`fast_jump_threshold = u64::MAX` because retries on already-known-
bad LBAs are its whole job.
2. **`ZONE_ENTRY_COOLDOWN_SECS = 30`**. The FIRST error after a
clean run (when `consecutive_outer_failures == 1` and we're not
bisecting) uses this long pause instead of the standard 5 s
FAIL_PAUSE_SECS. Gives the BU40N's firmware / bridge internal
retry counters 30 s of breathing room before the next read,
preventing the "7 errors in 6.5 s" cascade. Subsequent errors
in the same zone use the standard 5 s pause (we've already
jumped past the initial damage; further errors mean we landed
in another bad cluster).
Pass N exempt from the zone-entry cooldown — `bisect_on_marginal=
true` skips the long-pause arm. Pass N's per-sector retries on
known-bad LBAs would multiply uselessly with 30 s/error.
Test updates: 4 tests' expected behavior changed under the new
policy. Renamed `pass_1_marginal_skips_instead_of_bisecting` →
`pass_1_marginal_jumps_immediately_not_bisecting`. Renamed
`pass_1_jumps_after_4_consecutive_outer_failures` →
`pass_1_jumps_immediately_on_first_outer_failure`. Updated
`both_passes_pause_on_failed_read_for_wedge_avoidance` (now
`pass_1_zone_entry_uses_long_cooldown` + `pass_n_pauses_uniformly_on_failed_read`).
Cost analysis:
- Clean disc (no errors): unchanged. 0% overhead.
- Lightly damaged (1-2 zones): +30 s per zone = ~1 min total. Fine.
- Heavily damaged (10+ zones): +5+ min total. The trade for never
wedging the drive and getting a usable Pass N afterwards.
Expected behavior on the next damaged-disc rip:
- Pass 1 hits damage at LBA X → jumps 64 MB forward immediately,
pauses 30 s
- Drive's firmware never accumulates the retry pressure that triggers
IllegalRequest fast-fail
- bytes_maybe accumulates faster (we skip more), but Pass N picks up
the slack with proper per-sector recovery — and Pass N can actually
RUN because the drive isn't wedged
Two layered changes, in service of the empirical question "is CLPI
truly redundant with MPLS for label data?":
1. **clpi.rs ProgramInfo parser**. The existing CLPI parser only
walked the EP map (for sector-range lookups). Added a parser for
the ProgramInfo section's per-stream stream_coding_info table:
pid, coding_type, audio_format/rate, video_format/rate, ISO 639-2
language. Spec layout per libbluray clpi_parse.c. Best-effort —
malformed program_info leaves `streams: vec![]`, EP map keeps
working. `ClipInfo` gains a `streams: Vec<ClpiStream>` field.
2. **labels/clpi_audit.rs**. Diagnostic that walks both
`/BDMV/CLIPINF/*.clpi` (via the new program_info parser) and
`/BDMV/PLAYLIST/*.mpls`, builds a (PID → fields) merged view, and
classifies each row:
- `Match`: both sources agree (same coding_type + language)
- `ClpiOnly`: PID in CLPI but no MPLS playlist references it
(orphan stream on disc — reachable via low-level access, not via menu)
- `MplsOnly`: PID in MPLS but no CLPI lists it (would indicate a
parser bug; verified empirically that this NEVER happens)
- `Divergent`: same PID, different coding_type or language between
sources (playlist re-tagged or attribute encoding mismatch)
Surfaced via `labels-analyze` as `clpi_vs_mpls_audit: {matches,
clpi_only, mpls_only, divergent, total_pids}`. Doesn't affect the
label output — pure diagnostic.
Empirical findings on the 11-disc corpus (excl. disc-04 truncated):
- 226 matches / 0 mpls_only / 8 clpi_only / 5 divergent across 239 PIDs
- 6 of 10 non-truncated discs have CLPI-only streams (orphans)
- disc-02 (HDMV-only) is the most dramatic: 40% of its 5 streams are
CLPI-only — MPLS sees 3, CLPI sees 5
- Conclusion: CLPI is NOT truly redundant. ~5% of streams disc-wide
are CLPI-exclusive. Future work: layer CLPI as a tertiary source
below MPLS in the labels pipeline (orphan streams marked with even
lower confidence than MPLS).
LabelAnalysis gains `chapter_summary: Vec<ChapterSummary>` — one row
per .mpls file in /BDMV/PLAYLIST/, with chapter count (PlaylistMark
entries with mark_type ≤ 1) and approximate playlist duration in
seconds. Sorted by playlist filename.
Sourced from the existing crate::mpls parser (no new format work).
Useful for identifying the main feature playlist at a glance — it's
the one with the longest duration. Verified on disc-11 (Dune Pt 2):
00800.mpls correctly identified as 2h 45m 49s with 18 chapters
amid 30+ shorter playlists.
Doesn't touch the per-title `disc::DiscTitle::chapters` field which
disc::bluray.rs already populates from the same marks during disc
init — this is purely the diagnostic surface for labels-analyze.
When a framework parser (paramount, criterion, pixelogic, ctrm, dbp,
deluxe) is chosen but its label list covers only a subset of the
stream slots MPLS knows about, merge MPLS-derived entries for the
uncovered (stream_type, stream_number) slots. Framework labels keep
their richer fields (purpose=Commentary, codec_hint with "Atmos",
qualifier=Sdh); MPLS only fills slots the framework left unnamed.
Implementation:
- `fn fill_gaps_from_mpls` walks the MPLS label list, pushing any
entry whose (type, number) tuple isn't already in the framework
output. Stable sort by (type, number) groups audios before
subtitles in the merged result.
- Called from both `extract()` and `analyze()`. Skipped when the
chosen parser is itself `mpls_universal` (no gaps possible).
- `LabelAnalysis::gap_fill_added` field reports how many slots got
filled — useful diagnostic from `labels-analyze`.
- `StreamLabelType` gains `Eq + Hash` so the dedup HashSet works.
Tested via 4 new unit tests (155 of 155 labels tests passing, was
151). End-to-end on partial-yield corpus discs:
- disc-05 (Oppenheimer): pixelogic 4/5 already covered, gap_fill_added=0
- disc-11 (Dune Pt 2): pixelogic 8/11 already covered, gap_fill_added=0
(Real-world gap-fill activations are rare in the current corpus because
pixelogic already incorporates MPLS-equivalent data when matching;
the merge is defensive for less-thorough frameworks.)
Per-playlist stream_number counters were resetting between MPLS
files, so a disc with 2 MPLS files each listing the same 8 audio
streams produced labels with stream_number 1..8 then 1..8 again
(dedup kept whichever PID was different, leaving the numbering
visibly broken — multiple "audio1: eng" rows).
Move the counters outside the per-file loop and increment only
when an entry survives dedup. Surviving entries now get dense
1..N numbering across the whole disc per stream_type.
Verified on corpus disc-02 (HDMV-only): was `audio1, audio1, audio1`
for the 3 distinct audio codecs (TrueHD/AC-3/DTS-HD MA), now
`audio1, audio2, audio3`. Same fix applies to disc-01 (12 audio
streams across multiple MPLS) and disc-09 (14 audio streams).
Three new modules in the labels platform, all layered so framework-
specific parsers (paramount, criterion, pixelogic, ctrm, dbp, deluxe)
always take precedence over the universal layer.
**mpls_universal.rs** (~600 LOC, 9 tests): consumes the already-parsed
`crate::mpls::Playlist::streams` and emits StreamLabel entries with
language + codec_hint per stream. Returns `Confidence::Low` (new
variant) so framework parsers' Medium/High always win the registry's
max-by-confidence tiebreaker; MPLS only gets picked when no framework
matched. Closes the "no BD-J disc" case (HDMV-only navigation) that
previously produced zero labels — language and base codec are
spec-mandated in MPLS STN tables on every Blu-ray ever made.
**bdmt.rs** (~350 LOC, 10 tests): reads `/BDMV/META/DL/bdmt_<lang>.xml`
files into a new `DiscMetadata` struct (localized title names per
ISO 639-2 code, descriptions, optional box-set position). Runs
independently of the parser registry — disc-level metadata, not
per-stream, so the registry's confidence selection doesn't apply.
Surfaced on a new `LabelAnalysis::disc_metadata` field.
**png_filenames.rs** (noop stub): pattern documentation + dead-code
detect/parse for future reactivation. Deferred because MPLS already
delivers per-stream lang/codec/type on every disc; PNG filename
language tokens only add studio variant disambiguation (FRC vs FRP,
LAS vs CSP) — niche enough to not justify the implementation cost
right now.
Wiring changes in `mod.rs`:
- New `Confidence::Low` variant (PartialOrd places it below Medium/High)
- New `ParseResult::low()` constructor
- `mpls_universal` appended last to `PARSERS` registry
- `LabelAnalysis::disc_metadata: Option<DiscMetadata>` field
- `analyze()` runs `bdmt::parse` independently and surfaces result
- `pub use bdmt::DiscMetadata` re-export so the labels-analyze tool
in freemkv-tools can construct the JSON payload
Total: 151 of 151 labels tests passing (was 132 — added 19 new).
Pass N now reads at 32 sectors per attempt and drops to 1 only on
batch-read failure to probe each sector individually. After 16
consecutive clean single-sector reads it climbs back to 32. Net
effect: NonTrimmed regions walk ~32x faster in clean stretches
without sacrificing per-sector recovery quality — the drop-to-1
retry from the same cursor position guarantees every sector in a
failed batch is individually attempted.
Design contract:
- A batch-read failure (count > 1) is NOT a recorded failure: no
NonTrimmed mark, no consecutive_failures bump, no damage_window
push, cursor stays put. We just drop current_batch to 1 and the
loop re-attempts the same position at single-sector granularity.
- A single-sector failure (count == 1) follows the existing path:
NonTrimmed mark, consecutive_failures++, damage_window.push(false),
post-failure pause, wedge probes.
- Backtrack always at count=1: this path fills a gap that the main
loop's damage-window skip jumped over. Using batched reads there
would lump good sectors into NonTrimmed marks when the gap
contains even one bad sector.
State machine adds:
- `initial_batch` (from opts.block_sectors, default 32 in patch_internal)
- `current_batch` (mutable, starts at initial_batch, drops to 1 on
batch failure)
- `consecutive_singles_ok` (counter, resets on upscale + failure)
- `ADAPTIVE_UPSCALE_THRESHOLD = 16` (matches sweep's pattern for
"16 consecutive good = back to fast mode")
Tests:
- pass_n_size_aware_skip.rs PatternedSectorReader now fills each
sector with its OWN LBA byte (not the starting LBA's byte). This
matches real drive behavior — the pre-0.18.13 fixture's
"fill whole batch with one byte" was a shortcut that only worked
when patch read 1 sector at a time. Existing recovery-quality
assertions all still pass under adaptive batching.
User spec: "try 32, pass, great, fail -> do 1 sector"
User design call after watching Pass 2 mark ~20 KB as "Cosmetic"
(permanently Unreadable) after just 10 retries within a single pass:
"i think it's good or maybe until all passes are done. then it's
gone."
That contradicts what the multi-pass design promises a user. The
project goal in project docs is "recover 100% of readable data from any
optical disc, automatically." Marking sectors Unreadable after a
SINGLE pass's per-range retry budget gives up on sectors that
subsequent passes might recover — drive reads are stochastic, the
sector that fails 10 times in Pass 2 may succeed on attempt 1 in
Pass 3 after temperature / bus state / prior-read patterns shift.
The patch.rs doc comment already noted ~36% of patch-marked
Unreadable sectors turned out to be readable in re-rip experiments.
Three sites in `Disc::patch` were emitting `PatchItem::Unreadable`
mid-pass:
- backtrack hit damage (line ~2659)
- all-retries-exhausted on a single LBA (line ~2846)
- redundant second mark after the wedge-suspicion log (line ~2970)
All three now emit `PatchItem::NonTrimmed` instead. Failed bytes
stay "maybe" (NonTrimmed) so the next pass gets another shot. The
per-range skip-limit (10) and per-pass wedge-threshold (50) still
bound time-per-pass; they just no longer turn the bytes terminal.
The `PatchItem::Unreadable` variant stays in the enum (with
#[allow(dead_code)]) because the orchestrator-side end-of-recovery
promotion will use it: autorip, after the final retry pass
completes, scans the mapfile and promotes still-NonTrimmed →
Unreadable. That promotion lands in a follow-up commit on the
autorip side — separable from this libfreemkv change.
Loss accounting unchanged: `bytes_pending + bytes_unreadable` is
the "lost or pending" total that `abort_on_lost_secs` consults
(disc/mod.rs:1327). Moving bytes from one bucket to the other
mid-pass doesn't affect whether the rip would abort; it only
affects display (UI shows "Maybe" vs "Cosmetic") and whether
subsequent passes retry the bytes (the actual fix).
Test update: `test_pass_progress_separates_unreadable_from_pending`
was renamed to `test_pass2_leaves_failed_reads_as_pending_not_unreadable`
and rewritten to assert the new invariant — Pass 2 leaves all
failed bytes as bytes_pending (no mid-pass Unreadable promotion).
Original assertions were checking the pre-design-call behavior.
Precommit (cargo +1.86 fmt + clippy + test) green.
User's design call after watching the avoidance work prevent a wedge
on the live rip (no wedge events across 6 read errors): "Pass N
and 1 should both be very very similar in recovery. almost identical
just smaller sectors imo in pass n. pause times the same imo as a
failed read is a failed read."
The error-handling code path was already centralized (one
handle_read_error fn, called by both Disc::sweep and Disc::patch).
The TUNING was split — Pass 1 used 5 s inter-error pauses + a
wedge-skip-and-continue policy; Pass N used 1 s pauses + immediate
AbortPass on HARDWARE_ERROR / ILLEGAL_REQUEST. That asymmetry made
Pass N vulnerable to the same wedge that Pass 1's avoidance fixed.
Changes:
1. FAIL_PAUSE_SECS = 5 — single constant, applied uniformly to both
passes. Dropped PASS_1_FAIL_PAUSE_SECS and POST_FAILURE_PAUSE_SECS
in favor of one value. CONSECUTIVE_FAIL_LONG_PAUSE_SECS kept as a
distinct (but currently equal) value for future tuning escalation.
2. HARDWARE_ERROR / ILLEGAL_REQUEST path is now symmetric:
- Pass 1: JumpAhead WEDGE_JUMP_SECTORS (1 GB) + WEDGE_PAUSE_SECS
cooldown, mark skipped region NonTrimmed.
- Pass N: JumpAhead WEDGE_PASS_N_SKIP_SECTORS (64 sectors / 128 KB)
+ WEDGE_PAUSE_SECS cooldown. Pass N's batch=1 means a 1 GB skip
would abandon the entire current NonTrimmed range; small skip
moves past the bricked LBA + buffer, outer patch loop picks up
the next sector.
- Both share WEDGE_ABORT_THRESHOLD — same 16-skip budget before
real AbortPass on a permanently stuck drive.
3. wedge_skip / wedge_abort tracing logs now include `pass=1|N`
so post-mortems can see which pass hit the wedge condition.
Cost analysis:
Pre-reframe worry was "5 s × 5500 NonTrimmed sectors per Pass N
pass × 7 passes = 53 hours." Reality: most NonTrimmed sectors
recover on first or second retry, so most reads are successful and
pay 0 pause. The few that DON'T recover hit the 10-skip budget and
get marked Unreadable — bounded at 10 × 5 s = 50 s per truly-bad
sector. Worst-case Pass N pause overhead on a typical damaged disc
is single-digit minutes, not hours. And it's strictly cheaper than
the alternative (wedge kills the entire multi-pass recovery).
Tests:
- `both_passes_pause_on_failed_read_for_wedge_avoidance` — locks the
unified pause-tuning policy (was pass_1_pauses_briefly).
- `pass_n_hardware_error_also_skips_not_aborts` — was
`pass_n_hardware_error_still_aborts`. New behavior verified:
JumpAhead with WEDGE_PASS_N_SKIP_SECTORS + WEDGE_PAUSE_SECS.
- `pass_n_hardware_error_aborts_after_threshold` — new. Confirms
Pass N respects the same WEDGE_ABORT_THRESHOLD as Pass 1.
- pass_1_does_not_pause_on_skip is gone (it was the old "Pass 1
pause=0" assertion, irrelevant after the avoidance work).
Empirical validation: avoidance was already proven on a live rip
tonight — 6 read errors on a damaged disc, sense_family=Medium
throughout, wedge_count=0, Pass 1 continued cleanly past 40%
where it previously died at 48%. This commit extends the same
discipline to Pass N's recovery loop.
Precommit (cargo +1.86 fmt + clippy + test) green.
Replaces the speculative arg-position heuristic with type-presence
detection driven by real disc bytecode. Ground truth captured in
(internal)/research/deluxe-poc/data/ via POC v0.3 binding-
bytecode dumps against disc-01 (Disney) and disc-09 (Warner).
What changed:
1. StackVal::CodingType(String) — new variant. getstatic against
org/bluray/ti/CodingType (the BD-J spec codec enum) now pushes
this, carrying the field name (e.g. DOLBY_LOSSLESS_AUDIO). The
pre-fix code was treating codecs as a Deluxe-internal enum
subclass walk (Phase B), which is the wrong model — codecs are
standard BD-J API references.
2. coding_type_to_codec_hint(field) — new function. Maps
org.bluray.ti.CodingType field names to human-readable codec
strings (DOLBY_LOSSLESS_AUDIO -> "Dolby TrueHD", DOLBY_AC3_AUDIO
-> "Dolby Digital", etc.). Unknown field names pass through
verbatim so future codec values still surface something.
3. find_binding_classes — multi-class variant. Some Deluxe discs
split per-stream tables across two binding classes (audio +
subtitle). Returns top-K candidates by getstatic count, filtered
to >=40% of the top count and capped at 4. Replaces the old
single-class find_binding_class (which was unused after this
change).
4. interpret_streams — rewritten. Args identified by TYPE not
position:
- First EnumRef{kind:"Language"} -> language
- First EnumRef{kind:"Purpose"} -> purpose
- First CodingType(name) -> codec_hint
- First Int(n) -> stream index hint (traced
only; per-type sequential stream_number still wins because BD
spec stream-numbering is anchored on MPLS)
- Construction has CodingType -> Audio stream; otherwise Subtitle
- No Language -> skip (not a stream construction)
This handles BOTH the Disney 5-arg pattern (I, Lbe, Llp, I,
LCodingType) and the Warner 4-arg pattern (I, Law, Lgp,
LCodingType) automatically — same code path because args are
identified by type rather than constructor-signature shape.
5. parse() now walks all binding-class candidates and unions
their constructions before calling interpret_streams. Logs each
candidate at INFO with getstatic_count for diagnosis.
Tests:
- 2 new tests verify the CodingType -> codec_hint mapping for
known + unknown field names.
- Existing interpret_streams tests updated to use the new
signature (dropped CodecTable arg).
- Audio-emission test rewritten to use CodingType arg instead of
the old binding_type substring-match approach.
Confidence is still Medium for now (single-corpus verification);
ready to promote to High once tested against a third Deluxe disc.
Precommit (cargo +1.86 fmt + clippy + test) green.
Wires the existing PassSummary infrastructure (in read_error.rs as
of a832bad) into the sweep loop's exit path. One INFO log line per
Pass 1 completion gives operators an at-a-glance damage profile
without grepping per-error WARN lines:
INFO pass1_summary total_reads_ok=384521 total_errors=5
zones_entered=1 jumps_taken=2
bytes_good=38_725_644_288 bytes_pending=46_GB
copy_elapsed_ms=1751650
Particularly useful for post-mortem analysis when combined with
the per-error structured WARN logs (ms_since_last_error /
ms_since_last_success / sense_family / wedge_transition) shipped
in 0.18.10. Single line tells you the pass shape; preceding WARN
lines tell you the per-error detail.
Pass N (Disc::patch) intentionally NOT covered in this commit —
Pass N has its own retry-budget summary semantics that warrant a
separate design pass. Pass 1 sweep is where wedge incidents
originate, so it gets the diagnostic surface first.
Staged for 0.18.11. 0.18.10 already shipped the per-error WARN
layer; this is the finishing companion log.
Adds the observability we need to debug wedge incidents from logs
alone — without needing to enable verbose TRACE-level SCSI tracing.
Goal stated by user: "when error occurs we can debug and code
correctly."
Pre-fix the WARN log on each read error showed only sense codes
and consecutive_failures. Missing: timing context (was the failed
read fast or slow?), gap to previous events (cumulative vs.
immediate failure?), and family transitions (did the drive just
flip into wedge mode, or has it been there?).
New fields on ReadCtx (no caller signature change):
last_success_at: Option<Instant>
last_error_at: Option<Instant>
last_error_family: Option<SenseFamily>
total_errors: u64
total_reads_ok: u64
zones_entered: u64
jumps_taken: u64
in_damage_zone: bool
New SenseFamily enum (NotReady / Medium / Hardware / IllegalRequest
/ Other) with is_wedge_family predicate.
handle_read_error WARN log now carries:
consecutive_failures
consecutive_outer_failures
ms_since_last_error NEW gap between this and previous error
ms_since_last_success NEW gap to last good read
total_errors NEW aggregate this pass
total_reads_ok NEW
wedge_count
sense_family NEW typed category, easier to filter
sense_key / asc / ascq (existing)
NEW WARN log "wedge_transition" fires once when the sense family
changes from non-wedge to wedge (Medium to Hardware/IllegalRequest).
That's the moment the drive's firmware flipped into fast-fail
mode. Single timestamped event in the log so post-mortems can
pinpoint the transition without scanning thousands of TRACE lines.
Worked example: if the next wedge incident shows
read_error ms_since_last_success=18234 ms_since_last_error=null
read_error ms_since_last_success=28000 ms_since_last_error=10000
read_error ms_since_last_success=43000 ms_since_last_error=68
(drive returned <100ms = wedge symptom)
wedge_transition errors_in_zone=5 ms_since_last_success=43000
we can immediately tell cumulative damage, 5 errors over 43 s,
drive went into fast-fail mode at the 5th. If instead we see
read_error ms_since_last_success=200 ms_since_last_error=null sense_family=Hardware
wedge_transition errors_in_zone=1
the wedge was triggered by ONE read at a physically-bricked LBA
(immediate fast-fail, no warm-up).
These two patterns demand different tuning responses (longer
pause vs. larger initial jump), and now we can distinguish them
from a single WARN log line each instead of needing TRACE
verbose for the whole rip.
Plus jumps_taken / zones_entered counters that feed an end-of-pass
INFO summary (PassSummary). Caller invokes pass_summary at sweep
end and logs structured stats: "Pass 1 saw N errors / M ok reads
/ K zones / J jumps". Single-line post-mortem for any rip.
No caller signature change (timing is internal to the handler;
end-of-pass summary is a new method callers opt into). Precommit
green; 433+ tests pass. Staged for the 0.18.10 release once we
have user-validation data on 0.18.9's avoidance tuning.
Complements the wedge-skip backstop (fbdb50c) with proactive
avoidance so we don't HIT the wedge in the first place. User's
take after seeing the Dune Pt 2 rip wedge at 48%: 'we shouldn't be
wedging.'
Empirical observations from the 23:09:12-23:09:55 wedge timeline:
5 read errors over 43 s, ~8 s apart (drive's own ECC recovery
takes 5-10 s per failure). Not 'hammering' in any usual sense,
but cumulative firmware-state buildup over 5 in-cluster errors
was enough to tip the BU40N into wedge mode at the 5th error.
Damage cluster spanned ~140 MB (LBAs 19.898M-19.965M). Current
damage-jump base of 256 sectors × batch=32 = 16 MB first jump,
doubling to 32 MB, 64 MB... Each jump landed BACK INSIDE the
140 MB cluster, exposing the drive to MORE in-cluster errors.
Two avoidance levers:
1. Inter-error pause on Pass 1 (PASS_1_FAIL_PAUSE_SECS = 5 s):
pre-fix Pass 1 ran pause_secs=0 on all errors to 'zoom past'
damage zones. Successful reads still zoom at zero pause — the
pause applies only to FAILED reads, giving the drive's firmware
cool-down between cluster exposures. Cost: ~5 s per scattered
failure (~30-60 s total on a damage cluster); trivial vs.
crashing the rip at 48%.
2. Larger damage-jump base (JUMP_BASE_SECTORS = 1024, up from
256): first jump at batch=32 now covers 64 MB instead of 16 MB,
second jump 128 MB instead of 32 MB. Two jumps clear 192 MB —
well past most single-cluster damage patterns. Smaller jumps
were landing inside the cluster and adding to the wedge counter.
Plus a halt-aware sleep helper (sleep_secs_or_halt) so the new
inter-error pause doesn't degrade halt response time. Halt poll
granularity 100 ms — halt fires within ~100 ms regardless of
remaining pause time. Updated three sleep call sites in disc/mod.rs
(SkipBlock pause, JumpAhead post-pause, Retry pause).
The wedge-SKIP backstop (fbdb50c) stays — combined with this
avoidance work, the flow becomes:
damage cluster encountered →
pause 5 s, mark NonTrimmed →
second failure →
pause 5 s, mark NonTrimmed →
...
threshold hit →
damage-jump 64 MB (clears 95% of clusters) →
if jump lands in another cluster: 128 MB next jump →
only if drive STILL wedges after all this:
wedge-skip kicks in (1 GB jump + 30 s cooldown × 16 budget)
Tests:
pass_1_pauses_briefly_on_skip_for_wedge_avoidance — locks the
new 5 s pause behavior in place (replaces the old pause=0 test).
integration test threshold bumped from 5 s to 60 s with comment
explaining the new bound is 'not infinite' rather than
'milliseconds-fast'.
All 433+ tests green on cargo +1.86 fmt + clippy + test.
Precommit green.
Closes the final two audit items from this session.
labels::apply_labels: factored out of apply() so the matching logic
is unit-testable without needing a SectorReader / UdfFs. 11 new
tests in apply_tests cover:
- codec_hint + variant flow through to AudioStream.label
- purpose set on audio with no label English text
- name fallback only when purpose=Normal (CLI owns purpose i18n)
- subtitle SDH qualifier set; forced flag flipped on Forced
- per-type 1-based indexing (audio #2 maps to 2nd audio stream,
not 2nd stream overall)
- labels for nonexistent streams are no-ops
- empty labels list leaves streams untouched
- fill_defaults generates audio + video labels; preserves existing
class_reader: robustness smoke tests. ClassFile::parse must NEVER
panic on adversarial input — only return Err. 9 new tests:
- empty input
- short magic (0..4 bytes)
- wrong magic
- truncated after magic
- bad CP tag
- truncated UTF-8 in CP
- 200 random byte buffers (deterministic xorshift)
- 100 magic + random tail (most adversarial — magic check passes,
everything else garbage)
- instructions iter on random code (200 buffers)
- instruction_size on every opcode 0..255 with varied tail buffers
- modified_utf8 on random byte buffers (500)
The xorshift PRNG keeps the tests deterministic (no rand dep) and
reproducible — failures will be the same buffer every time. This is
the lightweight alternative to a cargo-fuzz setup; if/when we adopt
cargo-fuzz, these tests stay as regression cases.
All 451 tests passing on cargo +1.86 fmt + clippy + test.
Pre-fix: when the drive returned HARDWARE_ERROR or ILLEGAL_REQUEST
during Pass 1 sweep, libfreemkv immediately returned ReadAction::
AbortPass. Autorip surfaced this as a fatal error and stopped the
rip at whatever progress percentage Pass 1 had reached — typically
40-50%. On a disc with one physical-damage cluster, the user would
see Pass 1 die at ~48% with the cryptic message 'E6000: <lba>
0x02/0x04/0x3e' and have no rip output to work with.
Root cause analysis: BU40N firmware transitions into a fast-fail
state when it hits cumulative read failures in a small LBA range —
returns HARDWARE_ERROR for every subsequent read near that LBA, even
sectors that aren't physically damaged. Per project docs 'Bad-sector
handling' rule #2, 'Recovery requires eject+reload OR significant
cool-down.' Aborting on first wedge throws away the rest of the
disc; the right response is to SKIP the wedged region (mark as
NonTrimmed for Pass N), pause for drive cooldown, and continue.
Fix: in handle_read_error, the HARDWARE_ERROR / ILLEGAL_REQUEST arm
now branches on bisect_on_marginal:
Pass 1 (bisect_on_marginal=false): JumpAhead with WEDGE_JUMP_SECTORS
(1 GB at 2048 bytes/sector) and WEDGE_PAUSE_SECS (30 s cooldown).
Tracks wedge_count in ReadCtx; resets on any successful read.
Truly aborts only after WEDGE_ABORT_THRESHOLD (16) consecutive
wedges with no good read in between — generous enough to clear
most physical-damage clusters, bounded enough to not loop forever
on a permanently bricked drive.
Pass N (bisect_on_marginal=true): unchanged AbortPass. Pass N's
job is single-sector recovery; if the drive won't talk near a
specific LBA, skipping doesn't help. Pass N exits and lets the
outer layer decide retry/eject/surface.
5 unit tests cover the new policy:
pass_1_hardware_error_jumps_ahead_not_aborts — JumpAhead emitted
with correct sectors+pause, wedge_count incremented.
pass_1_hardware_error_aborts_after_threshold — AbortPass kicks in
on the WEDGE_ABORT_THRESHOLD-th consecutive wedge.
pass_1_good_read_resets_wedge_count — on_success clears
wedge_count; subsequent wedge gets fresh skip budget.
pass_n_hardware_error_still_aborts — Pass N's AbortPass behavior
intact.
pass_1_illegal_request_also_routes_to_wedge_skip — both wedge
sense families get the skip treatment.
Impact: on the Dune Pt 2 disc that consistently wedged at 48%
(physical damage at LBA ~19.9M), Pass 1 will now jump ahead 1 GB
on the wedge, give the drive 30 s cooldown, and continue scanning
the rest of the disc. The damaged region becomes Pass N's job to
revisit. Worst case if the drive stays wedged: 16 GB of NonTrimmed
disc area before honest AbortPass.
Precommit (cargo +1.86 fmt + clippy + test) green; 430 passing.
Replaces two near-duplicate hand-rolled XML scrapers in paramount.rs
and criterion.rs with a single labels::xml module that's robust to:
- Case-insensitive tag / attribute names ('<Playlist>' matches the
same as '<playlist>'; 'Name=...' matches 'name=...').
- XML namespace prefixes (matches '<ns:tag>' for tag='tag').
- Arbitrary whitespace inside open tags and around '=' separators
('<tag name = "X">' works).
- Both quote styles for attribute values (" and ').
- Self-closing tag forms ('<tag/>' and '<tag />').
- '>' chars inside quoted attribute values (no premature end-of-tag).
Three functions:
xml::attr(element, name) -> Option<String>
Extract attribute value from an open-tag fragment.
xml::text(xml, tag) -> Option<String>
Trimmed text content of first <tag>...</tag>.
xml::find_element(xml, tag, from) -> Option<(start, end)>
Locate next <tag>...</tag> for iteration; handles self-closing.
22 unit tests cover the robustness properties: case-insensitivity,
namespace stripping, whitespace tolerance, quote styles, self-close
forms, no-substring-false-positive (looking for 'lang' must NOT
match 'lang_id' or 'language'), '>' inside quoted attrs, iteration
across repeated elements.
paramount.rs: drops local extract_attr; find_feature_playlist now
walks xml::find_element('playlist', ...) so it works regardless of
case and self-closing style. Pre-refactor: required exactly
'<playlist ' (single space, exact case) and '/>' for self-close.
criterion.rs: drops local extract_tag; parse_stream_infos and
parse_playback_config iterate via xml::find_element. Same case-
sensitivity + namespace gains. The 'COMMENTARY' / 'SDH' / 'DS'
content-value match is now case-insensitive too (previously a disc
authored with 'commentary' would have been miscategorized as Normal).
Pre-refactor known failure modes (none observed yet, but trivial
to trip on a future disc): vendor switches whitespace around '=',
uses single quotes, capitalizes a tag, prefixes a namespace. All
now handled.
Out of scope by design: XML entity decoding (&, <), CDATA
sections, comments, processing instructions. None observed in BD-J
authored label data. If a future disc trips them, the entity
decoder is a localized addition.
Precommit (cargo +1.86 fmt + clippy + test) green.
Completes the Deluxe parser pipeline. Phase A (master enums) was
already shipping; this commit lands Phases B/C/D so the parser now
emits per-stream StreamLabel records on Deluxe-authored discs.
Phase B (decode_codec_enum): walks the codec enum's subclass
references (one .class per codec ordinal) and extracts the codec
name string from each subclass's constant pool. Heuristic: pick the
first Utf8 entry that's uppercase + underscored + >=4 chars, or one
of the known codec roots (ATMOS/DOLBY/DTS/TRUEHD/MLP/AC3/EAC3/PCM)
when no underscored candidate is found. CodecTable maps ordinal ->
codec string; empty string for ordinals where extraction failed
(logged via tracing, not fatal).
Phase C (find_binding_class): identifies the class that builds the
per-stream label table by counting getstatic operations targeting
any of the master enum classes from Phase A. Class with the highest
count >= 4 wins. Threshold is empirical (real binding classes have
50+ matches; floor of 4 admits small discs while rejecting incidental
single-reference classes).
Phase D (decode_binding + BindingDecoder): symbolic stack machine
that walks the binding class's <clinit> bytecode. Handles:
- constant pushes: iconst_<n>/bipush/sipush/ldc(Integer)
- new <X>: pushes uninit-object marker
- dup: stack copy
- getstatic <Y.Z>: pushes EnumRef when Y is in MasterEnumTable,
else Unknown
- invokespecial X.<init>(...)V: pops args per descriptor; when the
receiver is NewObj(X), emits a Construction { binding_type: X,
args: [...] }
- invokevirtual/invokestatic/invokeinterface: pop args per
descriptor, push return placeholder unless void
- pop/pop2/aastore/putstatic/putfield: standard stack effects
- branches/returns: clear stack (conservative resync — binding
<clinit> is straight-line in practice)
parse_method_arg_count: JVMS field-descriptor parser, handles
primitives, references (L...;), arrays ([...).
interpret_streams: converts Constructions to StreamLabels using
the master enum table + CodecTable. Each construction with a
Language ref becomes a stream. Audio when codec_hint resolves via
binding_type substring match against CodecTable; subtitle otherwise.
Purpose ordinal -> LabelPurpose via the verified Deluxe Purpose enum
order (Normal/Commentary/PiP/Trivia/Descriptive/Score/NoForced/
NoForcedDescriptive). Stream index = sequential per type. Language
goes through vocab::lang for ISO code + variant.
deluxe::parse now returns Some(ParseResult::medium(labels)) when
all four phases produce labels. Medium confidence — the bytecode
mechanism is rigorously tested but the signal-to-StreamLabel
mapping (which arg is which, audio vs subtitle classification) is
heuristic until corpus binding-class bytecode confirms the exact
pattern.
Test coverage: 13 new unit tests in deluxe.rs
parse_method_arg_count: 3 tests (basic types, references, malformed)
BindingDecoder: 4 tests (simple construction, with int pushes,
skips unmatched invokespecial, resolves master-enum ordinal)
interpret_streams: 4 tests (subtitle on no codec, audio on codec
match, purpose routing, skips no-language)
MasterEnumTable: 3 tests (resolve, value, class_name_set)
extract_codec_name: 1 test (uppercase+underscore matching)
class_reader.rs gained a #[cfg(test)] ConstantPool::from_entries
test-only constructor so Phase D tests can build synthetic CP
fixtures without writing raw .class bytes.
Precommit (cargo +1.86 fmt + clippy + test) green.
Replaces 'first-match-wins by array order' with 'highest-confidence-
wins, array order tiebreaker'. Removes the arbitrariness when more
than one parser can claim a disc (e.g. one with both
bluray_project.bin and playlists.xml).
New types in labels::mod:
pub enum Confidence { Medium, High }
pub struct ParseResult { labels: Vec<StreamLabel>, confidence }
ParseResult::high(labels) / ::medium(labels) constructors
Parser signature change: every parse() now returns
Option<ParseResult> instead of Option<Vec<StreamLabel>>. Updated all
six parsers in lockstep:
paramount: High (fully structured XML)
criterion: High (fully structured XML)
pixelogic: High by default, Medium when an unknown token component
is encountered (the skip-unknown path now propagates the
coverage gap to the caller instead of silently degrading)
ctrm: High (structured key-value)
dbp: High (anchor scan with vocab routing)
deluxe: still returns None pending Phase D — signature aligned
Registry behavior:
extract() iterates all detect-positive parsers, picks highest
Confidence with non-empty labels. Equal confidence falls to array
order (deterministic). Same selection logic in analyze().
LabelAnalysis grew a confidence: Option<Confidence> field so the
diagnostic surface (freemkv-tools labels-analyze) exposes which
confidence tier the selected parser claimed. labels-analyze JSON
and labels-corpus-check structural diff both gained the field.
Precommit (cargo +1.86 fmt + clippy + test) green.
Three targeted fixes from a second-pass audit of the labels module.
1. vocab::lang now returns Option<LangInfo> with both code AND a
human-readable variant string. Pre-fix: 'Brazilian Portuguese 5.1'
became language=por, variant='', dropping the dialect info the
disc had explicitly authored. Post-fix: language=por,
variant='Brazilian' — matches the convention pixelogic / ctrm /
criterion already use for their region variants. dbp now
populates StreamLabel::variant from this. Compound table grew a
3-tuple (needle, code, variant); bare matches still return
variant=''.
2. dbp and deluxe had duplicated detect() boilerplate (any top-level
.jar in /BDMV/JAR/). Both now call jar::has_any_top_level_jar.
The trait-level detect contract — see super::PARSERS — can't peek
inside a jar without a SectorReader, so loose-detect-plus-real-
check-in-parse is the unavoidable pattern for jar-content parsers.
Consolidating in jar.rs at least makes the duplication visible.
3. mod.rs comment about parser ordering said 'dbp last'; deluxe is
actually now last. Updated to explain the dbp-before-deluxe order
is by cost (cp-iteration cheaper than bytecode walking when Phase
D lands).
Plus a registry-level lock test in mod.rs::registry_tests — asserts
the PARSERS array order is exactly [paramount, criterion, pixelogic,
ctrm, dbp, deluxe]. This was previously implicit; if someone reorders
the array (which changes which parser wins on overlapping signals),
unit tests would have stayed green. Now they fail with an explanatory
message about why the order matters.
Audit findings deferred to follow-ups (each its own commit + design
discussion):
- Stronger detect contract — current loose-detect-real-check pattern
is forced by SectorReader-not-in-detect-signature; could be fixed
by changing the trait to take an Option<&mut dyn SectorReader> or
similar.
- Per-parser confidence scoring — registry currently first-match-wins.
A high-confidence parser ought to beat a low-confidence one
regardless of array order.
- class_reader fuzzing — handles malformed input via Result but no
adversarial corpus yet.
Precommit (cargo +1.86 fmt + clippy + test) green.
Closes the platform unification: every label parser now routes
purpose/qualifier/codec classification through one source of truth
(vocab.rs) instead of N hand-rolls, and every binary-blob byte
scanner goes through one helper (text::extract_ascii_strings).
pixelogic.rs:
- Drop local extract_strings (~20 lines) — use text::extract_ascii_strings.
- HARDENING: replace with skip-unknown-component +
trace log. Pre-refactor behavior: any single uncatalogued token part
(e.g. a future codec ID, new framework variant) silently dropped the
entire stream record. New behavior: skip just the unknown part,
surface what we know about the stream.
- 8 new unit tests cover basic audio/subtitle paths, commentary,
descriptive, region variant, the new skip-unknown-component
regression, and the non-audio/non-subtitle early-out.
ctrm.rs:
- Replace with
vocab::purpose(&name). Now word-boundary matched — 'Commenter Pro
Track' no longer false-matches Commentary.
- Replace with vocab::qualifier(&name).
Same word-boundary tightening, plus picks up Forced and
DescriptiveService for free.
- Preserved structural commentary signal via
as a fallback when name is silent (e.g. 'audio_commentary_1.name=Track 2').
- 6 new unit tests including the 'Commenter' false-positive regression
and the SDH-only-on-subtitles boundary.
text.rs:
- Drop module-level dead_code allow now that pixelogic uses
extract_ascii_strings.
Net: all 5 framework parsers now on the unified platform. Future work
(deluxe Phase D, paramount/criterion XML hardening) builds on the
same scaffolding.
Precommit green.
Closes the 'detected but no parser' gap on Deluxe-authored BD-J discs
(com/bydeluxe/ package signature; ~20% of UHD discs in our corpus per
the 2026-05-10 11-disc capture session).
What ships in this commit:
- detect() — registers the parser in the chain (loose pre-check at the
/BDMV/JAR/ level; real signal in parse via has_path_prefix on
'com/bydeluxe/').
- Phase A master enum identification — walks every .class's <clinit>
ldc sequence and matches against framework-stable fingerprints for:
Language (70 ldcs starting English/French/Spanish/Dutch)
Purpose (8 ldcs starting Normal/Commentary/PiP/Trivia)
VideoFormat (7 ldcs starting HD/HDR10 Plus/HD Dolby)
Region (22 ldcs starting USA_D1/LIC1/LIC2/LIC3 — Disney-only)
Studio (6 ldcs starting Disney/Marvel/Pixar — Disney-only)
All identifications verified out-of-tree on corpus disc-01 (Disney,
The Amateur) and disc-09 (Warner, Dune Part 1) via the standalone
POC.
- Phase B structural skeleton (find_codec_enum) — identifies the
codec enum class by structural signature (>=20 'new' ops, 0 ldcs),
returns the ordered subclass list. Codec string extraction from
subclasses is dead-coded pending the follow-up commit.
What does NOT ship yet:
- Phase D (per-stream binding-class decoder). parse() returns None
intentionally — the master enums alone don't yield StreamLabels
without the streamTable.put(...) bytecode walker. analyze() will
show 'deluxe' in parsers_detected with the enum identification in
tracing logs, so the analyzer reports honestly: 'detected, can't
emit labels yet' rather than silent failure.
Why ship A without D: A is proven on real corpus discs; D's design
needs ground-truth binding bytecode from at least 2 corpus discs
side-by-side to verify the stack-machine pattern. The Phase A
infrastructure (master enum identification + ordinal->name table)
is what D will consume — landing it now unblocks D's design without
holding back the parser registration.
5 unit tests cover the fingerprint matcher logic + a roster lock that
forces explicit consideration when adding/removing fingerprints.
Precommit green.
Establishes the shared infrastructure layer for label parsers so that
Java-touching parsers (dbp, deluxe) don't reimplement jar walking and
all parsers route language/purpose/qualifier classification through
one source of truth instead of N hand-rolls.
New modules:
vocab.rs expanded from 27 -> ~370 lines
+ lang(text) -> Option<&'static str> (English/multi-word
-> ISO 639-2; ~45
languages, compound
phrases like
'Brazilian Portuguese'
and 'Castilian Spanish')
+ purpose(text) -> LabelPurpose (Commentary,
Descriptive, Score,
Ime; word-boundary
matched)
+ qualifier(text) -> LabelQualifier (SDH, Forced,
DescriptiveService)
+ has_word internal primitive — enforces word-boundary
matching so 'Commenter' no longer matches 'commentary' and
'engineering' no longer matches 'english'. Existing parsers
used .contains() and got lucky on the corpus; vocab now
guarantees the boundary in one place. 20+ unit tests.
text.rs NEW (~85 lines)
+ extract_ascii_strings(data, min_len) — promoted from two
near-duplicate copies (pixelogic min=4, dbp min=5);
threshold passed in. 7 unit tests including
trailing-without-terminator + high-bit-byte handling.
jar.rs NEW (~120 lines)
+ for_each_jar(reader, udf, fn) — walk every top-level
.jar under /BDMV/JAR/,
yield to callback.
+ has_path_prefix(archive, prefix) — cheap 'is this MY
framework's jar?' check
via central-dir filenames.
+ for_each_class(archive, fn) — parse every .class entry
through class_reader,
yield (name, &ClassFile).
+ try_each_class(archive, fn) — same with early-return on
first Some(R) match.
Refactored:
dbp.rs v2 on the new platform:
- dropped extract_printable raw byte scan
- dropped its own English -> ISO 639-2 map
- dropped its own parse_attributes hand-roll
+ iterates CpInfo::Utf8 via class_reader (structurally clean,
no false-positive risk from method bytecode bytes)
+ routes language/purpose/qualifier through vocab
All 7 prior dbp tests still pass; +2 new ones cover
vocab routing.
dead-code allows on text.rs (extract_ascii_strings) and jar.rs
(try_each_class) come off when pixelogic and deluxe land — they're
staged for next steps.
Precommit green (cargo +1.86 fmt + clippy + test).
Foundation for label parsers that need structured access to .class
files inside /BDMV/JAR/<x>.jar. Replaces noak (~3KLOC dep) with a
~1000-line std-only reader.
Public API:
- ClassFile::parse(&[u8]) -> Result<ClassFile>
- ConstantPool::{get, utf8, class_name, string, integer, member_ref, iter}
- Member::code(&pool) -> Option<CodeAttribute>
- CodeAttribute::instructions() -> Instructions iterator
- Instruction::{name, operand_u8, operand_u16, cp_index}
- Opcode constants (LDC, AASTORE, NEW, GETSTATIC, INVOKESPECIAL, ...)
Spec coverage:
- Constant pool: all 17 tag types incl. Long/Double 2-slot quirk
- Modified UTF-8 incl. 0xC0 0x80 -> U+0000 special case
- Bytecode iteration with full opcode size table
- Variable-length tableswitch / lookupswitch / wide
12 unit tests cover the opcode table edge cases (padded switch tables,
wide-iinc 6-byte form), modified-UTF-8 decoder, and iterator
stop-on-truncated behaviour.
Module is currently #![allow(dead_code)] — the public API is staged
for labels::deluxe (Phases A-E bytecode walker) and a labels::dbp
refactor onto the constant-pool iterator. Tests exercise the API
in isolation. The allow comes off as those callers land.
Also fixes two pre-existing clippy lints that 1.86's stricter checks
flagged after I touched the labels module:
- src/mux/disc.rs: while-let-loop in test fixture
- tests/pass_n_size_aware_skip.rs: type_complexity in helper signature
Precommit (cargo +1.86 fmt + clippy + test) green.
5th BD-J authoring framework recognized. Discriminator: any top-
level .jar in /BDMV/JAR/ that contains com/dbp/ package paths.
Identified during the 2026-05-10 corpus session via string-mining
disc-07's BD-J jar — perm files reference bd-live.magpictures.com
(Magnolia / Magnet Releasing).
Stream labels live as plain ASCII strings inside compiled .class
files in the form
LTextField,Audio1,English Dolby Atmos,Fontstrip_Composite,...
HTextField,Subtitle1,English SDH,Fontstrip_Composite,...
ATextField,Subtitle0,None,Fontstrip_Composite,...
— a quirk of the menu-rendering layer encoding TextField positions
and content as constant strings the Java compiler retained in the
class string pool. The leading single-letter prefix is string-pool
ordering noise; parser anchors on `TextField,`. Subtitle0 is the
disable-subtitles button and is skipped.
The parser:
- reads each top-level .jar via udf.read_file
- opens it with the existing zip dependency
- confirms com/dbp/ presence in the central directory
- walks .class entries, extracts printable strings, matches the
`TextField,(Audio|Subtitle)<N>,<label>,...` pattern
- maps human-readable language names ("English", "Castilian
Spanish", "Brazilian Portuguese", "Canadian French", ...) to
ISO 639-2 codes via a parser-local table (per the rules-of-
engagement memo, each parser knows its own format)
- preserves the full disc-authored label string in `name` so
consumers display it raw without the lib guessing further
structure
- detects SDH / Forced qualifiers and Commentary / Descriptive
purposes from substring matches; everything else falls through
to fill_defaults using BD-spec MPLS data
Verified live on the corpus: disc-07 (Civil War UHD) now matches
parser=dbp with 3 audio + 2 subtitle labels, exactly the count
visible in the disc's authored TextField definitions and what BD
spec MPLS reports.
Limitation: dbp's detect() returns true for ANY top-level .jar in
/BDMV/JAR/ (every BD-J disc has one), since the discriminator
trait function takes only `&UdfFs` and can't read jar contents.
parse() does the real com/dbp/ check — a non-dbp disc gets
parsed-as-dbp, archive_has_dbp returns false, parse() returns
None, and we fall through. Diagnostic noise: parsers_detected
includes "dbp" on non-dbp BD-J discs. Real fix is refactoring the
DetectFn signature to take a SectorReader; deferred.
7 unit tests cover the TextField extraction, language detection
(simple + compound: "Brazilian Portuguese", "Castilian Spanish",
"Canadian French", "Latin American Spanish", "Australian English"
plus the disc-corpus typo "Austrailian English"), SDH/Forced/RNIB
qualifier detection, and Commentary/Descriptive purpose detection.
Don't-guess discipline preserved: unknown languages return ""
(consumer falls back to MPLS spec data via fill_defaults).
Pass 1 sweep was grinding through damage zones because the marginal-
media handler returned `Bisect` for every failed 32-sector batch —
forcing 32 single-sector reads per bad block at ~5s each on a real
BU40N-vs-Dune-Pt-2 trace. AND the JumpAhead trigger required a 16-
block damage window to fill before firing, so entry into a
contiguous damage zone took ~40 minutes of grinding before the
first jump fired. Architecturally wrong: Pass 1's job is "fast and
accurate, get the most data in the shortest time." Bisection +
recovery is Pass N's purpose-built role.
ReadCtx now carries two new fields:
- `consecutive_outer_failures: u64` — outer-batch failures since
last outer success. Bisect inner failures don't count.
- `bisect_on_marginal: bool` — whether to return Bisect on a
marginal-media batch failure.
- `fast_jump_threshold: u64` — outer-failures count that triggers
JumpAhead before the damage window has filled.
`for_sweep` (Pass 1) sets `bisect_on_marginal=false`,
`fast_jump_threshold=4`, and zeroes the post-failure pause. Failed
batches become SkipBlock → whole block NonTrimmed → advance, no
sleep. After 4 consecutive outer failures: JumpAhead with the
existing escalating multiplier.
`for_patch` (Pass N) sets `bisect_on_marginal=true`,
`fast_jump_threshold=u64::MAX`, keeps the original cooldown pauses.
Pass N's whole reason to exist is to grind on bad ranges with
proper recovery semantics — single-sector reads, 60s recovery
timeout, retry budget, escalating skip — and that's unchanged.
`on_success` resets `consecutive_outer_failures` only when not
bisecting, so a good single-sector read inside Pass N's bisect
doesn't pretend we've escaped the damaged batch.
Tests:
- `pass_n_marginal_with_batch_gt_1_bisects` — Pass N still bisects.
- `pass_1_marginal_skips_instead_of_bisecting` — Pass 1 doesn't.
- `pass_1_jumps_after_4_consecutive_outer_failures` — fast-entry.
- `pass_n_does_not_fast_jump` — fast-entry is Pass-1-only.
- `outer_success_resets_consecutive_outer_failures` — counter reset.
- `bisect_inner_success_does_not_reset_outer_counter` — semantics.
- `pass_1_does_not_pause_on_skip` — explicit zero-pause contract.
- `long_failure_streak_extends_pause_on_pass_n` — Pass N still
extends pauses on long failure streaks (renamed from the old
sweep-based test).
Integration test `test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed`
updated: it used to assert Pass 1 recovers all sectors via bisect
(bytes_good=total). New contract: Pass 1 marks NonTrimmed; Pass N
recovers. Test now asserts Pass-1-only outcome (bytes_pending=total,
complete=false) consistent with the redesign.
Real-world impact on the user's BU40N + Dune Pt 2 trace from this
session: a damage zone that was on track to take ~40 minutes of
Pass-1 grinding will now jump in ~20 seconds. Pass N still has the
full 7-pass recovery budget to revisit those NonTrimmed ranges.
Adds `parsers_detected: Vec<&'static str>` to `LabelAnalysis`. Records
every parser whose discriminator matched, regardless of whether its
parse() then returned Some/None.
Why: when `parser=None` we currently can't tell apart:
(a) no parser recognized this disc — missing parser, candidate for a
new module
(b) a parser recognized it but parse() returned None / empty —
capture truncated, or genuine empty authoring data, or a parser
bug
Surfaced concretely on the 11-disc capture session 2026-05-10:
disc-04 had `bluray_project.bin` in jar_inventory (pixelogic detect()
returned true) but parse() returned None because file content was
past the 1 GB capture window. Old API: parser=None — looked like
"missing parser." New API: detected=[pixelogic], parser=None — clearly
"capture problem, not a parser gap."
The tracing log line on the no-parser-emitted-labels path now
distinguishes the two cases too.
No behavior change to production rip path. extract() is unchanged;
only the diagnostic analyze() returns the richer result.
Promotes `mod labels` to `pub mod labels` and adds `analyze()` plus
`LabelAnalysis` (both `#[doc(hidden)]`) so an out-of-tree diagnostic
binary (freemkv-tools labels-analyze) can introspect which BD-J parser
matched a given disc, what JAR files the discriminators saw, and what
labels came out — without going through the production `apply()` path
that mutates DiscTitles.
Also adds `tracing::info!(parser = name, "label parser matched")` /
"no label parser matched" inside `extract()`. Dev-only signal: users
get the same seamless behavior; developers can finally tell whether a
disc hit a real parser or fell through to the codec-name fallback in
fill_defaults().
The new `jar_inventory()` helper deduplicates and sorts filenames
under any `/BDMV/JAR/<x>/` subdirectory — same plumbing the existing
`jar_file_exists()` discriminators use, just enumerated rather than
predicate-tested. Used by `analyze()` to surface unrecognized
parser-source files when no parser matches, which is the input to
"do we need a new parser?" triage.
No behavior change to the production label path. `apply()` and
`extract()` remain functionally identical; the new public surface
exists alongside.
libfreemkv is a library — Cargo.lock is gitignored (standard for
libs). --locked refuses to create a lockfile on a fresh runner,
so it always fails CI. --locked stays in the binary crates
(freemkv, autorip, bdemu) which DO track Cargo.lock and benefit
from the dependency-race hard-fail behaviour.
The 0.18.4 commit landed with rustfmt diffs in the new
canonical_order tests because my local validation script piped
'cargo fmt --check' to 'tail -1', masking the diff output and
reporting green when fmt was actually unhappy. CI's lint job
caught it immediately. No code change — pure formatting.
Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.
Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips ← impossible
Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip ← the movie
92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.
New canonical_title_order:
1. Real titles (size_bytes <= capacity_bytes) before virtual
composites — capacity gate is hard physical truth.
2. Among real titles, fewer clips first (1-clip wins as the
canonical main feature; multi-clip is either chapter-stitched
or composite).
3. Tiebreak on longer duration first.
Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
already the movie.
- Branching UHDs: virtual composite drops to the back, the real
movie surfaces at index 0.
Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.
Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.
Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.
This commit is purely doc-side:
- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
with caller-orchestrated multipass; Streams table footnote and
Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
trait splits, deprecations (kept alive through 0.18.x, deletion
target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
drive-access,udf}.md: every Disc::copy / pes::Stream /
SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
README.md and CHANGELOG.md now, matching the workspace-top
FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.
No code changes.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.
- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
at crate root.
Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.
See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.
Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.
See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
existing call sites — sweep producer and DiscStream demux
Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:
- Disc::sweep (disc/mod.rs): producer wraps the input reader
in DecryptingSectorSource::new(reader, keys) before the read loop.
The inline decrypt_sectors call goes away — read_sectors yields
plaintext directly.
- DiscStream (mux/disc.rs): constructor wraps the underlying
Box<dyn SectorReader> in DecryptingSectorSource so the internal
fill_extents / read path sees plaintext bytes. The DecryptKeys
field stays on DiscStream for metadata-side use; it just no
longer drives decryption.
Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.
Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
&mut dyn SectorReader and Box<dyn SectorReader>, mirroring
std's Read forwarding pattern. Generic blankets would conflict
with the existing SectorReader → SectorSource blanket under the
orphan rule (downstream could impl SectorReader for &mut U), so
the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
DiscStream::set_raw() can flip the wrapped reader to a
DecryptKeys::None pass-through without rebuilding the decorator
(which would require moving the inner Box out from behind &mut self).
After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.
Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.
- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
carries the file-write + mapfile.record per WorkItem; close()
drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
(so the consumer thread keeps showing up as
freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
The producer-side ReadCtx state machine, decrypt, set_speed,
halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
preserve identifiable thread names without the primitive baking
one in. Also adds Pipeline::try_send for the throttled
StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
module-private types. WorkItem::Finish dropped — dropping the
channel is the end-of-stream signal Pipeline already uses.
Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.
See (internal)/memory/0_18_redesign.md.
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.
- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
WritebackFile + Mapfile. apply() seeks+writes recovered bytes
and records mapfile state per item; close() runs sync_all and
mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
buffering — back-pressure should kick in immediately so the
drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
producer (reverse walk, damage-window skip, NOT_READY pauses,
bridge-degradation handling, wedge exit, range watchdog).
WritebackFile ownership moves to the sink.
Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.
No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.
Single contributor: MattJackson.
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.
The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:
- Audits DiscStream's interior types for Send (its Box<dyn
SectorReader> already requires Send via the trait's super-bound;
verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
over a DiscStream, reads frames through the trait object, and
exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
if call-site ergonomics demand it; otherwise relies on the
blanket.
No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:
- src/halt.rs: clippy::new_without_default fires when a public new()
exists without Default. The polish pass dropped the derive thinking
it was redundant — clippy disagrees, so add a manual impl that
forwards to new(). Doc-comment notes why both exist.
- src/disc/read_error.rs:372: pre-existing
assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
precommits passed because that lint sat outside the gate; the
round-2 commits brought enough new clippy surface that it now
shows up. Trivial cleanup: assert!(...) instead of assert_eq!.
Single contributor: MattJackson.
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.
The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.
FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.
Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.
Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).
The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.
FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.
Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.
See (internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
(internal)/memory/0_18_redesign.md for full context.
Single contributor: MattJackson.
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.
Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.
No alias kept; this is a clean 0.18 rename. See
(internal)/memory/0_18_redesign.md.
Single contributor: MattJackson.
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
(internal)/memory/0_18_redesign.md.
Single contributor: MattJackson (no attribution trailers).
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.
This release plugs those gaps:
- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
Writer before any seek / write. sync_all on Writer cleanly drains
the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
of sequential output; the page cache no longer absorbs that as a
single hot blast on slow targets.
Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.
Mapfile now batches the rename to once per second:
- record() always updates in-memory state and stats; only fires
write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
consumer at end-of-sweep and by Disc::patch at end-of-patch,
after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
silently lose pending state.
Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.
Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)
Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
Pre-0.17.11 sweep ran strictly serialised: SCSI read → decrypt → seek
+ write → mapfile.record → next read. Drive idled for the post-read
work; throughput capped at the sum of both costs. On a healthy disc
that's ~7-12 ms read + ~5-15 ms write/record per 64 KB batch, limiting
sustained throughput to ~10-12 MB/s on the test bed (BU40N + UHD inner
zone), well below the ~14-16 MB/s drive ceiling.
Decouples them: producer thread (caller's) owns SectorReader +
read_error state + decrypt + set_speed + halt; consumer thread (one
spawn) owns Writer + Mapfile, receives WorkItem messages, applies
file write + mapfile record. Bounded mpsc::sync_channel(4) gives
natural back-pressure. While the consumer writes batch N, the
producer is already reading batch N+1 — steady-state throughput is
now bound by the slower of the two pipelines (drive on healthy
discs), not their sum.
Side effects:
- Bisect path now decrypts. Pre-0.17.11 the bisect inner loop wrote
raw cyphertext for single-sector recoveries on encrypted discs —
quiet correctness bug exercised only by batch-fail-then-
bisect-succeed on encrypted media. New producer-side decrypt
covers main + bisect success paths uniformly.
- All read_ctx state stays single-threaded on producer (damage
window, jump multiplier, etc.). No locking added.
- Mapfile remains single-writer on consumer. No locking.
- Halt latency: producer breaks loop, sends Finish, consumer drains
≤4 in-flight items + sync_all. ~1 batch (~12 ms) typical.
- BU40N + Initio bridge wedge concern unchanged: still single SCSI
command in flight, error-path timing identical, no new retries.
New module: src/disc/sweep_pipeline.rs (WorkItem, ProgressSnapshot,
ConsumerInputs, spawn_consumer, consumer_loop, helpers). Public API
unchanged — Disc::copy / CopyOptions / CopyResult identical.
Patch (Pass N) is NOT changed; it's bound by drive recovery time, not
the read/write serialisation.
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.
- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
Wraps a per-platform WritebackPipeline that on Linux schedules
sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
mux) can adopt the same wrapper as a one-line change later.
Version bump to keep the four freemkv crates at unified versioning
after autorip's v0.17.6 + v0.17.7 work today. No libfreemkv code
changes; republished to crates.io so downstream consumers stay
aligned on the latest patch version.
Direct-SATA BU40N + Dune Part Two UHD live testing exposed that the
v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-
sector LBAs that dd if=/dev/sr0 recovers on the same drive. This
release closes that gap and fixes adjacent bugs silently capping
recovery.
- /dev/sr0 pread fallback in Drive::read (Linux only): on SCSI READ
Err, fall back to posix_fadvise(DONTNEED) + pread() against the
corresponding block device. Kernel sr_mod runs ~5 internal retries
with no per-attempt mid-layer escalation overhead — the mechanism
behind dd's recovery advantage. End-to-end byte verification
confirms the fallback path returns real disc data.
- Disc::patch per-range watchdog fix: MAX_RANGE_SECS was breaking
'outer (one slow range killed the entire patch). Now skips to the
next range. Pre-fix patch died after 4 sectors of range 1 of 47.
- Per-sector range budget: range_budget = sectors × 25 s, capped at
1800 s. Replaces the flat 180 s/range that was unfair to medium
ranges and pointlessly generous to single-sector ones.
- consecutive_failures resets per range. The wedge-exit detector is
for stuck-on-one-range, not many-small-ranges-with-one-fail-each.
- Reverted inline 5× retry experiment (was hurting: each retry paid
kernel SCSI escalation overhead). Restored READ_RECOVERY_TIMEOUT_MS
to 60 s. The kernel-auto-retry pattern is now provided by sr0
fallback.
Empirical: pass 1 recovered 94.6 MB / 11 s of main title (33 sr0
saves). Pass 2 added 0.6 MB. Remaining ~233 MB on the test disc
appears physically unrecoverable on this hardware.
tests/scsi_recovery.rs:
- Add `use std::time::Duration` inside both `#[cfg(target_os = "linux")]`
blocks. Locally on macOS the linux blocks are cfg-out so the missing
import was invisible to precommit on macOS.
- Bug pre-dated this branch but only surfaced when v0.17.2 release CI
ran the test compile on Linux.
Cargo.toml: 0.17.2 -> 0.17.3.
Cargo.toml: 0.17.1 -> 0.17.2. Functionally identical to the prior
commit; 0.17.1 was never published to crates.io but a tag exists on
the remote pointing at an unrelated commit. Bumping past it.
src/disc/mod.rs:
- Cache priming (3-sector lookback) before patch's single-sector reads.
Drive read-ahead pulls in adjacent pages so the target may already be
cached when we ask for it. Throwaway reads — failures here don't
update mapfile state.
- When patch hits skip-limit on a range, leave remaining sectors
NonTrimmed instead of marking Unreadable. We never tried to read those
sectors, so don't give them terminal status — drive state evolves
between passes (cache, mechanical settle), and a later pass may
succeed.
tests/pass_n_patch_fix.rs:
- New regression test for the decrypt key inversion bug at
src/disc/mod.rs:1938-1942. Asserts decrypt_sectors is invoked with
the correct key when opts.decrypt=true.
tests/pass_n_size_aware_skip.rs:
- rustfmt-only changes.
Cargo.toml: 0.17.0 -> 0.17.1.
New disc/read_error.rs as the single entry point all read failures flow
through. Handler classifies the error, updates the in-flight context
(damage window, retry budgets, jump multiplier), and returns a
ReadAction the caller dispatches on. Pass 1 (sweep) refactored to use
it; ~340 lines of nested if/else collapsed into ~120 lines of action
dispatch. Adding a new error class = one match arm. Logging is in one
place. Bisect inner failures don't poison the damage window. Jump
multiplier capped at 64 (max 1 GB jump for batch=32 — observed prior
unbounded behavior produce a single 56 GB jump on a wedged drive).
Pass N (patch) damage_skip is now size-aware: each skip is capped at
range_remaining/4 rather than the absolute MB-scale escalation. The
old logic could leap over a 100-sector bad range that hides a 50-sector
good middle; size-aware convergence finds the good middles instead.
Tests in tests/pass_n_size_aware_skip.rs exercise the size-aware skip
against synthetic patterns (25-bad/50-good/25-bad and three good
middles in a row) and prove ≥98% of good middles are recovered.
Existing test test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed
updated to reflect that MEDIUM_ERROR now triggers single-sector
bisect (which the BlockSizeFailingReader succeeds at).
- Fix unwrap in disc/mod.rs sweep() hot path using pattern matching
- Patch pass excludes Unreadable sectors from work list
- Expose bytes_bad_in_title for accurate UI reporting
- All 256 tests pass, cargo clippy clean with -D warnings
- DAMAGE_WINDOW 50→16, DAMAGE_THRESHOLD_PCT 25→12: triggers on 2nd scattered failure
- Previous 50/25% was too diluted by good reads between sparse failures
- copy() dispatch checks mapfile total_size == disc capacity_bytes (covers_disc)
- NonTried regions → sweep with resume=true (preserves mapfile)
- Only NonTrimmed/NonScraped/Unreadable → patch
- sweep_internal takes resume param: true when dispatching from existing mapfile
- Verified: Pass 1 from 30-100% completes in ~20 min with correct jumps through 3 damage zones
- CopyOptions: replace resume/skip_on_error/batch_sectors with single multipass bool
- Disc::copy() auto-detects pass from mapfile state: no mapfile or NonTried → sweep, only NonTrimmed/Unreadable → patch
- Fix bug where mapfile with NonTried regions incorrectly dispatched to patch mode
- SectorReader::set_speed() default method, Drive impl sends SET CD SPEED
- On damage zone entry: set_speed(0x0000) for better error recovery
- On damage zone exit (50 consecutive good): set_speed(0xFFFF) to restore max speed
- Disc::mapfile_for() returns /tmp/<name>.mapfile for null:// output
- patch_internal/sweep_internal as private helpers, CopyResult gains recovered_this_pass
- Adaptive probe algorithm in Disc::copy skip_on_error mode: after 4
consecutive errors, probe 1 sector at 256x batch (8 MB) ahead. If
good, zero-fill gap, mark NonTrimmed, jump. Clears bad zones in
seconds instead of hours.
- NOT_READY sense key (0x02) now retries up to 3x with 3s pause before
marking NonTrimmed. BU40N returns NOT READY for bad sectors, not
MEDIUM ERROR.
- PassProgress struct gains bytes_bad_total field for consumer-side
bad/retryable byte counts.
- Mapfile header version string fix: no longer duplicates 'libfreemkv v'
prefix on each write.
- Structured sense_key/asc/ascq logging in copy error path.
macOS SCSI transport rewritten from hybrid MMC+pread to single-path
raw CDB dispatch through SCSITaskDeviceInterface. All CDBs (INQUIRY,
READ, REPORT KEY, etc.) now go through ExecuteTaskSync — 1:1 with
the Linux SG_IO backend.
Key changes:
- New macos_shim.c: diskutil unmount → find IOBDServices →
ObtainExclusiveAccess → raw CDB dispatch. Eliminates Rust-side
IOKit COM vtable complexity.
- build.rs compiles macos_shim.c via cc into static lib
- macos.rs simplified to three FFI calls (open/close/execute)
- disc/mod.rs: graduated batch restore after errors, skip-ahead
through bad zones, configurable error pause
- as_bytes() requires OsStrExt import which is platform-specific
- as_encoded_bytes() is the portable API available on all platforms
- Fixes CI failure on Linux (Ubuntu) in GitHub Actions
0.13.28 - when drive returns MEDIUM ERROR (bad sector), skip the sector
and continue instead of retrying or bailing. Write zero-fill, mark as
Unreadable for pass 2+ recovery.
Closes: (internal)#20260427
Fix extract_scsi_context() and Error::scsi_sense() to handle Error::DiscRead
in addition to Error::ScsiError, so is_marginal_read() works for DiscRead
errors and disc::copy() can properly route MEDIUM ERROR as a marginal
(bad sector) instead of bailing.
Closes: (internal)#20260427
bytes_pending was an opaque aggregate of NonTried + NonTrimmed +
NonScraped. UIs that wanted a "will retry in Pass 2-N" bucket were
stuck showing the entire unread disc as Maybe at pct=0.
Adds two granular fields to MapStats:
bytes_nontried — Pass 1 hasn't read these yet
bytes_retryable — NonTrimmed + NonScraped, Pass 2-N will retry
bytes_pending stays for back-compat (= bytes_nontried + bytes_retryable).
Also picks up the cargo fmt --check lint that's been red on main CI
since v0.13.18 (rustfmt fold differences on a few long format-string
layouts; functional no-op).
Through the entire 0.13.x line, every CHECK CONDITION reply from the
drive (the standard way SCSI tells you why a sector failed) was being
collapsed into a synthetic status=0xFF, sense_key=0 transport-wedge
sentinel and the actual sense data was thrown away. Confirmed live on
the BU40N reading Dune 2 on 2026-04-27: drive returned host_status=0,
driver_status=8, status=2, exec_elapsed_ms=1416 on every bad sector
— a clean CHECK CONDITION carrying full sense data — and Disc::copy
was bailing on it as if the bridge had wedged.
Root cause: scsi/linux.rs's wedge check was
`host_status != 0 || driver_status != 0`
SG's DRIVER_SENSE bit (0x08) is set on every CHECK CONDITION reply
just to flag "sense buffer is populated" — it's not a transport
failure on its own. Pre-fix we conflated the two and silently lost
every drive-reported error reason. macOS and Windows backends had
the same shape: they extracted sense_key only, dropping ASC/ASCQ.
API restructure (clean separation):
Error::ScsiError {
opcode: u8,
status: u8, // 0xFF = synthetic transport-failure
sense: Option<ScsiSense>, // None ⇔ no sense delivered
}
pub struct ScsiSense { sense_key: u8, asc: u8, ascq: u8 }
impl ScsiSense {
pub fn is_marginal(&self) -> bool // keys 0/1/3/B
pub fn is_medium_error(&self) -> bool
pub fn is_hardware_error(&self) -> bool
pub fn is_unit_attention(&self) -> bool
pub fn is_data_protect(&self) -> bool
pub fn is_not_ready(&self) -> bool
pub fn is_illegal_request(&self) -> bool
pub fn is_aborted_command(&self) -> bool
}
impl Error {
pub fn scsi_sense(&self) -> Option<&ScsiSense>
pub fn is_scsi_transport_failure(&self) -> bool
pub fn is_marginal_read(&self) -> bool
}
SCSI protocol constants (SCSI_STATUS_*, SENSE_KEY_*) moved from
error.rs to scsi/mod.rs where they belong alongside SCSI_INQUIRY,
SCSI_READ_10, etc. parse_sense replaces parse_sense_key (returns the
full triple, not just the key); inline tests now exercise ASC/ASCQ
extraction at the right offsets for both descriptor (0x72/0x73) and
fixed (0x70/0x71) sense formats.
Disc::copy + Disc::patch sense-aware dispatch:
- marginal sense (MEDIUM ERROR / ABORTED COMMAND / RECOVERED ERROR
/ NO SENSE) → engage hysteresis (Block→Single, bpt=1)
- non-marginal sense (HARDWARE / DATA PROTECT / UNIT ATTENTION /
NOT READY / ILLEGAL REQUEST / transport failure / kernel
IoError) → bail with full sense info preserved; caller (autorip)
surfaces "physical replug" / "drive failing" / "media changed"
Pre-fix: every CHECK CONDITION → 0xFF synthetic → Disc::copy bailed
→ bytes_good froze at the bad zone. The hysteresis from v0.13.22
was correct but never got to run. This release unblocks it.
Disc::patch's wedged_threshold (50 consecutive failures) stays as
defense-in-depth for chains of marginal failures; a single
non-marginal sense now short-circuits it.
New phase=bail trace event records the bail reason with the sense
triple. phase=transport_err remains for genuine bridge wedges /
kernel timeouts; phase=scsi_err carries the parsed sense_key, asc,
ascq for drive-reported errors.
All 350 tests pass. Clippy clean across all targets.
The v0.13.21 bisect-on-fail recovery was correct (100% of recoverable
sectors picked up) but slow on dense damage clusters. Live test on
Dune 2 v0.13.21 burned ~30 s per damaged 60-block — paying a ~5 s
kernel ABORT/timeout at every level of a log₂(60) ≈ 6 deep DFS, on
the failing branch each time.
Replaced with a two-state hysteresis machine in Disc::copy:
Block(batch):
read(batch) ok → write, advance, stay Block
read(batch) fail → switch to Single, retry SAME range at bpt=1
Single:
read(1) ok → write, consecutive_good++
if consecutive_good >= BPT1_EXIT_THRESHOLD:
switch to Block, reset counter
read(1) fail → mark NonTrimmed, consecutive_good = 0
BPT1_EXIT_THRESHOLD = 10_000 sectors (= 20 MB clean run). Calibrated
from the 2026-04-26 BU40N empirical probe data; tunable.
Per-block math on a damaged 60-block with 1 truly bad sector:
Bisect (v0.13.21): ~30 s (5 s × 6 levels)
Hysteresis (v0.13.22): ~10 s (5 s bpt=batch fail
+ 59 × 1 ms good
+ 1 × 5 s bad)
Inside a damaged cluster spanning many 60-blocks the win compounds:
hysteresis pays the bpt=batch fail cost ONCE on entry, then stays at
bpt=1 across the cluster; bisection re-paid it every 60 sectors. For
Dune 2's ~1248-sector boundary cluster that's ~21 fewer 5-sec
kernel timeouts ≈ 100 s saved per pass.
Telemetry: new phase=mode_change trace event with from, to, lba, and
consecutive_good. Replaces v0.13.21's phase=bisect. Worklist DFS is
gone — single iterative for s in 0..count on the failure path.
Test rename, same fixture and same 100% recovery expectation:
test_disc_copy_bisect_recovers_via_single_sector_reads
→ test_disc_copy_hysteresis_recovers_via_single_sector_reads
Also adds DamageSeverity (Clean / Cosmetic / Moderate / Serious) +
classify_damage(bad_sectors, lost_ms), re-exported from libfreemkv,
so applications can render structured severity instead of formatting
their own from raw counters.
Fixes the BU40N wedge cycle that has been chasing us through
v0.13.18-20. Two changes, both backed by empirical live-hardware
probes recorded in (internal)/docs/TEST_PLAN.md:
1. scsi/mod.rs: READ_TIMEOUT_MS 1500 → 10000 ms.
Cold-start seek on the BU40N takes ~1.5 s. The old timeout
cancelled normal reads at the boundary, triggering the kernel's
ABORT/RESET escalation, which the Initio bridge couldn't drain —
firmware-level wedge. 10 s catches every legitimate slow read
(max successful ECC recovery: 2.6 s; cold-start: 1.5 s) with
margin and short-circuits truly bad sectors at ~10 s.
2. disc/mod.rs: Disc::copy bisect-on-fail (replaces skip-forward).
Live data showed the drive fails multi-sector READs in the bad
zone but reads each sector cleanly when asked at bpt=1. Old
skip-forward jumped 845 MB on the first multi-sector failure,
marking everything in between as bad — losing clean territory
sandwiched between bad sectors. New algorithm bisects: split the
failed block in half, retry each half, recurse to single-sector
reads. Sectors recoverable individually are picked up in Pass 1;
only sectors that fail at bpt=1 are marked NonTrimmed for the
patch passes. Stack-based DFS, log2(batch) = 6 levels for the
default 60-sector batch.
Multi-pass machinery is untouched. Pass 2..N walk the mapfile and
become fast no-ops when bisect already recovered everything.
Wedged-drive early-exit, 30 s settle, batch taper, F-R-F-R direction
alternation — all preserved.
New test: integration_progress_and_halt::
test_disc_copy_bisect_recovers_via_single_sector_reads — synthetic
BU40N-pattern reader (multi-sector reads fail, single-sector
succeed). Pre-patch: lost everything to skip-forward. Post-patch:
100 % bytes_good. Plus the 10 sense-key parser tests from the
0.13.20 test-coverage pass.
Empirical recovery on Dune 2 UHD on the BU40N (per TEST_PLAN.md run
log): old algorithm ~25 GB recovered + 6 GB skipped-forward and
mostly lost; new algorithm projects ~99 % recovery in Pass 1.
Audits + raw probe data:
- (internal)/docs/TEST_PLAN.md (run log)
- (internal)/docs/audits/2026-04-26-scsi-architecture-research.md
- scsi/linux.rs: full rewrite from async write/poll/read+1.5s timeout+
close-on-timeout to one synchronous ioctl(fd, SG_IO, &hdr). Kernel
honors hdr.timeout and runs its own ABORT/RESET escalation. Errors
check host_status and driver_status (both 0xFF-synthesised) plus
status. Sense-key parser handles descriptor (0x72/0x73) + fixed
(0x70/0x71) formats. Deleted fd_recovery, bg close+open thread, fd
swap dance. -331/+155 lines.
- scsi/macos.rs: try_recover() removed (userspace handle-recovery on
task failure was the same anti-pattern stripped from Linux). bsd_name
field deleted. Errors bubble up directly.
- scsi/windows.rs: try_recover() removed, wide_path field deleted,
INVALID_HANDLE guard removed.
- scsi/mod.rs: parse_sense_key() helper extracted (used by all three
platforms now — single canonical sense-key parse rather than three
inlined copies). +10 unit tests covering descriptor format, fixed
format, truncated buffers, unknown response codes.
- drive/mod.rs: Drive::reset() deleted (escalating eject + STOP/START +
reinit recovery — per audit, kernel handles its own escalation;
userspace shouldn't).
pub fn find_drives() -> Vec<Drive> deleted (opened N drives just to
throw most away). find_drive() now uses discover_drives() directly.
wait_ready() simplified — drops the reset path on sense_key=5,
just keeps polling TUR for 60 iterations.
- lib.rs: find_drives re-export removed.
- benches/sgio_read.rs: switched to find_drive() (no longer iterates a
drive list).
Net: 9 files changed, 226 insertions(+), 473 deletions(-). 329 tests
pass, clippy -D warnings clean. No consumer breakage (CLI, autorip,
bdemu compile + test green).
Architecture decision documented in
(internal)/docs/audits/2026-04-26-scsi-architecture-research.md
(primary-source survey of MakeMKV, sg_dd, ddrescue, and the kernel
mid-layer's own scsi_eh.rst escalation ladder).
Pre-0.13.16 the rip API leaked internal mapfile concepts (pos,
bytes_good, work_done, bytes_pending, Finished/NonTrimmed) into per-pass
positional callbacks Fn(u64, u64, u64). Consumers reinvented the math
each time, and v0.13.15's UI bug surfaced exactly because of this —
autorip's web JS computed pct from bytes_good while the backend
computed from pos, silent drift, frozen UI bar.
This release replaces both Disc::copy::on_progress and
Disc::patch::on_progress callbacks with a single Progress trait +
PassProgress struct (new progress module).
pub struct PassProgress {
pub kind: PassKind, // Sweep | Trim {reverse} | Scrape {reverse} | Mux
pub work_done: u64,
pub work_total: u64,
pub bytes_good_total: u64,
pub bytes_total_disc: u64,
}
pub trait Progress {
fn report(&self, p: &PassProgress);
}
impl<F: Fn(&PassProgress)> Progress for F { ... } // closures work directly
CopyOptions::on_progress and PatchOptions::on_progress are renamed to
progress: Option<&dyn Progress>. Closure callers update trivially via
the blanket impl; struct callers gain a clean named-field shape with no
positional-arg confusion.
PassKind carries the semantic (sweep vs trim vs scrape vs mux) so
consumers can label phases without reinventing detection logic.
Disc::patch reports Trim {reverse} for retry passes with block_sectors
>= 2 and Scrape {reverse} when block_sectors == 1. Direction comes
through reverse: bool. Mux variant is reserved for v0.13.17 when the
mux pipeline emits progress.
Tests + clippy clean across all 4 crates.
Breaking: CopyOptions::on_progress + PatchOptions::on_progress now take
Fn(bytes_good, pos, total). Consumers display `pos` for "% swept" — the
true Pass 1 progress that advances through skip-forward bad zones, where
bytes_good (Finished sectors only) freezes. v0.13.14 live trace proved
the existing UI was lying for ~14 minutes about Dune 2 being "stuck at
30%" while Pass 1 was actually 83% through the disc via skip-forward.
PatchOptions::reverse: walk bad ranges from highest LBA to lowest. For
drives that wedge after a forward read of a bad sector, approaching the
post-bad-zone NonTrimmed range from end-of-disc reads good sectors before
the drive sees a bad one. Hypothesis informed by the BU40N + Initio
bridge live data — Pass 2 forward saw zero successful reads in 7 min
while Pass 1's pos walked all the way to end-of-disc.
PatchOptions::wedged_threshold: > 0 → exit early after that many
consecutive failures with zero successes in the same pass. Saves the
wallclock budget for productive grinding when the drive has wedged on
the bad zone for THIS pass; a different direction or block size in the
next pass may still recover. New PatchResult::wedged_exit reports it.
Trace: patch_start (block_sectors, recovery, reverse, wedged_threshold,
num_ranges) and patch_done (blocks_attempted, blocks_read_ok,
blocks_read_failed, wedged_exit, halted, bytes_recovered) at the
freemkv::disc target.
v0.13.12 shipped the async fd_recovery design but a live test on Dune 2
showed Pass 1 sat for 14 minutes with bytes_good=0 — the inner loop iterates
(throttled on_progress log fires every 78s) but each iteration evidently
takes ~60s instead of the microseconds the design promises on fast-fail.
Without trace-level telemetry at the SCSI + Disc::copy boundaries we
can't tell where the time goes.
This release is instrumentation only — no behavior change.
- New dep: tracing 0.1. Per project docs, debug/trace logging is allowed in
libfreemkv (the no-English rule applies to errors). Consumers wire a
tracing subscriber.
- SgIoTransport::execute (Linux): trace at every state transition (entry,
recovery_swap_ok, recovery_pending, write_ok / write_err, poll_done,
timeout_spawn_recovery, scsi_err, read_err, ok). Each event includes
opcode + elapsed timing. The bg recovery thread also traces close_ms +
open_ms so we can see if the kernel really takes 60s to close+open on a
wedged Initio bridge.
- Disc::copy: trace at copy_start, outer_loop, region_enter, every 100
inner-loop iterations (iter_progress with pos/region_end/skip_size/
bytes_good/read_ok_count/read_err_count/last_read_ms/copy_elapsed_ms),
copy_done.
- All trace events use targets `freemkv::scsi` and `freemkv::disc` so
consumers can filter by subsystem (e.g. autorip /api/debug?q=freemkv::scsi).
Next: run the live test on Dune 2 again, read the autorip JSONL log,
diagnose why each iter is slow, fix the actual bug.
The wallclock-based halt timing failed on fast CI runners where a 2 GB
synthetic-disc skip-forward sweep finishes in <100 ms — well under the
200 ms halt fire delay. Reader now signals halt on first read; the
inner-loop halt check on iteration 2 breaks 'outer. No wallclock race.
Fix 1: delete stall guard from Disc::copy. Pass 1 must sweep end-to-end
per ddrescue model (RIP_DESIGN.md §2.1, §3, §9). The v0.13.9 guard at
disc/mod.rs broke Pass 1 at 30% on Dune 2 with 56 GB still NonTried.
Removed stall_secs field, narrative comment in scsi/linux.rs, and the
broken regression test. Replaced with test_disc_copy_completes_full_disc_
with_failing_reader and test_disc_copy_halts_promptly_on_failing_reader.
Fix 2: async SCSI transport recovery. Added Arc<AtomicI32> fd_recovery
on SgIoTransport. On poll timeout: spawn close + spawn open in
background, return Err immediately. Top of execute() swaps fd from
recovery atomic. Main thread never blocked beyond ~1.5s poll budget
(was up to ~60s per timeout because kernel serialized main-thread
open() against in-flight close()). Drop drains pending recovery fd.
§15.1 cross-platform parity: Windows + macOS now have the same
observable recovery contract. SptiTransport gets try_recover()
(synchronous CloseHandle + CreateFileW; Windows close is fast, no
in-flight CDB drain like Linux). MacScsiTransport gets try_recover()
(release IOKit interface + reacquire via new acquire_device_iface()
helper); stores bsd_name for re-resolution. Drop guards null'd-out
interfaces. Stripped English error strings ("try as root" / "run as
administrator") on Linux + Windows. Fixed Windows TimeOutValue
ms→s ceiling so 1500ms gets 2s (was 1s; broke Drive::read fast path).
Fix 4: instrument Disc::patch arms. PatchResult exposes
blocks_attempted, blocks_read_ok, blocks_read_failed so the v0.13.11
mystery (Dune 2 Pass 2 recovered 0 bytes in 100 min) is diagnosable
from the live device log without re-instrumenting from outside.
Cleanup: honor PatchOptions::full_recovery (was read into _ and
ignored; now routed to read_sectors recovery arg). Updated
CopyOptions::batch_sectors doc to describe the actual production
path (sysfs detect_max_batch_sectors, typically 60 sectors / ~120 KB
on BU40N) rather than the test-only 32-sector internal default.
All four crates clippy-clean and tests green on the host targets
(macOS native + cargo check on Linux). Cross-platform CI watches
Linux + Windows + macOS builds + tests.
v0.13.10's 'fd=-1 on first poll timeout' was too aggressive: a single
transient killed the entire transport, Pass 1 finished in 45ms with
0 GB good on Dune 2.
Revert to spawn-close + main-thread-reopen (the v0.13.5/8 pattern).
Per-timeout cost is up to ~60s while the kernel completes the
abandoned command, but the v0.13.9 Disc::copy stall guard caps
catastrophic stalls at 120s of bytes_good non-advance. Pass 1 bails
cleanly with NonTrimmed ranges; Pass 2 has a working Drive for
retries with recovery=true + 30s timeouts.
Fixes the silent Pass 1 hang observed on Dune 2 with v0.13.8 (drive
grinding through bad sectors at 0 KB/s, errs=0, no error surfaced).
Root cause: SgIoTransport::execute's reopen-after-poll-timeout opened
a fresh /dev/sg* fd on the main thread, which serialized against the
spawned close() of the old fd via the kernel's per-device state lock.
The userspace 1.5s timeout still fired, but the abandon-and-reopen
recovery itself blocked the main thread for as long as close() took.
Net: reads returned slowly, skip-forward fired on every iteration,
bytes_good never advanced.
- SgIoTransport::execute: on poll timeout, spawn close, set fd=-1,
return Err. No reopen on the main thread. The transport is now
invalidated until the consumer creates a fresh Drive.
- Disc::copy: add stall guard. CopyOptions.stall_secs (default 120s).
If bytes_good doesn't advance for the threshold, break 'outer
cleanly with complete=false, bytes_pending > 0 so Pass 2 retries
pick up the NonTrimmed ranges with recovery=true 30s timeouts.
- New regression test: test_disc_copy_stall_detection_triggers_
skip_forward in tests/integration_progress_and_halt.rs.
Follow-up to the rip-recovery + drive-access updates: aligns the
remaining docs with the v0.13.6 single-shot read model and the
three-layer recovery architecture.
- architecture.md: module map says single-shot read; new paragraph on
layered recovery with postmortem pointer.
- api-design.md: EventKind enum example expanded; emission notes
document that BytesRead now fires from DiscStream::fill_extents and
Retry/SectorRecovered are no longer emitted in 0.13.6+.
- disc-to-rip.md: Step 10 of the pipeline diagram + module table
reflect single-shot read.
- docs/README.md: added rip-recovery.md to the TOC.
Updates docs/ to reflect the recovery-loop strip:
- rip-recovery.md: drops Phase 1/2/3 description, replaces with three-layer
model (Disc::patch multi-pass / DiscStream batch halving / Drive::read
single-shot). Notes that no SCSI resets fire from any retry path.
- drive-access.md: removes SG_SCSI_RESET + STOP/START UNIT escalation
references; SgIoTransport::reset is now kernel SG_IO flush + ALLOW
MEDIUM REMOVAL only.
- src/mux/disc.rs + tests/: cargo fmt cleanup.
Drive::read is now single-shot. Phase 1/2/3 retries + scsi::reset+reopen
removed (~80 lines). recovery=true bumps timeout to 30s; recovery=false
stays at 1.5s. On any failure returns Err(DiscRead) immediately — caller
(Disc::patch outer loop, DiscStream batch halver) handles retries.
Inline reset+reopen WAS the wedge primitive on the LG BU40N. Per prior
post-mortem, every USB/SCSI reset path tested fails to recover the
wedged Initio bridge — the inline retry was pure cost.
SgIoTransport::reset (Linux) trimmed to kernel SG_IO state flush +
ALLOW MEDIUM REMOVAL. SG_SCSI_RESET ioctl + STOP/START UNIT escalation
removed. macOS reset removed (no-op). scsi::reset() top-level family
removed (no callers).
EventKind::BytesRead { bytes, total } now actually emitted from
DiscStream::fill_extents after each successful sector read. Was
declared in 0.13.0, never fired. Drives autorip per-device progress
in direct mode.
EventKind::Retry / SectorRecovered no longer emitted (variants kept
for forward compat). SpeedChange still emitted via Drive::set_speed
public path.
Tests: new tests/integration_progress_and_halt.rs (5 tests). 233 unit
tests + 5 integration green.
USB/SCSI recovery escalation in drive_has_disc (0.13.1-0.13.3) tested
on LG BU40N USB BD-RE: USBDEVFS_RESET, authorized toggle, driver
unbind/rebind, SCSI host rescan — all succeed at the USB transport
layer but the drive firmware below the bridge stays locked. Only
physical unplug-replug clears it. Rolled back so consumers can
surface the real failure to the user.
New: list_drives falls back to sysfs-cached vendor/model/rev from
/sys/class/scsi_generic/sgN/device/ when live INQUIRY returns empty,
so wedged drives still show their identity in UIs.
Removed: scsi::usb_reset, usb_reset_with_timeout, per-platform
usb_reset methods, recover_then_probe, is_wedge_signature. Breadcrumb
comment in scsi/linux.rs::drive_has_disc points at v0.13.3 tag for
the full implementation if future hardware needs it back.
Linux/macOS/Windows pass-through symmetric; 233 tests passing.
0.13.2's is_wedge_signature gated on opcode=SCSI_INQUIRY (0x12), but
drive_has_disc issues TEST UNIT READY (0x00). Production wedge errors
(E4000: 0x00/0xff/0x00) never matched → SCSI reset + USB reset
escalation never fired.
Drop the opcode gate. Status byte 0xFF is synthesised by our own
execute() path on poll() timeout — it's the ground-truth wedge
marker for any opcode.
Linux-only; macOS/Windows use sense-key-based wedge detection.
Architectural cleanup. autorip + freemkv CLI were reimplementing drive
discovery (sysfs walking, type-5 filtering, sg-path construction) and
calling SCSI reset primitives directly. All of that hardware-aware code
moves into libfreemkv with two cheap public probes:
- DriveInfo + list_drives() — multi-OS enumeration (Linux/macOS/Windows)
with peripheral-type-5 filtering and INQUIRY identity. Cheap.
- drive_has_disc(path) — single TUR with internal wedge recovery
escalation (SCSI reset → USB reset → retry) hidden from callers.
USB-layer reset (USBDEVFS_RESET / IOUSBDeviceInterface::ResetDevice /
storport's combined reset) wired across all three platforms.
Visibility tightening — scsi::reset, scsi::usb_reset, and the timeout
constants are now pub(crate). Compile-time guarantee that no consumer
crate can issue SCSI commands directly.
233 lib tests pass; clippy clean.
Production incident: autorip's poll loop called scsi::reset() on a
wedged BU40N USB drive. The Linux SG_SCSI_RESET ioctl blocked
indefinitely (kernel SCSI subsystem waiting for a bus-wedged device to
ack a reset that will never come). Caller's poll loop hung for 60+
seconds before manual intervention.
scsi::reset() now spawns a detached worker for the platform-specific
reset and bounds the caller's wait via mpsc::recv_timeout. Default
30 s (DEFAULT_RESET_TIMEOUT_SECS); reset_with_timeout(device, dur)
exposes the bound for callers that want a different value. Returns
DeviceResetFailed on timeout. Worker thread keeps running until the
kernel eventually unblocks — leaks one OS thread per hard wedge, but
the daemon stays responsive instead of hanging forever.
Follow-up flagged for 0.13.2: USB-attached drives wedge at the USB
Mass Storage layer below SCSI; SG_SCSI_RESET doesn't help. A
scsi::usb_reset(path) using USBDEVFS_RESET is the proper escalation.
Audit pass against the project docs "no English text in library code" rule.
Found 9 call sites that violated the contract by stuffing English into
io::Error::new(kind, "…") or by abusing Error::DeviceNotFound { path }
as a free-form description field. Each is now a typed Error variant.
New variants and codes: ScsiInterfaceUnavailable (E1004), DeviceLocked
(E1005), IoKitPluginFailed (E1006), UnsupportedPlatform (E2003),
PlatformNotImplemented (E2004), MapfileInvalid (E6011), DiscUrlNotDirect
(E9009).
labels::apply() previously pushed Commentary/Descriptive/Score/IME and
" (Secondary)" English literals into AudioStream.label, leaking into
MKV titles + autorip UI. AudioStream now exposes structured `purpose:
LabelPurpose`, SubtitleStream `qualifier: LabelQualifier`. Callers
translate to localized text. label keeps codec-formatting only.
API hygiene: 11 mux/* modules dropped from `pub` to `pub(crate)` —
their *types* are still re-exported from lib.rs, but the modules were
leaking low-level EBML/TS/network primitives. Stream trait gets a real
rustdoc explaining read-vs-write split. lib.rs grouped re-exports into
documented sections. ScanOptions::with_keydb() removed (one-method-per-
action rule); use struct literal.
Dead-code sweep: removed lookahead.rs (orphan, never declared as mod),
tsreader.rs (TsDemuxReader unused), ebml::{write_int,read_vint,SEEK_*},
ts::{scan_first/last_pts,scan_duration,SCAN_HEAD/TAIL_SIZE,take/set_
remainder}, MkvMuxer codec_private_slots/filled fields and
fill_codec_private method (deferred-codecPrivate path never used since
the v0.10 PES rewrite). cargo clippy --all-targets -D warnings clean.
Tests: new error::tests for variant codes + Display "no English" guard +
io::ErrorKind mapping. 233 lib tests, all green (was 230).
Breaking: ScanOptions::with_keydb removed; mux/* modules pub(crate);
AudioStream and SubtitleStream gained required fields; UnsupportedDrive
{ product_revision: "Renesas not yet implemented" } no longer produced
(use PlatformNotImplemented).
Drive::read is now halt-check-free in its body. Previously, the halt
flag was checked in 4 places and the sleep logic was scattered across
4 "if halt_aware_sleep { return Halted }" call sites — correct, but
the ugliness invited drift: a new sleep added by someone unfamiliar
with the pattern would silently swallow Stop requests.
Two private primitives now own halt awareness:
checked_sleep(Duration) -> Result<()>
checked_exec(cdb, dir, buf, timeout_ms) -> Result<ScsiResult>
Both return Err(Halted) instead of a bool. The ? operator in read()
then propagates halts for free. The recovery path reads top-to-bottom
with no halt vocabulary.
sleep_until_halted lives as a free function so it's unit-testable
without a live Drive. 4 new tests: completes normally, bails on
pre-set flag within one slice, wakes mid-sleep, zero-duration no-op.
Public API unchanged — halt_flag/halt/clear_halt still exposed, the
refactor is entirely internal.
Stop was waiting up to 30 s to register when the drive hit L-EC
recovery mid-read — the recovery phase does 30 s sleeps between
retries and the halt flag was only checked at the start of each sleep.
UI feels broken ("stop isnt working") even though the halt was set.
halt_aware_sleep breaks each wait into 100 ms slices and returns
early on halt. Applied to all 4 sleeps in the recovery path (both
30 s retry delays, both 5 s reset-phase delays).
Disc::copy fast pass (skip_on_error=true, recovery=false) was giving
the drive 5 s per 64 KB block. On structure-protected / marginal UHD
sectors the drive grinds L-EC for nearly the full budget per block,
pinning throughput at ~13 KB/s even though skip_forward would happily
skip past the region.
1500ms bounds the floor at ~43 KB/s/block. Recoverable sectors that
would have succeeded at 3-5 s get picked up on Disc::patch (pass 2+)
where recovery=true and the per-read budget is 30 s.
- README quick-start gains a multi-pass example using the new
Disc::copy + Disc::patch primitives.
- New docs/rip-recovery.md documents the two-stage rip model: mapfile
format (ddrescue-compatible), CopyOptions/PatchOptions surface, the
pass-1/pass-2 algorithm, and the design decisions (why no MODE
SELECT, why ISO intermediate, why ddrescue mapfile).
No code change.
autorip 0.11.22 ships the full multi-pass UI (live mapfile stats,
bad-range viz, Recovery settings). libfreemkv API is unchanged from
0.11.21. Part of the 0.11.22 ecosystem sync.
New primitives for two-stage rip workflows: fast forward pass with
zero-fill on failures, then targeted retries of bad ranges via a
ddrescue-compatible mapfile.
- Disc::copy now takes &CopyOptions (breaking change from positional
args). Always writes a sidecar .mapfile. Opt-in skip_on_error +
skip_forward give ddrescue-style fast sweep: 64 KB blocks,
exponential skip-forward on failure, zero-fill bad blocks. Defaults
preserve pre-0.11.21 behavior (recovery reads, abort on bad sector).
- Disc::patch is new and idempotent. Reads the mapfile, re-reads every
non-finished range with full drive recovery, patches good bytes back
into the ISO at exact offsets. Call N times for N retry attempts.
- disc::mapfile is a new module. ddrescue text format, crash-safe
(flushed on every record()), greppable, human-editable, tool-compatible.
Status chars match ddrescue: ? / * / / / - / +.
- Re-exports FileSectorReader from the crate root.
- freemkv CLI caller (pipe.rs) updated to the new Disc::copy signature
in lockstep — shipped in the 0.11.21 freemkv CLI release.
Part of the 0.11.21 ecosystem sync (libfreemkv + freemkv + bdemu +
autorip all on 0.11.21).
DiscStream::fill_extents loops internally while the demuxer waits for
enough clean data to emit a PES frame. In a dense bad zone that loop
can run for minutes without returning to the outer read() call, so
the caller's Stop signal never gets serviced until a frame is finally
emitted — which may be very far away.
Add DiscStream::set_halt(Arc<AtomicBool>) — typically wired to
Drive::halt_flag() for unified Stop across drive recovery phases and
stream sector processing. fill_extents checks the flag at the top of
every retry iteration; raising it returns Err(Error::Halted) within
one SCSI round-trip.
No behavior change for callers that don't call set_halt. Unblocks the
architectural fix for the "Stop doesn't stop" bug observed on a
damaged UHD disc.
Replace read_with_binary_search + 3×5s light recovery with an adaptive
sizer that shrinks on failure (halve, 3-aligned ≥6) and probes back up
after 100 MiB (51,200 sectors) of clean reads. Descent cost is paid
once per bad region, not once per bad sector.
Emit BatchSizeChanged { new_size, reason } on shrink and probe-up.
Remove BinarySearch event — no longer produced.
Side fix: scsi/macos.rs one-liner for manual_c_str_literals clippy
lint that surfaced on a newer toolchain.
Fix trailing sectors dropped at extent boundaries when sector_count % 3 != 0.
Add verify_title stop support via progress callback returning bool.
Add O_CLOEXEC on all SCSI fd opens to prevent leak to child processes.
Fix SCSI sense descriptor format detection (0x72/0x73 vs 0x70/0x71).
Replace blocking ioctl(SG_IO) with the sg driver's async interface.
Commands are submitted via write(), waited on via poll() with a hard
wall-clock timeout, and completed via read(). If poll() times out,
the fd is abandoned and a fresh one opened — the kernel can no longer
hold us hostage during USB error recovery.
- write() submits command, returns immediately
- poll() enforces exact timeout (EINTR-safe with deadline tracking)
- read() retrieves result + copies data to caller's buffer
- On timeout: old fd closed in background thread, new fd opened
- No SG_FLAG_DIRECT_IO — kernel buffers for safe timeout abandonment
- Store device_path for fd reopen after timeout
- Drop guards fd=-1 (abandoned fd)
Use UDF file_extents() to read actual allocation descriptors instead
of assuming m2ts files are contiguous from file_start_lba. Dual-layer
UHD discs split large files across 70+ extents (~1 GB each) — the old
code created one extent from packet count which only covered the first
chunk, causing silent truncation at ~37%.
Also changed fill_extents() to return io::Result<bool> so read errors
propagate instead of being silently treated as EOF.
A stream is a stream. DiscStream::new() takes reader + title + keys +
batch + format — same pattern as every other stream constructor.
Deleted: open_drive(), open_iso(), from_reader() — these were helper
functions that chained multiple operations. Library provides primitives,
callers decide the sequence.
Removed disc:// case from input() — callers use Drive::open() +
Disc::scan() + DiscStream::new() directly for disc sources.
- DTS: buffer with core sync detection + frame size from header
- TrueHD: buffer with unit length field parsing
- Same pattern as AC3 fix: incomplete frames held for next PES
- When PES boundaries align (normal case), buffering is a no-op
- Add state to Ac3Parser (was stateless, split frames at PES boundaries)
- Buffer leftover bytes from incomplete frames for next PES packet
- Calculate exact AC3 frame size from fscod/frmsizecod table
- Calculate EAC3 frame size from frmsiz field
- Skip invalid frame sizes (0 or >8192)
- Eliminates all AC3 decode errors on BD and UHD output
- DVD PS path now calls parser.parse() like BD-TS path does
- MPEG-2 sequence headers extracted for codec_private
- Keyframe detection from parser instead of always-true
- Fix CSS roundtrip tests: descramble uses TAB1 permutation, not pure XOR
- Implement complete CSS key chain: bus auth → disc key → title key
- Add 31 player keys for disc key decryption
- Read disc key via READ DVD STRUCTURE format 0x02
- Read title key via REPORT KEY format 0x04
- Fix CryptKey round 1: use original scratch for term, not modified tmp1
- Fix decrypt_key: use TAB5 for LFSR1 output, TAB4 for LFSR0^invert
- Fix descramble_sector: use TAB5 for LFSR1, TAB4 for LFSR0 (no invert),
and apply TAB1 permutation to ciphertext before XOR
- Fix title key bus XOR: forward order (bus_key[i]), not reversed
- Two-session auth: disc key and title key need separate AGID sessions
- Fix crack_key: scan across extents for scrambled sectors
- Fix TsDemuxer: dynamic PID table size for DVD PIDs
- Set max read speed after scan for DVD riplock removal
2026-04-16 04:42:42 +00:00
158 changed files with 38842 additions and 3908 deletions
Streams implement `IOStream` (byte-level) and `pes::Stream` (frame-level). `input()` / `output()` resolve URL strings to PES stream instances. `open_input()` / `open_output()` resolve to byte-level IOStream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption.
### Keys
DVDs (CSS) decrypt out of the box — the 1999-era public player keys are compiled into the library.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). The file holds all DKs, PKs, host certs, and per-disc VUKs. No AACS key material is compiled into the binary.
Common problems and solutions for optical drive ripping with freemkv.
---
## 1. USB-SATA Bridge Issues
This is the single most common source of problems when ripping discs over USB.
### Symptoms
- The drive disappears mid-rip. The ripping tool reports the device is gone, and `ls /dev/sg*` no longer shows it.
- The device re-enumerates under a different name: `sg4` becomes `sg5`, then `sg7`, then `sg11` after each USB port reset.
- `dmesg` shows USB port resets: `usb X-Y: reset high-speed USB device`, `xhci_hcd 0000:00:14.0: Cannot enable. Maybe the USB cable is bad?`, or `usb-storage: device reset failed`.
- The SCSI layer reports `host_status=7` (Linux USB transport error) in sense data.
- The drive works fine for reading data discs or burning, but crashes when hitting damaged sectors during a rip.
- After the crash, the drive is completely invisible until physically unplugged and reconnected.
### Root Cause
USB-SATA bridges translate between the USB Mass Storage protocol (BOT or UAS) and the drive's native SATA interface. When the optical drive encounters an unreadable sector, it returns a SCSI CHECK CONDITION with sense key 0x03 (MEDIUM ERROR). Some bridge chipsets -- particularly the Initio INIC-36xx family -- have buggy firmware that mishandles this error response.
Specific failure modes:
- **Incorrect residue reporting.** The bridge claims a different number of bytes transferred than what actually occurred. The Linux USB storage driver sees this discrepancy as a protocol violation and resets the port to recover. The `US_FL_IGNORE_RESIDUE` quirk exists specifically for this class of bug (see `drivers/usb/storage/transport.c` in the Linux kernel).
- **Bridge firmware crash.** On some Initio bridges, a malformed SCSI error response from the drive causes the bridge MCU to hang entirely. The USB controller sees the device stop responding and initiates a port reset. The bridge recovers (it re-enumerates), but the rip is dead -- all state is lost.
- **Sense data corruption.** The bridge forwards garbled or truncated sense data to the host, which the SCSI midlayer cannot parse, leading to a transport reset.
This is a hardware + firmware problem, not a software bug. The same drive connected via direct SATA does not exhibit these symptoms.
### Known Affected Bridges
| Chipset | USB IDs | Notes |
|---------|---------|-------|
| Initio INIC-3609 | `13fd:3609` | Very common in cheap SATA-to-USB enclosures. Highly problematic. |
| Initio INIC-3619 | `13fd:3940` | Same firmware family as INIC-3609. |
| Initio INIC-3069 | `13fd:0840` | Older variant, same residue bug. |
| ASMedia ASM1051 | `174c:5106` | Early ASM SATA bridge. Residue issues on error paths. |
| JMicron JMB36x | `152d:0561` | Some firmware versions. Not all JMicroon chips are affected. |
If your drive came in a pre-built external enclosure (Vantec, Sabrent, OWC, etc.), it almost certainly uses one of these bridge chips internally.
### The Fix: USB Storage Quirk
Apply the `US_FL_IGNORE_RESIDUE` kernel quirk for your bridge. This tells the Linux USB storage driver to ignore the residue field in SCSI response frames, preventing the port reset on mismatched byte counts.
**Step 1: Identify your bridge's vendor:product ID.**
```bash
lsusb
```
Look for your drive's entry. Example output:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation INIC-3609
```
Here the vendor ID is `13fd` and the product ID is `0840`.
Replace `13fd:0840` with your device's actual IDs. The `:i` flag means `US_FL_IGNORE_RESIDUE`.
You can combine multiple flags. Common additions:
- `:i` -- ignore residue (`US_FL_IGNORE_RESIDUE`)
- `:u` -- force BOT mode instead of UAS, for bridges with UAS bugs
**Step 3: Reconnect the drive.** Unplug and replug the USB cable, or bind/unbind the device. The quirk is applied per-module-load, so existing sessions may need the drive reconnected.
### Making It Persistent
Add the quirk to your kernel boot parameters so it survives reboots.
Edit `/etc/default/grub` (GRUB) and add to `GRUB_CMDLINE_LINUX_DEFAULT`:
Avoid any enclosure or adapter listing an Initio chipset.
### Best Solution: Direct SATA
Connect your optical drive directly to a motherboard SATA port. This eliminates the USB-SATA bridge entirely and is the most reliable configuration:
- No USB protocol overhead or translation errors.
- No bridge firmware bugs.
- No port resets or re-enumeration.
- Full SATA error recovery handled natively by the kernel's libata driver.
- Sustained read speeds are limited only by the drive, not the USB bus.
If your machine has a free SATA port, use it.
---
## 2. Damaged Disc Handling
### Symptoms
- SCSI MEDIUM ERROR (sense key 0x03) at specific LBAs. `dmesg` shows `sr X:0:0:0: [srY] Unrecoverable read error` or similar.
- Read speed drops to near zero when approaching a damaged area.
- The drive makes audible retrying noises (laser repositioning, spindle speed changes).
- On USB-connected drives: the bridge crashes (see section 1 above) when the drive returns the error.
### How freemkv Handles This
freemkv uses a three-layer recovery model. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for full details.
- **Pass 1 (Disc::copy):** Fast sweep with 64 KB reads. On failure, zero-fills the block and skips forward. Writes a ddrescue-format mapfile for later retry.
- **Pass 2+ (Disc::patch):** Targeted re-reads of bad ranges with a long 30-second timeout per CDB. The drive firmware performs its own ECC and laser power retries within that window.
- **In-stream (DiscStream):** Adaptive batch halving -- reduces request size on failure to isolate bad sectors within a larger block.
This means a disc with some bad sectors will still produce a usable ISO. The damaged areas are zero-filled in pass 1 and retried in subsequent passes. Structure-protected sectors (deliberate unreadable regions from copy protection) will never yield, which is expected.
### The Drive Taint Issue (LG BU40N)
Some drives, notably the LG BU40N, exhibit a "taint" behavior after encountering MEDIUM ERRORs:
1. The drive hits a damaged sector and returns a MEDIUM ERROR.
2. From that point forward, **all subsequent reads fail** -- even reads to sectors that were previously successful.
3. The only recovery is to physically unplug and reconnect the drive (or power-cycle it).
This is not a freemkv bug. It is a drive firmware behavior triggered by the interaction between the drive's internal error recovery and the USB-SATA bridge's handling of the error response. The drive firmware enters a degraded state that it does not recover from without a power cycle.
Workarounds:
- **Use a direct SATA connection.** This eliminates the bridge interaction that triggers the taint.
- **Use a different bridge.** The ASM1153 and JMS578 are less likely to trigger this behavior.
- **Accept the partial ISO.** freemkv's skip-forward recovery will zero-fill the unreadable blocks and continue. The resulting ISO may be playable with minor glitches in the affected areas.
- **Physical replug between retry passes.** If running multi-pass patch, replug the drive between passes to clear the taint state.
freemkv deliberately does not attempt inline SCSI resets or eject cycles to recover from this state, because those operations were found to make the problem worse on affected hardware (see the design rationale in [`docs/rip-recovery.md`](docs/rip-recovery.md)).
---
## 3. Drive Not Detected
### Check Hardware Visibility
```bash
lsusb
```
Verify the drive appears in the USB device list. If it does not show up, the drive is not visible to the host at all -- check cables, power, and USB port.
```bash
ls /dev/sg*
```
On Linux, optical drives appear as `/dev/sg*` devices (the SCSI Generic interface). freemkv uses `/dev/sg*`, not `/dev/sr*`. If `lsusb` shows the device but no `/dev/sg*` entry exists, the `sg` kernel module may not be loaded:
```bash
sudo modprobe sg
```
### Check Kernel Messages
```bash
dmesg | grep -i usb | tail -30
dmesg | grep -i sg | tail -10
```
Look for:
- USB enumeration errors or failed port resets.
- `sg_add` messages confirming the sg device was registered.
- Permission denied or access errors.
### Permission Issues
On most Linux distributions, `/dev/sg*` devices are owned by `root:disk` or `root:cdrom` with restricted permissions. Running freemkv as an unprivileged user will fail with permission errors.
Options:
- Add your user to the appropriate group:
```bash
sudo usermod -aG disk $USER
```
Then log out and back in for the change to take effect. On some distributions the group is `cdrom` or `optical` instead of `disk`.
- Run with elevated privileges:
```bash
sudo freemkv ...
```
- Install a udev rule for persistent per-device permissions. Create `/etc/udev/rules.d/99-sg-optical.rules`:
sudo udevadm control --reload-rules && sudo udevadm trigger
```
### Spin-Up Delay
Optical drives take 30-60 seconds to spin up and become ready after hot-plug or disc insertion. During this window, SCSI commands may return NOT READY or timeout.
freemkv's `Drive::wait_ready()` handles this automatically by polling with TEST UNIT READY until the drive responds. If you are writing your own code using the library, always call `wait_ready()` before `init()`:
```rust
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // blocks until disc is ready, up to 30s
drive.init()?;
```
If the drive was just plugged in, wait a full minute before concluding it is not detected.
---
## 4. How to Identify Your USB-SATA Bridge
If you are experiencing the issues described in section 1, you need to know which bridge chipset your adapter or enclosure uses.
### Step 1: Find the Device
```bash
lsusb
```
Look for entries matching your drive or enclosure. Bridges may appear under their own manufacturer name or as a generic SATA device. Common examples:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation
Bus 002 Device 006: ID 174c:1153 ASMedia Technology Inc. ASM1153
Bus 002 Device 007: ID 152d:0578 JMicron Technology Corp. JMS578
This often reveals the bridge chipset even when `lsusb` shows a generic name.
### Step 4: If the Enclosure Is Sealed
Many external drive enclosures (Vantec NexStar, Sabrent, OWC, etc.) do not advertise the bridge chipset on the packaging. In this case:
1. Check `lsusb` while the enclosure is connected.
2. Search the vendor:product ID online -- there are community-maintained lists of which chipsets popular enclosures use.
3. If you cannot determine the chipset and are experiencing bridge crashes, assume it is an Initio and apply the quirk with its IDs.
4. The definitive test: connect the bare drive to a motherboard SATA port. If the problems disappear, the bridge was the cause.
---
## 5. General Debugging Checklist
When something goes wrong during a rip, gather this information before filing an issue:
1. **freemkv version:**`freemkv --version` or the crate version in `Cargo.toml`.
2. **Drive model:** from the drive label, or from `freemkv info`.
3. **Connection type:** USB (with bridge chipset if known) or direct SATA.
4. **Operating system and kernel:**`uname -a`.
5. **Kernel messages during the failure:**`dmesg | tail -50` immediately after the crash.
6. **SCSI device:** which `/dev/sg*` the drive was on, and whether it changed after the failure.
7. **The disc:** title, format (BD/DVD/UHD), condition.
Include all of the above in bug reports. SCSI transport errors that resolve with the `US_FL_IGNORE_RESIDUE` quirk or by switching to direct SATA are bridge firmware bugs, not freemkv bugs.
`libfreemkv` supports a multi-stage rip model for damaged or protection-bearing
discs: a fast forward sweep that tolerates read failures, in-loop request-size
adaptation that survives transient drive trouble without bailing, and targeted
retry passes against a persistent bad-range map. The stream pipeline
(`DiscStream` + `input`/`output`) operates against the resulting ISO image, so
the mux stage never touches the drive.
Recovery is layered cleanly. Each layer has one responsibility and does not
reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
@ -120,6 +120,14 @@ Each directory read involves two sector reads: one for the ICB, then one or more
`read_file()` reads a file by navigating the directory tree, reading the file's ICB to get its data extent, then reading the data sector by sector from the **physical partition** (partition_start + LBA, not metadata_start).
## Buffered Sector Reads
USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_filesystem()` and `read_file()` issue one SCSI READ per sector, a full disc scan can require hundreds of commands -- taking 10+ minutes on USB.
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
### UDF Filename Encoding
UDF filenames use a compression ID as the first byte:
