Disc::titles previously sorted purely by duration_secs descending,
which puts a play-all virtual playlist at index 0 on UHDs that ship
one. Such playlists reference the same source clips multiple times
for seamless alternate-angle / alternate-ending playback and report
inflated duration AND inflated size_bytes that exceeds the disc's
physical capacity.
Concrete observed case (The Amateur 2025 4K UHD, 58.5 GB BD-100):
Title 1 — 00020.mpls — 4h13m — 92.4 GB — 253 clips ← impossible
Title 2 — 00800.mpls — 2h02m — 57.2 GB — 1 clip ← the movie
92.4 GB > 58.5 GB capacity is proof of clip double-counting. With
the duration-only sort, freemkv -t 1 / disc.titles.first() / autorip's
main-feature picker all selected the 4-hour composite instead of the
2-hour movie.
New canonical_title_order:
1. Real titles (size_bytes <= capacity_bytes) before virtual
composites — capacity gate is hard physical truth.
2. Among real titles, fewer clips first (1-clip wins as the
canonical main feature; multi-clip is either chapter-stitched
or composite).
3. Tiebreak on longer duration first.
Behaviour:
- Non-branching discs: unchanged. The longest 1-clip title is
already the movie.
- Branching UHDs: virtual composite drops to the back, the real
movie surfaces at index 0.
Comparator exposed as Disc::canonical_title_order for downstream
consumers that need the same logic on custom title sets.
Three regression tests (disc::tests::canonical_order_*):
- pushes_oversize_play_all_behind_real_main (The Amateur)
- preserves_natural_ranking_on_normal_disc
- fewer_clips_wins_tiebreak
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.
Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.
This commit is purely doc-side:
- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
with caller-orchestrated multipass; Streams table footnote and
Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
trait splits, deprecations (kept alive through 0.18.x, deletion
target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
drive-access,udf}.md: every Disc::copy / pes::Stream /
SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
README.md and CHANGELOG.md now, matching the workspace-top
FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.
No code changes.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.
- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
at crate root.
Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.
See freemkv-private/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.
Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.
See freemkv-private/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.
Single contributor: MattJackson.
existing call sites — sweep producer and DiscStream demux
Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:
- Disc::sweep (disc/mod.rs): producer wraps the input reader
in DecryptingSectorSource::new(reader, keys) before the read loop.
The inline decrypt_sectors call goes away — read_sectors yields
plaintext directly.
- DiscStream (mux/disc.rs): constructor wraps the underlying
Box<dyn SectorReader> in DecryptingSectorSource so the internal
fill_extents / read path sees plaintext bytes. The DecryptKeys
field stays on DiscStream for metadata-side use; it just no
longer drives decryption.
Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.
Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
&mut dyn SectorReader and Box<dyn SectorReader>, mirroring
std's Read forwarding pattern. Generic blankets would conflict
with the existing SectorReader → SectorSource blanket under the
orphan rule (downstream could impl SectorReader for &mut U), so
the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
DiscStream::set_raw() can flip the wrapped reader to a
DecryptKeys::None pass-through without rebuilding the decorator
(which would require moving the inner Box out from behind &mut self).
After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.
Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 500894e). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.
- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
carries the file-write + mapfile.record per WorkItem; close()
drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
(so the consumer thread keeps showing up as
freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
The producer-side ReadCtx state machine, decrypt, set_speed,
halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
preserve identifiable thread names without the primitive baking
one in. Also adds Pipeline::try_send for the throttled
StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
module-private types. WorkItem::Finish dropped — dropping the
channel is the end-of-stream signal Pipeline already uses.
Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.
See freemkv-private/memory/0_18_redesign.md.
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.
- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
WritebackFile + Mapfile. apply() seeks+writes recovered bytes
and records mapfile state per item; close() runs sync_all and
mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
buffering — back-pressure should kick in immediately so the
drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
producer (reverse walk, damage-window skip, NOT_READY pauses,
bridge-degradation handling, wedge exit, range watchdog).
WritebackFile ownership moves to the sink.
Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.
No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.
Single contributor: MattJackson.
Mirror of the FrameSink concrete migrations slice (760e40b) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.
The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:
- Audits DiscStream's interior types for Send (its Box<dyn
SectorReader> already requires Send via the trait's super-bound;
verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
over a DiscStream, reads frames through the trait object, and
exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
if call-site ergonomics demand it; otherwise relies on the
blanket.
No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:
- src/halt.rs: clippy::new_without_default fires when a public new()
exists without Default. The polish pass dropped the derive thinking
it was redundant — clippy disagrees, so add a manual impl that
forwards to new(). Doc-comment notes why both exist.
- src/disc/read_error.rs:372: pre-existing
assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
precommits passed because that lint sat outside the gate; the
round-2 commits brought enough new clippy surface that it now
shows up. Trivial cleanup: assert!(...) instead of assert_eq!.
Single contributor: MattJackson.
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.
The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.
FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.
Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.
Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).
The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.
FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.
Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.
See freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
freemkv-private/memory/0_18_redesign.md for full context.
Single contributor: MattJackson.
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.
Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.
No alias kept; this is a clean 0.18 rename. See
freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson.
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
freemkv-private/memory/0_18_redesign.md.
Single contributor: MattJackson (no Co-Authored-By trailers).
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.
This release plugs those gaps:
- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
Writer before any seek / write. sync_all on Writer cleanly drains
the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
of sequential output; the page cache no longer absorbs that as a
single hot blast on slow targets.
Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.
Mapfile now batches the rename to once per second:
- record() always updates in-memory state and stats; only fires
write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
consumer at end-of-sweep and by Disc::patch at end-of-patch,
after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
silently lose pending state.
Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.
Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)
Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
Pre-0.17.11 sweep ran strictly serialised: SCSI read → decrypt → seek
+ write → mapfile.record → next read. Drive idled for the post-read
work; throughput capped at the sum of both costs. On a healthy disc
that's ~7-12 ms read + ~5-15 ms write/record per 64 KB batch, limiting
sustained throughput to ~10-12 MB/s on the test bed (BU40N + UHD inner
zone), well below the ~14-16 MB/s drive ceiling.
Decouples them: producer thread (caller's) owns SectorReader +
read_error state + decrypt + set_speed + halt; consumer thread (one
spawn) owns Writer + Mapfile, receives WorkItem messages, applies
file write + mapfile record. Bounded mpsc::sync_channel(4) gives
natural back-pressure. While the consumer writes batch N, the
producer is already reading batch N+1 — steady-state throughput is
now bound by the slower of the two pipelines (drive on healthy
discs), not their sum.
Side effects:
- Bisect path now decrypts. Pre-0.17.11 the bisect inner loop wrote
raw cyphertext for single-sector recoveries on encrypted discs —
quiet correctness bug exercised only by batch-fail-then-
bisect-succeed on encrypted media. New producer-side decrypt
covers main + bisect success paths uniformly.
- All read_ctx state stays single-threaded on producer (damage
window, jump multiplier, etc.). No locking added.
- Mapfile remains single-writer on consumer. No locking.
- Halt latency: producer breaks loop, sends Finish, consumer drains
≤4 in-flight items + sync_all. ~1 batch (~12 ms) typical.
- BU40N + Initio bridge wedge concern unchanged: still single SCSI
command in flight, error-path timing identical, no new retries.
New module: src/disc/sweep_pipeline.rs (WorkItem, ProgressSnapshot,
ConsumerInputs, spawn_consumer, consumer_loop, helpers). Public API
unchanged — Disc::copy / CopyOptions / CopyResult identical.
Patch (Pass N) is NOT changed; it's bound by drive recovery time, not
the read/write serialisation.
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.
- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
Wraps a per-platform WritebackPipeline that on Linux schedules
sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
mux) can adopt the same wrapper as a one-line change later.
Version bump to keep the four freemkv crates at unified versioning
after autorip's v0.17.6 + v0.17.7 work today. No libfreemkv code
changes; republished to crates.io so downstream consumers stay
aligned on the latest patch version.
Direct-SATA BU40N + Dune Part Two UHD live testing exposed that the
v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-
sector LBAs that dd if=/dev/sr0 recovers on the same drive. This
release closes that gap and fixes adjacent bugs silently capping
recovery.
- /dev/sr0 pread fallback in Drive::read (Linux only): on SCSI READ
Err, fall back to posix_fadvise(DONTNEED) + pread() against the
corresponding block device. Kernel sr_mod runs ~5 internal retries
with no per-attempt mid-layer escalation overhead — the mechanism
behind dd's recovery advantage. End-to-end byte verification
confirms the fallback path returns real disc data.
- Disc::patch per-range watchdog fix: MAX_RANGE_SECS was breaking
'outer (one slow range killed the entire patch). Now skips to the
next range. Pre-fix patch died after 4 sectors of range 1 of 47.
- Per-sector range budget: range_budget = sectors × 25 s, capped at
1800 s. Replaces the flat 180 s/range that was unfair to medium
ranges and pointlessly generous to single-sector ones.
- consecutive_failures resets per range. The wedge-exit detector is
for stuck-on-one-range, not many-small-ranges-with-one-fail-each.
- Reverted inline 5× retry experiment (was hurting: each retry paid
kernel SCSI escalation overhead). Restored READ_RECOVERY_TIMEOUT_MS
to 60 s. The kernel-auto-retry pattern is now provided by sr0
fallback.
Empirical: pass 1 recovered 94.6 MB / 11 s of main title (33 sr0
saves). Pass 2 added 0.6 MB. Remaining ~233 MB on the test disc
appears physically unrecoverable on this hardware.
tests/scsi_recovery.rs:
- Add `use std::time::Duration` inside both `#[cfg(target_os = "linux")]`
blocks. Locally on macOS the linux blocks are cfg-out so the missing
import was invisible to precommit on macOS.
- Bug pre-dated this branch but only surfaced when v0.17.2 release CI
ran the test compile on Linux.
Cargo.toml: 0.17.2 -> 0.17.3.
Cargo.toml: 0.17.1 -> 0.17.2. Functionally identical to the prior
commit; 0.17.1 was never published to crates.io but a tag exists on
the remote pointing at an unrelated commit. Bumping past it.
src/disc/mod.rs:
- Cache priming (3-sector lookback) before patch's single-sector reads.
Drive read-ahead pulls in adjacent pages so the target may already be
cached when we ask for it. Throwaway reads — failures here don't
update mapfile state.
- When patch hits skip-limit on a range, leave remaining sectors
NonTrimmed instead of marking Unreadable. We never tried to read those
sectors, so don't give them terminal status — drive state evolves
between passes (cache, mechanical settle), and a later pass may
succeed.
tests/pass_n_patch_fix.rs:
- New regression test for the decrypt key inversion bug at
src/disc/mod.rs:1938-1942. Asserts decrypt_sectors is invoked with
the correct key when opts.decrypt=true.
tests/pass_n_size_aware_skip.rs:
- rustfmt-only changes.
Cargo.toml: 0.17.0 -> 0.17.1.
New disc/read_error.rs as the single entry point all read failures flow
through. Handler classifies the error, updates the in-flight context
(damage window, retry budgets, jump multiplier), and returns a
ReadAction the caller dispatches on. Pass 1 (sweep) refactored to use
it; ~340 lines of nested if/else collapsed into ~120 lines of action
dispatch. Adding a new error class = one match arm. Logging is in one
place. Bisect inner failures don't poison the damage window. Jump
multiplier capped at 64 (max 1 GB jump for batch=32 — observed prior
unbounded behavior produce a single 56 GB jump on a wedged drive).
Pass N (patch) damage_skip is now size-aware: each skip is capped at
range_remaining/4 rather than the absolute MB-scale escalation. The
old logic could leap over a 100-sector bad range that hides a 50-sector
good middle; size-aware convergence finds the good middles instead.
Tests in tests/pass_n_size_aware_skip.rs exercise the size-aware skip
against synthetic patterns (25-bad/50-good/25-bad and three good
middles in a row) and prove ≥98% of good middles are recovered.
Existing test test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed
updated to reflect that MEDIUM_ERROR now triggers single-sector
bisect (which the BlockSizeFailingReader succeeds at).
- Fix unwrap in disc/mod.rs sweep() hot path using pattern matching
- Patch pass excludes Unreadable sectors from work list
- Expose bytes_bad_in_title for accurate UI reporting
- All 256 tests pass, cargo clippy clean with -D warnings
- DAMAGE_WINDOW 50→16, DAMAGE_THRESHOLD_PCT 25→12: triggers on 2nd scattered failure
- Previous 50/25% was too diluted by good reads between sparse failures
- copy() dispatch checks mapfile total_size == disc capacity_bytes (covers_disc)
- NonTried regions → sweep with resume=true (preserves mapfile)
- Only NonTrimmed/NonScraped/Unreadable → patch
- sweep_internal takes resume param: true when dispatching from existing mapfile
- Verified: Pass 1 from 30-100% completes in ~20 min with correct jumps through 3 damage zones
- CopyOptions: replace resume/skip_on_error/batch_sectors with single multipass bool
- Disc::copy() auto-detects pass from mapfile state: no mapfile or NonTried → sweep, only NonTrimmed/Unreadable → patch
- Fix bug where mapfile with NonTried regions incorrectly dispatched to patch mode
- SectorReader::set_speed() default method, Drive impl sends SET CD SPEED
- On damage zone entry: set_speed(0x0000) for better error recovery
- On damage zone exit (50 consecutive good): set_speed(0xFFFF) to restore max speed
- Disc::mapfile_for() returns /tmp/<name>.mapfile for null:// output
- patch_internal/sweep_internal as private helpers, CopyResult gains recovered_this_pass
- Adaptive probe algorithm in Disc::copy skip_on_error mode: after 4
consecutive errors, probe 1 sector at 256x batch (8 MB) ahead. If
good, zero-fill gap, mark NonTrimmed, jump. Clears bad zones in
seconds instead of hours.
- NOT_READY sense key (0x02) now retries up to 3x with 3s pause before
marking NonTrimmed. BU40N returns NOT READY for bad sectors, not
MEDIUM ERROR.
- PassProgress struct gains bytes_bad_total field for consumer-side
bad/retryable byte counts.
- Mapfile header version string fix: no longer duplicates 'libfreemkv v'
prefix on each write.
- Structured sense_key/asc/ascq logging in copy error path.
macOS SCSI transport rewritten from hybrid MMC+pread to single-path
raw CDB dispatch through SCSITaskDeviceInterface. All CDBs (INQUIRY,
READ, REPORT KEY, etc.) now go through ExecuteTaskSync — 1:1 with
the Linux SG_IO backend.
Key changes:
- New macos_shim.c: diskutil unmount → find IOBDServices →
ObtainExclusiveAccess → raw CDB dispatch. Eliminates Rust-side
IOKit COM vtable complexity.
- build.rs compiles macos_shim.c via cc into static lib
- macos.rs simplified to three FFI calls (open/close/execute)
- disc/mod.rs: graduated batch restore after errors, skip-ahead
through bad zones, configurable error pause
- as_bytes() requires OsStrExt import which is platform-specific
- as_encoded_bytes() is the portable API available on all platforms
- Fixes CI failure on Linux (Ubuntu) in GitHub Actions
0.13.28 - when drive returns MEDIUM ERROR (bad sector), skip the sector
and continue instead of retrying or bailing. Write zero-fill, mark as
Unreadable for pass 2+ recovery.
Closes: freemkv-private#20260427
Fix extract_scsi_context() and Error::scsi_sense() to handle Error::DiscRead
in addition to Error::ScsiError, so is_marginal_read() works for DiscRead
errors and disc::copy() can properly route MEDIUM ERROR as a marginal
(bad sector) instead of bailing.
Closes: freemkv-private#20260427
bytes_pending was an opaque aggregate of NonTried + NonTrimmed +
NonScraped. UIs that wanted a "will retry in Pass 2-N" bucket were
stuck showing the entire unread disc as Maybe at pct=0.
Adds two granular fields to MapStats:
bytes_nontried — Pass 1 hasn't read these yet
bytes_retryable — NonTrimmed + NonScraped, Pass 2-N will retry
bytes_pending stays for back-compat (= bytes_nontried + bytes_retryable).
Also picks up the cargo fmt --check lint that's been red on main CI
since v0.13.18 (rustfmt fold differences on a few long format-string
layouts; functional no-op).
Through the entire 0.13.x line, every CHECK CONDITION reply from the
drive (the standard way SCSI tells you why a sector failed) was being
collapsed into a synthetic status=0xFF, sense_key=0 transport-wedge
sentinel and the actual sense data was thrown away. Confirmed live on
the BU40N reading Dune 2 on 2026-04-27: drive returned host_status=0,
driver_status=8, status=2, exec_elapsed_ms=1416 on every bad sector
— a clean CHECK CONDITION carrying full sense data — and Disc::copy
was bailing on it as if the bridge had wedged.
Root cause: scsi/linux.rs's wedge check was
`host_status != 0 || driver_status != 0`
SG's DRIVER_SENSE bit (0x08) is set on every CHECK CONDITION reply
just to flag "sense buffer is populated" — it's not a transport
failure on its own. Pre-fix we conflated the two and silently lost
every drive-reported error reason. macOS and Windows backends had
the same shape: they extracted sense_key only, dropping ASC/ASCQ.
API restructure (clean separation):
Error::ScsiError {
opcode: u8,
status: u8, // 0xFF = synthetic transport-failure
sense: Option<ScsiSense>, // None ⇔ no sense delivered
}
pub struct ScsiSense { sense_key: u8, asc: u8, ascq: u8 }
impl ScsiSense {
pub fn is_marginal(&self) -> bool // keys 0/1/3/B
pub fn is_medium_error(&self) -> bool
pub fn is_hardware_error(&self) -> bool
pub fn is_unit_attention(&self) -> bool
pub fn is_data_protect(&self) -> bool
pub fn is_not_ready(&self) -> bool
pub fn is_illegal_request(&self) -> bool
pub fn is_aborted_command(&self) -> bool
}
impl Error {
pub fn scsi_sense(&self) -> Option<&ScsiSense>
pub fn is_scsi_transport_failure(&self) -> bool
pub fn is_marginal_read(&self) -> bool
}
SCSI protocol constants (SCSI_STATUS_*, SENSE_KEY_*) moved from
error.rs to scsi/mod.rs where they belong alongside SCSI_INQUIRY,
SCSI_READ_10, etc. parse_sense replaces parse_sense_key (returns the
full triple, not just the key); inline tests now exercise ASC/ASCQ
extraction at the right offsets for both descriptor (0x72/0x73) and
fixed (0x70/0x71) sense formats.
Disc::copy + Disc::patch sense-aware dispatch:
- marginal sense (MEDIUM ERROR / ABORTED COMMAND / RECOVERED ERROR
/ NO SENSE) → engage hysteresis (Block→Single, bpt=1)
- non-marginal sense (HARDWARE / DATA PROTECT / UNIT ATTENTION /
NOT READY / ILLEGAL REQUEST / transport failure / kernel
IoError) → bail with full sense info preserved; caller (autorip)
surfaces "physical replug" / "drive failing" / "media changed"
Pre-fix: every CHECK CONDITION → 0xFF synthetic → Disc::copy bailed
→ bytes_good froze at the bad zone. The hysteresis from v0.13.22
was correct but never got to run. This release unblocks it.
Disc::patch's wedged_threshold (50 consecutive failures) stays as
defense-in-depth for chains of marginal failures; a single
non-marginal sense now short-circuits it.
New phase=bail trace event records the bail reason with the sense
triple. phase=transport_err remains for genuine bridge wedges /
kernel timeouts; phase=scsi_err carries the parsed sense_key, asc,
ascq for drive-reported errors.
All 350 tests pass. Clippy clean across all targets.
The v0.13.21 bisect-on-fail recovery was correct (100% of recoverable
sectors picked up) but slow on dense damage clusters. Live test on
Dune 2 v0.13.21 burned ~30 s per damaged 60-block — paying a ~5 s
kernel ABORT/timeout at every level of a log₂(60) ≈ 6 deep DFS, on
the failing branch each time.
Replaced with a two-state hysteresis machine in Disc::copy:
Block(batch):
read(batch) ok → write, advance, stay Block
read(batch) fail → switch to Single, retry SAME range at bpt=1
Single:
read(1) ok → write, consecutive_good++
if consecutive_good >= BPT1_EXIT_THRESHOLD:
switch to Block, reset counter
read(1) fail → mark NonTrimmed, consecutive_good = 0
BPT1_EXIT_THRESHOLD = 10_000 sectors (= 20 MB clean run). Calibrated
from the 2026-04-26 BU40N empirical probe data; tunable.
Per-block math on a damaged 60-block with 1 truly bad sector:
Bisect (v0.13.21): ~30 s (5 s × 6 levels)
Hysteresis (v0.13.22): ~10 s (5 s bpt=batch fail
+ 59 × 1 ms good
+ 1 × 5 s bad)
Inside a damaged cluster spanning many 60-blocks the win compounds:
hysteresis pays the bpt=batch fail cost ONCE on entry, then stays at
bpt=1 across the cluster; bisection re-paid it every 60 sectors. For
Dune 2's ~1248-sector boundary cluster that's ~21 fewer 5-sec
kernel timeouts ≈ 100 s saved per pass.
Telemetry: new phase=mode_change trace event with from, to, lba, and
consecutive_good. Replaces v0.13.21's phase=bisect. Worklist DFS is
gone — single iterative for s in 0..count on the failure path.
Test rename, same fixture and same 100% recovery expectation:
test_disc_copy_bisect_recovers_via_single_sector_reads
→ test_disc_copy_hysteresis_recovers_via_single_sector_reads
Also adds DamageSeverity (Clean / Cosmetic / Moderate / Serious) +
classify_damage(bad_sectors, lost_ms), re-exported from libfreemkv,
so applications can render structured severity instead of formatting
their own from raw counters.
Fixes the BU40N wedge cycle that has been chasing us through
v0.13.18-20. Two changes, both backed by empirical live-hardware
probes recorded in freemkv-private/docs/TEST_PLAN.md:
1. scsi/mod.rs: READ_TIMEOUT_MS 1500 → 10000 ms.
Cold-start seek on the BU40N takes ~1.5 s. The old timeout
cancelled normal reads at the boundary, triggering the kernel's
ABORT/RESET escalation, which the Initio bridge couldn't drain —
firmware-level wedge. 10 s catches every legitimate slow read
(max successful ECC recovery: 2.6 s; cold-start: 1.5 s) with
margin and short-circuits truly bad sectors at ~10 s.
2. disc/mod.rs: Disc::copy bisect-on-fail (replaces skip-forward).
Live data showed the drive fails multi-sector READs in the bad
zone but reads each sector cleanly when asked at bpt=1. Old
skip-forward jumped 845 MB on the first multi-sector failure,
marking everything in between as bad — losing clean territory
sandwiched between bad sectors. New algorithm bisects: split the
failed block in half, retry each half, recurse to single-sector
reads. Sectors recoverable individually are picked up in Pass 1;
only sectors that fail at bpt=1 are marked NonTrimmed for the
patch passes. Stack-based DFS, log2(batch) = 6 levels for the
default 60-sector batch.
Multi-pass machinery is untouched. Pass 2..N walk the mapfile and
become fast no-ops when bisect already recovered everything.
Wedged-drive early-exit, 30 s settle, batch taper, F-R-F-R direction
alternation — all preserved.
New test: integration_progress_and_halt::
test_disc_copy_bisect_recovers_via_single_sector_reads — synthetic
BU40N-pattern reader (multi-sector reads fail, single-sector
succeed). Pre-patch: lost everything to skip-forward. Post-patch:
100 % bytes_good. Plus the 10 sense-key parser tests from the
0.13.20 test-coverage pass.
Empirical recovery on Dune 2 UHD on the BU40N (per TEST_PLAN.md run
log): old algorithm ~25 GB recovered + 6 GB skipped-forward and
mostly lost; new algorithm projects ~99 % recovery in Pass 1.
Audits + raw probe data:
- freemkv-private/docs/TEST_PLAN.md (run log)
- freemkv-private/docs/audits/2026-04-26-scsi-architecture-research.md
- scsi/linux.rs: full rewrite from async write/poll/read+1.5s timeout+
close-on-timeout to one synchronous ioctl(fd, SG_IO, &hdr). Kernel
honors hdr.timeout and runs its own ABORT/RESET escalation. Errors
check host_status and driver_status (both 0xFF-synthesised) plus
status. Sense-key parser handles descriptor (0x72/0x73) + fixed
(0x70/0x71) formats. Deleted fd_recovery, bg close+open thread, fd
swap dance. -331/+155 lines.
- scsi/macos.rs: try_recover() removed (userspace handle-recovery on
task failure was the same anti-pattern stripped from Linux). bsd_name
field deleted. Errors bubble up directly.
- scsi/windows.rs: try_recover() removed, wide_path field deleted,
INVALID_HANDLE guard removed.
- scsi/mod.rs: parse_sense_key() helper extracted (used by all three
platforms now — single canonical sense-key parse rather than three
inlined copies). +10 unit tests covering descriptor format, fixed
format, truncated buffers, unknown response codes.
- drive/mod.rs: Drive::reset() deleted (escalating eject + STOP/START +
reinit recovery — per audit, kernel handles its own escalation;
userspace shouldn't).
pub fn find_drives() -> Vec<Drive> deleted (opened N drives just to
throw most away). find_drive() now uses discover_drives() directly.
wait_ready() simplified — drops the reset path on sense_key=5,
just keeps polling TUR for 60 iterations.
- lib.rs: find_drives re-export removed.
- benches/sgio_read.rs: switched to find_drive() (no longer iterates a
drive list).
Net: 9 files changed, 226 insertions(+), 473 deletions(-). 329 tests
pass, clippy -D warnings clean. No consumer breakage (CLI, autorip,
bdemu compile + test green).
Architecture decision documented in
freemkv-private/docs/audits/2026-04-26-scsi-architecture-research.md
(primary-source survey of MakeMKV, sg_dd, ddrescue, and the kernel
mid-layer's own scsi_eh.rst escalation ladder).
Pre-0.13.16 the rip API leaked internal mapfile concepts (pos,
bytes_good, work_done, bytes_pending, Finished/NonTrimmed) into per-pass
positional callbacks Fn(u64, u64, u64). Consumers reinvented the math
each time, and v0.13.15's UI bug surfaced exactly because of this —
autorip's web JS computed pct from bytes_good while the backend
computed from pos, silent drift, frozen UI bar.
This release replaces both Disc::copy::on_progress and
Disc::patch::on_progress callbacks with a single Progress trait +
PassProgress struct (new progress module).
pub struct PassProgress {
pub kind: PassKind, // Sweep | Trim {reverse} | Scrape {reverse} | Mux
pub work_done: u64,
pub work_total: u64,
pub bytes_good_total: u64,
pub bytes_total_disc: u64,
}
pub trait Progress {
fn report(&self, p: &PassProgress);
}
impl<F: Fn(&PassProgress)> Progress for F { ... } // closures work directly
CopyOptions::on_progress and PatchOptions::on_progress are renamed to
progress: Option<&dyn Progress>. Closure callers update trivially via
the blanket impl; struct callers gain a clean named-field shape with no
positional-arg confusion.
PassKind carries the semantic (sweep vs trim vs scrape vs mux) so
consumers can label phases without reinventing detection logic.
Disc::patch reports Trim {reverse} for retry passes with block_sectors
>= 2 and Scrape {reverse} when block_sectors == 1. Direction comes
through reverse: bool. Mux variant is reserved for v0.13.17 when the
mux pipeline emits progress.
Tests + clippy clean across all 4 crates.
Breaking: CopyOptions::on_progress + PatchOptions::on_progress now take
Fn(bytes_good, pos, total). Consumers display `pos` for "% swept" — the
true Pass 1 progress that advances through skip-forward bad zones, where
bytes_good (Finished sectors only) freezes. v0.13.14 live trace proved
the existing UI was lying for ~14 minutes about Dune 2 being "stuck at
30%" while Pass 1 was actually 83% through the disc via skip-forward.
PatchOptions::reverse: walk bad ranges from highest LBA to lowest. For
drives that wedge after a forward read of a bad sector, approaching the
post-bad-zone NonTrimmed range from end-of-disc reads good sectors before
the drive sees a bad one. Hypothesis informed by the BU40N + Initio
bridge live data — Pass 2 forward saw zero successful reads in 7 min
while Pass 1's pos walked all the way to end-of-disc.
PatchOptions::wedged_threshold: > 0 → exit early after that many
consecutive failures with zero successes in the same pass. Saves the
wallclock budget for productive grinding when the drive has wedged on
the bad zone for THIS pass; a different direction or block size in the
next pass may still recover. New PatchResult::wedged_exit reports it.
Trace: patch_start (block_sectors, recovery, reverse, wedged_threshold,
num_ranges) and patch_done (blocks_attempted, blocks_read_ok,
blocks_read_failed, wedged_exit, halted, bytes_recovered) at the
freemkv::disc target.
v0.13.12 shipped the async fd_recovery design but a live test on Dune 2
showed Pass 1 sat for 14 minutes with bytes_good=0 — the inner loop iterates
(throttled on_progress log fires every 78s) but each iteration evidently
takes ~60s instead of the microseconds the design promises on fast-fail.
Without trace-level telemetry at the SCSI + Disc::copy boundaries we
can't tell where the time goes.
This release is instrumentation only — no behavior change.
- New dep: tracing 0.1. Per CLAUDE.md, debug/trace logging is allowed in
libfreemkv (the no-English rule applies to errors). Consumers wire a
tracing subscriber.
- SgIoTransport::execute (Linux): trace at every state transition (entry,
recovery_swap_ok, recovery_pending, write_ok / write_err, poll_done,
timeout_spawn_recovery, scsi_err, read_err, ok). Each event includes
opcode + elapsed timing. The bg recovery thread also traces close_ms +
open_ms so we can see if the kernel really takes 60s to close+open on a
wedged Initio bridge.
- Disc::copy: trace at copy_start, outer_loop, region_enter, every 100
inner-loop iterations (iter_progress with pos/region_end/skip_size/
bytes_good/read_ok_count/read_err_count/last_read_ms/copy_elapsed_ms),
copy_done.
- All trace events use targets `freemkv::scsi` and `freemkv::disc` so
consumers can filter by subsystem (e.g. autorip /api/debug?q=freemkv::scsi).
Next: run the live test on Dune 2 again, read the autorip JSONL log,
diagnose why each iter is slow, fix the actual bug.
matthew