Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
26e2d847d5 | ||
|
|
4442fa2df6 | ||
|
|
23dc55661c | ||
|
|
fbdb50c79f | ||
|
|
7cc74f0087 | ||
|
|
3aa1e528c5 | ||
|
|
7c6b0f82ab | ||
|
|
4226a53e73 | ||
|
|
92f34a289e | ||
|
|
fdbe469d50 | ||
|
|
4ec75a03f2 | ||
|
|
99236ffd55 | ||
|
|
9e3327a569 | ||
|
|
0081955686 | ||
|
|
3244fdd683 | ||
|
|
ef5487e5a5 | ||
|
|
e7dffa63d2 | ||
|
|
e5989c8270 | ||
|
|
e1c938b82a | ||
|
|
ed801a708b | ||
|
|
d94451954c | ||
|
|
385c9f094c | ||
|
|
9f2a13739d | ||
|
|
d9ce69bc9d | ||
|
|
7cd2c937ed | ||
|
|
107524da5e | ||
|
|
055a3c5276 | ||
|
|
50b04790ce | ||
|
|
a00b2c884b | ||
|
|
44e2c64d7e | ||
|
|
d6535b8f57 | ||
|
|
90ab00ed45 | ||
|
|
6bfd7dfd13 | ||
|
|
b9a7f601d4 | ||
|
|
cf0a61f8b4 | ||
|
|
f98f07b2d3 | ||
|
|
b53454fa09 | ||
|
|
f28b6ee6d9 | ||
|
|
d9d778fa64 | ||
|
|
bbfb887a35 | ||
|
|
9884346c24 | ||
|
|
766b7c6636 | ||
|
|
925c30686b | ||
|
|
f52e4c5d22 | ||
|
|
8d67790d2e | ||
|
|
246a439990 | ||
|
|
bd19041648 | ||
|
|
198268b725 | ||
|
|
31424c8203 | ||
|
|
c747ecc589 | ||
|
|
283a561c12 | ||
|
|
2667d68675 | ||
|
|
57f2c22e30 | ||
|
|
5f3545d244 | ||
|
|
6e17ef0859 | ||
|
|
6ec97af104 | ||
|
|
3a6c1aa5a3 | ||
|
|
1ba3264747 | ||
|
|
ae2909fe8d | ||
|
|
6a3d19a453 | ||
|
|
a35596d2d1 |
55 changed files with 10165 additions and 894 deletions
4
.github/workflows/ci.yml
vendored
4
.github/workflows/ci.yml
vendored
|
|
@ -14,6 +14,10 @@ jobs:
|
|||
with:
|
||||
components: clippy, rustfmt
|
||||
- run: cargo fmt --check
|
||||
# libfreemkv is a library — Cargo.lock is gitignored. --locked
|
||||
# would always fail on a fresh runner because there's no committed
|
||||
# lockfile to lock against. The binary crates (freemkv, autorip,
|
||||
# bdemu) track Cargo.lock and DO use --locked.
|
||||
- run: cargo clippy -- -D warnings
|
||||
|
||||
test:
|
||||
|
|
|
|||
2
.github/workflows/release.yml
vendored
2
.github/workflows/release.yml
vendored
|
|
@ -28,6 +28,8 @@ jobs:
|
|||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: dtolnay/rust-toolchain@1.86.0
|
||||
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
|
||||
# would always fail (no lockfile to lock against on a fresh runner).
|
||||
- run: cargo test
|
||||
|
||||
publish:
|
||||
|
|
|
|||
391
CHANGELOG.md
391
CHANGELOG.md
|
|
@ -1,5 +1,396 @@
|
|||
# Changelog
|
||||
|
||||
## 0.18.4 (2026-05-09)
|
||||
|
||||
### Build / CI hardening — no library code changes
|
||||
|
||||
- All `cargo` invocations in `.github/workflows/*.yml` now use
|
||||
`--locked`. Previously `cargo build` / `cargo test` could silently
|
||||
re-resolve `Cargo.lock` if a lock-pinned version wasn't yet on
|
||||
crates.io — masking dependency races between same-tag releases of
|
||||
sibling crates. The 0.18.3 release hit this: autorip's docker image
|
||||
built `libfreemkv v0.18.2` because libfreemkv 0.18.3 hadn't
|
||||
published yet at the time autorip's CI ran, and cargo silently fell
|
||||
back to the previous version. With `--locked`, that race now hard-
|
||||
fails the build with a clear "package X is not in registry" error
|
||||
and we retrigger after the upstream lands.
|
||||
|
||||
- `cargo publish --locked` in the release workflow guards against
|
||||
publishing a lockfile-mismatched release.
|
||||
|
||||
## 0.18.3 (2026-05-09)
|
||||
|
||||
### Behaviour change
|
||||
|
||||
- **`Disc::titles[0]` is now the canonical main feature on branching
|
||||
discs, not the longest playlist.** Previously titles were sorted
|
||||
purely by `duration_secs` descending, which puts a "play-all"
|
||||
virtual playlist (alternate angles / seamless branching) at index 0
|
||||
on UHDs that ship one — its inflated duration overshoots the real
|
||||
movie. Concrete example: *The Amateur (2025)* 4K UHD on a 58.5 GB
|
||||
disc has Title 1 = `00020.mpls` 4h13m / 92.4 GB / 253 clips
|
||||
(impossible — the size exceeds the disc capacity, proving it's a
|
||||
virtual composite) and Title 2 = `00800.mpls` 2h02m / 57.2 GB /
|
||||
1 clip (the actual film, matching TMDB).
|
||||
|
||||
New sort priority:
|
||||
1. Real titles (`size_bytes <= capacity_bytes`) before virtual
|
||||
composites.
|
||||
2. Among real titles, fewer clips first (1-clip wins).
|
||||
3. Tiebreak on longer duration first.
|
||||
|
||||
**Migration:** consumers calling `disc.titles.first()` /
|
||||
`disc.titles[0]` automatically get the corrected title — no code
|
||||
changes needed. CLI users invoking `freemkv -t 1 disc:// …` now hit
|
||||
the actual main feature on branching discs (this was the user-
|
||||
visible bug). On non-branching discs the order is unchanged.
|
||||
|
||||
The comparator is exposed as
|
||||
[`Disc::canonical_title_order`](#) for callers that need to
|
||||
re-sort a custom title set with the same logic.
|
||||
|
||||
Regression tests: `disc::tests::canonical_order_*` (three cases:
|
||||
branching-UHD, normal disc, clip-count tiebreak).
|
||||
|
||||
## 0.18.2 (2026-05-09)
|
||||
|
||||
### Bug fixes
|
||||
|
||||
- **Nav-file scramble during AACS rip** (`decrypt::decrypt_sectors`). 0.18's
|
||||
`DecryptingSectorSource` decorator broadened the call surface of
|
||||
`decrypt_sectors` to every sector flowing through sweep, including UDF
|
||||
navigation files (MPLS playlists, CLPI clip-info). The byte-0 heuristic
|
||||
in `aacs::is_unit_encrypted` correctly fires on m2ts source-packet copy
|
||||
markers but false-positives on any binary file whose first byte happens
|
||||
to have the top 2 bits set — most notably MPLS files (start with 'M' =
|
||||
0x4D) and CLPI files (start with 'H' = 0x48). `decrypt_unit_full`
|
||||
already self-checks the result via TS-sync verification and returns
|
||||
`false` on a misfire, but the chunk had been mutated by then;
|
||||
`decrypt_sectors` discarded the return value, leaving scrambled bytes
|
||||
in the ISO. Fix: snapshot the chunk before decryption and restore on
|
||||
verification failure (same pattern `decrypt_unit_try_keys` already used
|
||||
for multi-key discs). Symptom: `freemkv info iso://UHD.iso` and
|
||||
`iso:// → mkv://` returned E6009 NoStreams on freshly-ripped UHD ISOs;
|
||||
affected only the iso-source path, not the disc-source path or the
|
||||
m2ts video payload itself. Regression test:
|
||||
`decrypt::tests::nav_file_unit_survives_decrypt_attempt`.
|
||||
|
||||
- **Sweep progress display can regress to zero** (`Disc::sweep`). The
|
||||
consumer-side `bytes_good` snapshot lags producer-side `bytes_done`
|
||||
whenever the consumer is behind on draining the work channel. Until
|
||||
the first snapshot arrived, the placeholder branch reported
|
||||
`bytes_done` correctly; once a stale snapshot landed, the report
|
||||
switched to `snap.bytes_good` and could regress below `bytes_done`.
|
||||
Fix: `bytes_good_total = max(snap.bytes_good, bytes_done)`, so the
|
||||
user-visible counter never moves backward.
|
||||
|
||||
## 0.18.1 (2026-05-09)
|
||||
|
||||
### I/O stack redesign — primitives over orchestration
|
||||
|
||||
0.18 reshapes the read/write surface from "library does the multipass dance"
|
||||
to "library hands the caller flat verbs and a few composable primitives." All
|
||||
in-tree consumers (autorip, the `freemkv` CLI) drove their own multipass
|
||||
loops in 0.18 rounds 1-3; 0.18.1 lands the cleanup with the deprecated names
|
||||
still alive for one minor-version window.
|
||||
|
||||
The 0.18 design notes are in
|
||||
`(internal)/memory/0_18_redesign.md` (private) — this entry sticks to
|
||||
what changed at the public surface.
|
||||
|
||||
#### Flat verbs
|
||||
|
||||
`Disc::sweep` is the forward Pass 1. `Disc::patch` is one retry pass over the
|
||||
mapfile. Neither knows about pass index, retry budget, or accept-loss policy
|
||||
— the caller invokes them in whatever sequence its use case dictates. The
|
||||
old multipass-aware `Disc::copy` dispatcher is deprecated and slated for
|
||||
deletion in 0.18.2; no in-tree caller still uses it.
|
||||
|
||||
#### Trait splits — direction-typed at compile time
|
||||
|
||||
- `pes::Stream` (combined read+write) is split into `FrameSource` and
|
||||
`FrameSink`. Calling `read()` on a write-only sink is now a compile error,
|
||||
not the runtime `E9001` (`StreamWriteOnly`) it used to be.
|
||||
- `SectorReader` is split into `SectorSource` (read) and `SectorSink`
|
||||
(write). `Drive` impls `SectorSource` only; `FileSectorSource` /
|
||||
`FileSectorSink` replace `FileSectorReader` for ISO-backed I/O.
|
||||
- A blanket impl bridges legacy `SectorReader` callers onto the new
|
||||
`SectorSource` so existing code keeps compiling through the deprecation
|
||||
window.
|
||||
|
||||
#### New primitives
|
||||
|
||||
- `Halt` — one cancellation token (cloneable, `Arc<AtomicBool>` under the
|
||||
hood) replaces the three near-duplicate halt flags scattered through the
|
||||
workspace. Threaded through every long-running loop.
|
||||
- `Pipeline<I, R>` + `Sink<I>` — generic producer/consumer primitive in
|
||||
`crate::io`. Replaces the bespoke `disc/sweep_pipeline.rs` and now also
|
||||
drives `Disc::patch` and the autorip mux loop. `DEFAULT_PIPELINE_DEPTH`
|
||||
for streaming reads (`4`); `WRITE_THROUGH_DEPTH` for write-through patch
|
||||
semantics (`1`).
|
||||
- `WritebackFile` — was `crate::io::Writer`. The renamed type makes its job
|
||||
explicit: a `File` wrapper that runs continuous `sync_file_range` +
|
||||
`posix_fadvise(DONTNEED)` to keep the kernel dirty-page cache bounded on
|
||||
long sequential writes.
|
||||
- `DecryptingSectorSource<S>` — a single decorator wrapping any
|
||||
`SectorSource` to yield plaintext sectors. One audit surface for AACS /
|
||||
CSS / passthrough; the previous two-site decrypt (sweep producer +
|
||||
`DiscStream` demux) is gone.
|
||||
|
||||
#### Throughput
|
||||
|
||||
The round-2 producer/consumer split is now applied uniformly to sweep,
|
||||
patch, and mux. Mux on NFS-staged UHD measured ~16 MB/s sustained on the
|
||||
test bed (was ~12 MB/s pre-round-2).
|
||||
|
||||
#### Module reorg
|
||||
|
||||
- `sector/` and `io/` are now module directories.
|
||||
- `disc/sweep.rs`, `disc/patch.rs`, and `disc/mapfile.rs` split out of the
|
||||
monolithic `disc/mod.rs`.
|
||||
|
||||
#### Renames (no behavior change)
|
||||
|
||||
- `crate::io::Writer` → `crate::io::WritebackFile`.
|
||||
- `Apply` → `Flow` (`Sink::apply` return value).
|
||||
- `DEFAULT_DEPTH` → `DEFAULT_PIPELINE_DEPTH`.
|
||||
- `PatchOpts` → `PatchOptions`.
|
||||
|
||||
#### Deprecated (alive in 0.18.1, deletion target 0.18.2)
|
||||
|
||||
`Disc::copy`, `pes::Stream`, `SectorReader`, `FileSectorReader`,
|
||||
`CopyOptions`, `CopyResult`, `DiscStream::set_halt`. Each compiles with a
|
||||
deprecation warning; all in-tree call sites have migrated.
|
||||
|
||||
## 0.17.13 (2026-05-09)
|
||||
|
||||
### Use `crate::io::Writer` uniformly for all binary file output
|
||||
|
||||
0.17.10 introduced the bounded-cache writeback wrapper (`sync_file_range`
|
||||
+ `posix_fadvise(DONTNEED)` per chunk) and 0.17.11 wired it into
|
||||
`Disc::sweep`. The other two big-write paths in the crate were still
|
||||
opening raw `std::fs::File` and would have hit the same dirty-page
|
||||
burst pathology against slow / network-attached staging.
|
||||
|
||||
This release threads `crate::io::Writer` through the remaining sites:
|
||||
|
||||
- **`Disc::patch`** (`disc/mod.rs:1981`): the ISO file reopened for
|
||||
Pass-N recovery now wraps in `Writer` before any seek / write.
|
||||
Recovery writes are sparse, but the wrapper costs nothing when
|
||||
there's no chunk crossing — and on heavily-damaged discs it
|
||||
matters as patch accumulates GB of recovered data.
|
||||
- **MKV mux output** (`mux/resolve.rs:243`): `BufWriter::with_capacity`
|
||||
now wraps `Writer::new(file)` instead of a raw `File`. UHD MKVs
|
||||
routinely exceed 70 GB of sequential writes; pre-0.17.13 those
|
||||
bursts went straight to the kernel writeback queue.
|
||||
- **M2TS mux output** (`mux/resolve.rs:251`): same change for
|
||||
parity with the MKV path — anyone using `m2ts://` URLs gets it
|
||||
too.
|
||||
|
||||
No new public surface. `Writer::sync_all()` is called from `Disc::patch`
|
||||
and `sweep_pipeline`'s consumer at end of pass; mux exits via Drop on
|
||||
the wrapping `BufWriter`, which propagates flush down to `Writer`'s
|
||||
final `note_progress` (kernel finishes the in-flight chunk on file
|
||||
close — there's no explicit `sync_all` for mux today, same behaviour
|
||||
as before).
|
||||
|
||||
## 0.17.12 (2026-05-09)
|
||||
|
||||
### Mapfile time-batched persistence — unblock NFS staging
|
||||
|
||||
Pre-0.17.12 every `Mapfile::record()` wrote the entire mapfile to disk
|
||||
via tempfile-create + buffered-write + atomic-rename. On local LVM
|
||||
that's effectively free (page cache + microsecond-scale renames). On
|
||||
NFS (autorip's intended staging path for centralised media) each
|
||||
record() became three RPCs through the unraid user share's shfs-fuse
|
||||
layer — measured end-to-end at multiple ms each. With ~170 record()
|
||||
calls per second of sustained sweep, the mapfile path alone burned
|
||||
multiple seconds of wall time per real-world second of work, dragging
|
||||
the rip from ~11 MB/s on local to ~1.5 MB/s on NFS.
|
||||
|
||||
Fix: time-batch the persistence inside `Mapfile`.
|
||||
|
||||
- `record()` always updates in-memory state and stats (so callers'
|
||||
`stats()` reads stay coherent in the same process).
|
||||
- The `write_to_disk()` rename only fires when ≥ `FLUSH_INTERVAL`
|
||||
(1 s) has elapsed since the last persist.
|
||||
- New `flush()` method forces an out-of-band persist; called by
|
||||
`sweep_pipeline`'s consumer at end-of-sweep and by `Disc::patch` at
|
||||
end-of-patch, after the file's `sync_all()`.
|
||||
- `Drop` impl best-effort flushes so an early-return / unwind doesn't
|
||||
silently lose pending state.
|
||||
|
||||
Crash-safety changes from "lose at most one block" to "lose at most
|
||||
1 s of recorded progress" — the ISO file's payload bytes are
|
||||
unaffected; only the mapfile's authority over which sectors are
|
||||
already-good is at risk, and a resume re-reads anything Pass 1 had
|
||||
already covered. Acceptable for a 7× throughput recovery on the
|
||||
target deployment.
|
||||
|
||||
The internal `round_trip_load` test now calls `flush()` before
|
||||
`Mapfile::load` to read back what the in-memory state asserts.
|
||||
External patch / copy tests are unaffected: `patch` and
|
||||
`sweep_pipeline` flush at completion before returning, so any
|
||||
`Mapfile::load` at the call-site sees fully persisted state.
|
||||
|
||||
## 0.17.11 (2026-05-09)
|
||||
|
||||
### Sweep producer/consumer split — overlap drive read with file write
|
||||
|
||||
Pre-0.17.11 the sweep loop ran strictly serialised: SCSI read → decrypt
|
||||
→ seek + write → mapfile.record → next read. The drive idled for the
|
||||
post-read work; throughput capped at the **sum** of both costs. On a
|
||||
healthy disc that's ~7-12 ms read + ~5-15 ms write/record per 64 KB
|
||||
batch — limiting sustained throughput to ~10-12 MB/s on the test bed
|
||||
(BU40N + UHD inner zone), well below the drive's ceiling of ~14-16 MB/s.
|
||||
|
||||
This release decouples them with a producer / consumer split:
|
||||
|
||||
- **Producer** (caller's thread): owns the `SectorReader`, the entire
|
||||
`read_error` state machine (Retry, Bisect, SkipBlock, JumpAhead,
|
||||
AbortPass), `set_speed` damage-zone transitions, halt check, and
|
||||
decrypt. Hands plaintext bytes to the consumer.
|
||||
- **Consumer** (one spawned thread): owns the `crate::io::Writer` and
|
||||
the `Mapfile`. Receives `WorkItem` messages and applies the file
|
||||
write + mapfile record per item.
|
||||
- **Channel**: bounded `mpsc::sync_channel(4)` — natural back-pressure
|
||||
via blocking `send` when consumer falls behind.
|
||||
|
||||
While the consumer is writing batch N to disk and updating the mapfile,
|
||||
the producer is already reading batch N+1 from the drive. Steady-state
|
||||
throughput is now bound by the slower of the two pipelines (the drive,
|
||||
on a healthy disc) instead of their sum.
|
||||
|
||||
Side effects of the refactor:
|
||||
|
||||
- **Bisect path now decrypts.** Pre-0.17.11 the bisect inner loop wrote
|
||||
raw cyphertext when `decrypt=true` and a single sector was recovered
|
||||
via single-sector retry — a quiet correctness bug exercised only by
|
||||
the (rare) batch-fail-then-bisect-succeed path on encrypted discs.
|
||||
The new producer-side decrypt covers both the main success path and
|
||||
the bisect inner success path.
|
||||
- All `read_error::ReadCtx` state stays single-threaded on the producer
|
||||
(damage window, jump multiplier, consecutive-good count, etc.). No
|
||||
locking added.
|
||||
- Mapfile remains single-writer (consumer-only). No locking.
|
||||
- Halt-flag responsiveness unchanged: producer breaks the loop on
|
||||
signal, sends `Finish`, consumer drains its ≤4 in-flight items and
|
||||
exits within ~1 batch (~12 ms typical).
|
||||
- BU40N + Initio bridge wedge concern unchanged: still one SCSI command
|
||||
in flight, error-path timing identical, no new retry logic.
|
||||
|
||||
New module: `src/disc/sweep_pipeline.rs` (`WorkItem`, `ProgressSnapshot`,
|
||||
`ConsumerInputs`, `spawn_consumer`, `consumer_loop`, send/recv helpers).
|
||||
Public API surface unchanged — `Disc::copy` / `CopyOptions` /
|
||||
`CopyResult` look identical to callers.
|
||||
|
||||
Patch (Pass N) is **not** affected by this release. Patch is bound by
|
||||
drive recovery time (60 s timeouts on bad sectors), not the read↔write
|
||||
serialisation; a similar split there would yield negligible benefit.
|
||||
|
||||
## 0.17.10 (2026-05-09)
|
||||
|
||||
### Bounded-cache writeback for big sequential writes
|
||||
|
||||
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
|
||||
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
|
||||
settings. Cause: the kernel's `vm.dirty_ratio` (~20 % of RAM) lets
|
||||
hundreds of MB of dirty pages accumulate, then bursts a flush at
|
||||
99 % disk utilisation that blocks app writes for ~1 s. Confirmed
|
||||
empirically on the BU40N test bed: dirty pages grew 112 → 563 MB
|
||||
between bursts; lowering `vm.dirty_bytes` to 64 MB at the host
|
||||
sysctl level eliminated the dips.
|
||||
|
||||
This release ships the equivalent inside libfreemkv so users don't
|
||||
need to tune the host kernel:
|
||||
|
||||
- New `crate::io::Writer` — drop-in `File` wrapper implementing
|
||||
`Write` + `Seek`. Wraps a per-platform `WritebackPipeline` that on
|
||||
Linux schedules `sync_file_range(WRITE)` + lagging
|
||||
`sync_file_range(WAIT_AFTER)` + `posix_fadvise(DONTNEED)` calls in
|
||||
32 MB chunks, keeping dirty cache bounded at ~64 MB. macOS and
|
||||
Windows ship a no-op stub — their default cache policies don't
|
||||
exhibit the same pathology for our access pattern.
|
||||
- Disc::sweep wraps its output `File` in `Writer`. No changes to the
|
||||
loop body — `Writer` forwards `seek`/`write_all` to `File` and
|
||||
drives the pipeline transparently.
|
||||
- Module is purpose-built for any large sequential output (sweep,
|
||||
patch, future mux) — they can adopt `crate::io::Writer` with a
|
||||
one-line wrapper and inherit the same behaviour.
|
||||
|
||||
## 0.17.7 (2026-05-08)
|
||||
|
||||
### Sync release — no functional libfreemkv changes
|
||||
|
||||
Version bump to keep the four freemkv crates at unified versioning.
|
||||
autorip shipped UI smoothness + audit fixes today (v0.17.6, v0.17.7);
|
||||
libfreemkv carries no code changes, but is bumped + republished to
|
||||
crates.io so downstream `libfreemkv = "0.17"` consumers stay aligned
|
||||
on the latest patch version. See autorip CHANGELOG for the
|
||||
user-visible work in this release window.
|
||||
|
||||
## 0.17.5 (2026-05-08)
|
||||
|
||||
### Pass N recovery — kernel block-device fallback + per-range fixes
|
||||
|
||||
Live testing on direct-SATA BU40N + AACS-encrypted UHD disc revealed that
|
||||
the v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-sector
|
||||
LBAs that `dd if=/dev/sr0` recovers on the same drive. This release closes
|
||||
that gap and fixes several adjacent bugs that were silently capping
|
||||
recovery.
|
||||
|
||||
- **`/dev/sr0` pread fallback in `Drive::read` (Linux)**: when a SCSI READ
|
||||
via `/dev/sg*` returns Err, fall back to `posix_fadvise(POSIX_FADV_DONTNEED)
|
||||
+ pread()` against the corresponding block device. The kernel `sr_mod`
|
||||
driver runs ~5 internal retries per command without the per-attempt
|
||||
error-escalation overhead that userspace SG_IO retries pay, which is the
|
||||
source of dd's recovery advantage. End-to-end byte-verification confirms
|
||||
the fallback path returns real disc data (md5-equivalent to fresh dd from
|
||||
the same drive session). The block fd is opened in `Drive::open` by
|
||||
resolving `/sys/class/scsi_generic/sgN/device/block`; best-effort, with no
|
||||
fallback if open fails.
|
||||
|
||||
- **`Disc::patch` per-range watchdog fix**: when a range hit `MAX_RANGE_SECS`,
|
||||
the old code did `wedged_exit = true; break 'outer;` — a single slow range
|
||||
killed the entire patch. Now `break;` (skip this range, advance the outer
|
||||
for loop). Pre-fix, patch was dying after 4 sectors of the first slow
|
||||
range and never reaching the other 46.
|
||||
|
||||
- **Per-sector range budget**: replaced flat `MAX_RANGE_SECS = 180` with
|
||||
`range_budget_secs = (range_sectors × SECONDS_PER_SECTOR).min(RANGE_BUDGET_CAP_SECS)`.
|
||||
Tiny ranges exit fast (1-sector range = 25 s); medium ranges get
|
||||
proportional time (51-sector range ≈ 1275 s); large ranges still bounded
|
||||
by the 1800 s cap so they cannot monopolise pass 1.
|
||||
|
||||
- **`consecutive_failures` resets per range**: the wedge-exit detector
|
||||
(`>= 50 consecutive failures`) is for "stuck on the same range"; pre-fix
|
||||
the counter persisted across ranges, so 50 small post-bisection ranges
|
||||
with one failure each falsely tripped the wedge mid-pass. Reset at every
|
||||
range boundary.
|
||||
|
||||
- **Reverted inline 5× retry in patch**: a brief experiment that was
|
||||
measurably harmful — each "2 s" SCSI timeout paid ~1.5 s kernel SCSI
|
||||
mid-layer error-escalation overhead, so 5× retry took ~17 s per LBA and
|
||||
triggered the per-range watchdog in 4 sectors. Restored
|
||||
`READ_RECOVERY_TIMEOUT_MS = 60_000` (the v0.17.3 baseline; the kernel-
|
||||
auto-retry pattern is now provided by the `/dev/sr0` fallback above).
|
||||
|
||||
### Empirical results (Dune Part Two UHD, BU40N direct SATA)
|
||||
|
||||
- Pass 1: **94.6 MB recovered** (28% of formerly-bad data, 33 sr0 fallback
|
||||
saves), **11 s of main-title content** restored. Patch completed all 47
|
||||
retryable ranges naturally (was wedging at range 1 of 47 in v0.17.3).
|
||||
- Pass 2 cumulative: 95.2 MB (+0.6 MB; diminishing returns curve).
|
||||
- Remaining ~233 MB on the test disc appears physically unrecoverable on
|
||||
this hardware (kernel auto-retry can't decode it either).
|
||||
|
||||
### Behavioural notes
|
||||
|
||||
- The `/dev/sr0` fallback is Linux only; macOS and Windows fall back to
|
||||
the existing single-shot SCSI behaviour. The fallback is gated to
|
||||
`recovery=true` reads (only fires from the patch path, not the sweep
|
||||
path) to avoid page-cache pressure during multi-GB sequential ripping.
|
||||
|
||||
## 0.17.0 (2026-05-04)
|
||||
|
||||
### Code quality: unwrap safety, clippy compliance, test coverage
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[package]
|
||||
name = "libfreemkv"
|
||||
version = "0.17.3"
|
||||
version = "0.18.9"
|
||||
edition = "2024"
|
||||
rust-version = "1.86"
|
||||
license = "AGPL-3.0-only"
|
||||
|
|
|
|||
34
FEATURES.md
34
FEATURES.md
|
|
@ -1,34 +0,0 @@
|
|||
# libfreemkv — Feature List
|
||||
|
||||
## v0.10.10 (current)
|
||||
|
||||
### Done
|
||||
- [x] Drive access: open, identify, unlock, firmware upload, speed calibration, eject
|
||||
- [x] 206 bundled drive profiles (MediaTek MT1959 A + B variants)
|
||||
- [x] SCSI transport: Linux SG_IO, macOS IOKit, Windows SPTI
|
||||
- [x] UDF 2.50 filesystem parser (metadata partitions, Blu-ray profile)
|
||||
- [x] MPLS playlist parser (play items, STN table, secondary streams)
|
||||
- [x] CLPI clip info parser (EP map, sector extents)
|
||||
- [x] AACS 1.0 decryption (4 VUK paths: KEYDB, media key, processing key, device key)
|
||||
- [x] AACS 2.0 SCSI handshake (P-256/SHA-256 ECDH, bus decryption, read data key)
|
||||
- [x] DVD IFO parser (VMG, VTS, PGC chains, cell addresses)
|
||||
- [x] DVD CSS decryption (bus auth, disc key via player keys, title key, sector descramble)
|
||||
- [x] CSS Stevenson plaintext attack for ISO key recovery
|
||||
- [x] MPEG-2 PS demuxer (DVD Program Stream with PES extraction)
|
||||
- [x] MPEG-2 video codec parser (sequence header, quantizer matrices, keyframe detection)
|
||||
- [x] KEYDB.cfg download, verify, save (raw TCP, zero HTTP deps)
|
||||
- [x] Content reading with adaptive batch sizing, error recovery, 12+ MB/s
|
||||
- [x] Stream labels: 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
|
||||
- [x] MKV muxer: 14 codec parsers (H.264, HEVC, MPEG-2, AC-3, DTS, TrueHD, PGS, DVD Sub, VC-1, LPCM, +4 more)
|
||||
- [x] SectorReader trait: decouples disc scanning from SCSI
|
||||
- [x] 7 stream types: Disc, ISO, MKV, M2TS, Network, Stdio, Null
|
||||
- [x] PES pipeline with unified Stream trait (any source → any dest)
|
||||
- [x] FMKV metadata header for M2TS and network streams
|
||||
- [x] Numeric error codes only (no English text in library)
|
||||
- [x] Event system for progress callbacks
|
||||
|
||||
### Planned
|
||||
- [ ] Windows testing on real hardware
|
||||
- [ ] Pioneer Renesas platform support (48 drives, need GET_CONFIG 010C)
|
||||
- [ ] TranscodeStream (ffmpeg integration)
|
||||
- [ ] ISO write with BD-compliant UDF structure
|
||||
60
README.md
60
README.md
|
|
@ -18,7 +18,7 @@ Part of the [freemkv](https://github.com/freemkv) project.
|
|||
|
||||
```toml
|
||||
[dependencies]
|
||||
libfreemkv = "0.17"
|
||||
libfreemkv = "0.18"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
|
@ -53,27 +53,48 @@ output.finish()?;
|
|||
|
||||
### Multi-pass recovery rip
|
||||
|
||||
For damaged discs, the library offers a two-stage rip model: fast sweep with zero-fill and a ddrescue-format mapfile, then targeted retry of bad ranges. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for the full architecture.
|
||||
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
|
||||
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
|
||||
loops; the multipass policy is the caller's job. See
|
||||
[`docs/rip-recovery.md`](docs/rip-recovery.md) and the design notes in
|
||||
`(internal)/memory/0_18_redesign.md`.
|
||||
|
||||
```rust
|
||||
use libfreemkv::disc::{CopyOptions, PatchOptions};
|
||||
use libfreemkv::{SweepOptions, PatchOptions};
|
||||
use libfreemkv::disc::{mapfile, mapfile_path_for};
|
||||
use std::path::Path;
|
||||
|
||||
// Pass 1: disc → ISO. Fast 64 KB reads, skip-forward on failure,
|
||||
// zero-fill bad blocks, write a sidecar .mapfile.
|
||||
let mut result = disc.copy(
|
||||
&mut drive,
|
||||
Path::new("disc.iso"),
|
||||
&CopyOptions { skip_on_error: true, skip_forward: true, ..Default::default() },
|
||||
)?;
|
||||
let iso = Path::new("disc.iso");
|
||||
|
||||
// Pass 2..N: retry bad ranges with full drive recovery.
|
||||
// Idempotent — call as many times as you want.
|
||||
while result.bytes_unreadable + result.bytes_pending > 0 {
|
||||
let pr = disc.patch(&mut drive, Path::new("disc.iso"), &PatchOptions::default())?;
|
||||
if pr.bytes_recovered_this_pass == 0 { break; }
|
||||
// Pass 1: disc → ISO. Skip-on-error, zero-fill, write the sidecar mapfile.
|
||||
disc.sweep(&mut drive, iso, &SweepOptions {
|
||||
decrypt: true,
|
||||
resume: false,
|
||||
batch_sectors: None,
|
||||
skip_on_error: true,
|
||||
progress: None,
|
||||
halt: None,
|
||||
})?;
|
||||
|
||||
// Pass 2..N: retry every non-finished range. Idempotent.
|
||||
loop {
|
||||
let map = mapfile::Mapfile::load(&mapfile_path_for(iso))?;
|
||||
let stats = map.stats();
|
||||
if stats.bytes_pending + stats.bytes_unreadable == 0 { break; }
|
||||
|
||||
let outcome = disc.patch(&mut drive, iso, &PatchOptions {
|
||||
decrypt: true,
|
||||
block_sectors: None,
|
||||
full_recovery: true,
|
||||
reverse: true,
|
||||
wedged_threshold: 50,
|
||||
progress: None,
|
||||
halt: None,
|
||||
})?;
|
||||
if outcome.bytes_recovered_this_pass == 0 { break; }
|
||||
}
|
||||
|
||||
// Then mux from the ISO via the normal stream pipeline (no drive involvement).
|
||||
// Mux from the ISO via the normal stream pipeline (no drive involvement).
|
||||
```
|
||||
|
||||
## What It Does
|
||||
|
|
@ -92,14 +113,14 @@ while result.bytes_unreadable + result.bytes_pending > 0 {
|
|||
| Stream | Input | Output | Transport |
|
||||
|--------|-------|--------|-----------|
|
||||
| DiscStream | Yes | -- | Optical drive via SCSI |
|
||||
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::copy()`) |
|
||||
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::sweep()`) |
|
||||
| MkvStream | Yes | Yes | Matroska container |
|
||||
| M2tsStream | Yes | Yes | BD transport stream with FMKV metadata header |
|
||||
| NetworkStream | Yes (listen) | Yes (connect) | TCP with FMKV metadata header |
|
||||
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
|
||||
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
|
||||
|
||||
Streams implement `pes::Stream` (frame-level). `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
|
||||
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
|
||||
|
||||
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption.
|
||||
|
||||
|
|
@ -122,7 +143,8 @@ Disc — scan titles, streams, AACS/CSS state
|
|||
└── KEYDB — download + verify + save
|
||||
|
||||
Streams — unified PES pipeline
|
||||
├── pes::Stream — read()/write() PES frames
|
||||
├── FrameSource — read() PES frames (direction-typed)
|
||||
├── FrameSink — write() PES frames (direction-typed)
|
||||
├── DiscStream — sectors → decrypt → TS demux → PES
|
||||
├── IsoStream — ISO file → decrypt → TS demux → PES
|
||||
├── MkvStream — MKV mux/demux
|
||||
|
|
|
|||
|
|
@ -44,35 +44,30 @@ while let Ok(Some(frame)) = input.read() {
|
|||
output.finish()?;
|
||||
```
|
||||
|
||||
The `pes::Stream` trait:
|
||||
The `FrameSource` and `FrameSink` traits — direction is type-checked, so
|
||||
calling `read()` on a write-only sink (or `write()` on a read-only source)
|
||||
is a compile error rather than a runtime fault:
|
||||
|
||||
```rust
|
||||
pub trait Stream {
|
||||
fn read(&mut self) -> io::Result<Option<PesFrame>>;
|
||||
fn write(&mut self, frame: &PesFrame) -> io::Result<()>;
|
||||
fn finish(&mut self) -> io::Result<()>;
|
||||
pub trait FrameSource: Send {
|
||||
fn read(&mut self) -> Result<Option<PesFrame>, Error>;
|
||||
fn info(&self) -> &DiscTitle;
|
||||
fn codec_private(&self, track: usize) -> Option<Vec<u8>>;
|
||||
fn headers_ready(&self) -> bool;
|
||||
fn codec_private(&self, track: usize) -> Option<Vec<u8>> { None }
|
||||
fn headers_ready(&self) -> bool { true }
|
||||
}
|
||||
```
|
||||
|
||||
## IOStream (byte-level API)
|
||||
|
||||
For raw byte copies (disc→ISO, resume, benchmarks). Lower level than PES.
|
||||
|
||||
```rust
|
||||
let opts = InputOptions::default();
|
||||
let mut input = open_input("iso://Disc.iso", &opts)?;
|
||||
let mut output = open_output("mkv://Movie.mkv", input.info())?;
|
||||
io::copy(&mut *input, &mut *output)?;
|
||||
output.finish()?;
|
||||
pub trait FrameSink: Send {
|
||||
fn write(&mut self, frame: &PesFrame) -> Result<(), Error>;
|
||||
fn finish(self: Box<Self>) -> Result<(), Error>;
|
||||
fn info(&self) -> &DiscTitle;
|
||||
}
|
||||
```
|
||||
|
||||
## Streams
|
||||
|
||||
All streams implement `IOStream` (byte-level) and/or `pes::Stream` (frame-level).
|
||||
URL-based resolvers open any stream by string.
|
||||
All streams implement `FrameSource` (read) and/or `FrameSink` (write); the
|
||||
directional split prevents runtime "wrong-direction" errors. URL-based
|
||||
resolvers open any stream by string.
|
||||
|
||||
| Stream | Input | Output | URL | Transport |
|
||||
|--------|-------|--------|-----|-----------|
|
||||
|
|
@ -87,21 +82,14 @@ URL-based resolvers open any stream by string.
|
|||
All URLs require a `scheme://path` format. Bare paths are rejected.
|
||||
|
||||
```rust
|
||||
// PES pipeline (frame-level)
|
||||
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
|
||||
// output() returns Box<dyn FrameSink>.
|
||||
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
|
||||
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
|
||||
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
|
||||
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
|
||||
let output = libfreemkv::output("network://10.0.0.1:9000", &title)?; // NetworkOutputStream
|
||||
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
|
||||
|
||||
// IOStream (byte-level)
|
||||
let input = open_input("disc://", &opts)?; // DiscStream
|
||||
let input = open_input("iso://Dune.iso", &opts)?; // IsoStream
|
||||
let output = open_output("iso://Copy.iso", &meta)?; // IsoStream (write)
|
||||
let output = open_output("mkv://Dune.mkv", &meta)?; // MkvStream
|
||||
let output = open_output("m2ts://Dune.m2ts", &meta)?; // M2tsStream
|
||||
let output = open_output("null://", &meta)?; // NullStream
|
||||
```
|
||||
|
||||
### FMKV Metadata Header
|
||||
|
|
@ -175,20 +163,34 @@ libfreemkv/src/
|
|||
├── lib.rs Public exports
|
||||
├── error.rs Error codes (no English)
|
||||
├── event.rs Event types for callbacks
|
||||
├── halt.rs Halt cancellation token (Arc<AtomicBool> wrapper)
|
||||
├── io/ Pipeline + WritebackFile primitives
|
||||
│ ├── mod.rs Re-exports WritebackFile, Pipeline, Sink, Flow
|
||||
│ ├── pipeline.rs Generic Pipeline<I, R> + Sink trait
|
||||
│ ├── writeback_file.rs WritebackFile (was crate::io::Writer)
|
||||
│ └── writeback.rs sync_file_range pipeline
|
||||
├── drive/ Drive (open, init, single-shot read)
|
||||
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
|
||||
│ ├── capture.rs Drive profile capture for contribution
|
||||
│ ├── linux.rs Linux drive discovery
|
||||
│ ├── macos.rs macOS drive discovery
|
||||
│ └── windows.rs Windows drive discovery
|
||||
├── disc/ Disc (scan, titles, AACS setup)
|
||||
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
|
||||
│ ├── mod.rs Disc struct, scan, titles, formats
|
||||
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
|
||||
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
|
||||
│ ├── mapfile.rs ddrescue-format mapfile
|
||||
│ └── read_error.rs ReadCtx / ReadAction state machine
|
||||
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
|
||||
├── platform/ Drive unlock (MT1959 A/B)
|
||||
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
|
||||
├── css/ DVD CSS cipher
|
||||
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
|
||||
├── pes.rs PES frame types, Stream trait
|
||||
├── sector.rs SectorReader trait
|
||||
├── pes.rs PES frame types, FrameSource / FrameSink traits
|
||||
├── sector/ Sector I/O (was sector.rs in 0.17)
|
||||
│ ├── mod.rs SectorSource, SectorSink traits
|
||||
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
|
||||
│ └── decrypting.rs DecryptingSectorSource decorator
|
||||
├── udf.rs UDF 2.50 filesystem parser
|
||||
├── mpls.rs MPLS playlist parser
|
||||
├── clpi.rs CLPI clip info parser
|
||||
|
|
@ -199,15 +201,15 @@ libfreemkv/src/
|
|||
├── profile.rs Bundled drive profiles
|
||||
├── speed.rs DriveSpeed enum
|
||||
├── mux/
|
||||
│ ├── mod.rs IOStream trait, public exports
|
||||
│ ├── resolve.rs URL parser + open_input/open_output + input/output
|
||||
│ ├── mod.rs Public mux exports
|
||||
│ ├── resolve.rs URL parser + input/output (Box<dyn FrameSource/Sink>)
|
||||
│ ├── meta.rs FMKV header format
|
||||
│ ├── disc.rs DiscStream (optical drive → PES)
|
||||
│ ├── iso.rs IsoStream (ISO image read/write)
|
||||
│ ├── iso.rs IsoStream (ISO image read)
|
||||
│ ├── isowriter.rs ISO image writer (UDF, AVDP, multi-extent)
|
||||
│ ├── mkvstream.rs MkvStream (bidirectional Matroska, IOStream)
|
||||
│ ├── mkvstream.rs MkvStream (bidirectional Matroska)
|
||||
│ ├── mkvout.rs MkvOutputStream (PES → MKV)
|
||||
│ ├── m2ts.rs M2tsStream (BD-TS, IOStream)
|
||||
│ ├── m2ts.rs M2tsStream (BD-TS)
|
||||
│ ├── pesout.rs PES output streams (M2ts, Network, Stdio, Null)
|
||||
│ ├── network.rs NetworkStream (TCP + FMKV header)
|
||||
│ ├── stdio.rs StdioStream (stdin/stdout pipe)
|
||||
|
|
|
|||
|
|
@ -65,8 +65,12 @@ libfreemkv (lib.rs)
|
|||
│
|
||||
├── Streaming
|
||||
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
|
||||
│ ├── pes PES frame types, Stream trait (read/write frames)
|
||||
│ └── sector SectorReader trait — abstracts disc vs ISO vs file
|
||||
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
|
||||
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
|
||||
│
|
||||
├── I/O Primitives
|
||||
│ ├── halt Halt cancellation token (one Arc<AtomicBool>, cloneable)
|
||||
│ └── io/ Pipeline<I, R> + Sink trait + WritebackFile (bounded-cache writer)
|
||||
│
|
||||
├── Support
|
||||
│ ├── keydb KEYDB.cfg download, parse, verify, save
|
||||
|
|
|
|||
|
|
@ -97,7 +97,9 @@ drive.probe_disc()?;
|
|||
// Scan disc (UDF + playlists + AACS — all automatic)
|
||||
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
|
||||
|
||||
// Stream pipeline — PES frames from any source to any output
|
||||
// Stream pipeline — PES frames from any source to any output.
|
||||
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
|
||||
// direction is type-checked, so calling .write() on an input is a compile error.
|
||||
let opts = InputOptions::default();
|
||||
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
|
||||
let title = input.info().clone();
|
||||
|
|
@ -121,11 +123,13 @@ output.finish()?;
|
|||
| aacs/ | [aacs.md](aacs.md) | Key resolution + content decrypt + bus handshake |
|
||||
| css/ | -- | DVD CSS cipher |
|
||||
| decrypt.rs | -- | Unified decrypt dispatcher (AACS/CSS/None) |
|
||||
| disc/ | -- | High-level scan + read API |
|
||||
| disc/ | [rip-recovery.md](rip-recovery.md) | Disc::scan + Disc::sweep + Disc::patch + mapfile |
|
||||
| labels/ | -- | BD-J stream labels (5 format parsers) |
|
||||
| mux/ | -- | Stream implementations (7 stream types) |
|
||||
| pes.rs | -- | PES frame types + Stream trait |
|
||||
| sector.rs | -- | SectorReader trait |
|
||||
| pes.rs | -- | PES frame types + FrameSource / FrameSink traits |
|
||||
| sector/ | -- | SectorSource / SectorSink + DecryptingSectorSource decorator |
|
||||
| io/ | -- | Pipeline<I, R> + Sink trait + WritebackFile |
|
||||
| halt.rs | -- | Halt cancellation token |
|
||||
| keydb.rs | -- | KEYDB download, parse, save |
|
||||
| error.rs | -- | Error codes (E1xxx-E8xxx) |
|
||||
| event.rs | -- | Drive event system |
|
||||
|
|
|
|||
|
|
@ -52,8 +52,8 @@ selects the per-CDB timeout:
|
|||
|
||||
| `recovery` | Timeout | Used by |
|
||||
|------------|----------|------------------------------------------|
|
||||
| `false` | 1.5 s | `Disc::copy` fast skip-forward sweep, `DiscStream::fill_extents` |
|
||||
| `true` | 30 s | `Disc::patch` multi-pass over the mapfile |
|
||||
| `false` | 1.5 s | `Disc::sweep` fast skip-forward pass, `DiscStream::fill_extents` |
|
||||
| `true` | 30 s | `Disc::patch` retry pass over the mapfile |
|
||||
|
||||
On any SCSI failure or timeout, `read` returns `Err(DiscRead)` immediately.
|
||||
There are no inline retries, no SCSI reset, no Phase 1/2/3 escalation.
|
||||
|
|
|
|||
|
|
@ -12,21 +12,24 @@ reach into the others.
|
|||
|
||||
| Layer | Where it lives | What it does |
|
||||
|-------|---------------|--------------|
|
||||
| 1 — Bad-range retry | `Disc::patch` (multi-pass over the mapfile) | Re-reads non-`+` ranges with the long timeout. Idempotent; call N times. |
|
||||
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
|
||||
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
|
||||
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
|
||||
|
||||
The caller orchestrates layer 1. Autorip's `rip_disc` loops `copy` then
|
||||
N × `patch` per the `MAX_RETRIES` config, then hands the ISO off to the
|
||||
existing mux pipeline. Layer 3 runs inside any consumer of `DiscStream`
|
||||
(direct PES pipeline, ISO playback, etc.) without caller involvement.
|
||||
The library exposes flat verbs; the caller drives the multipass loop. Autorip
|
||||
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
|
||||
clean or the configured retry budget is exhausted, then hands the ISO off to
|
||||
the mux pipeline. The `freemkv` CLI does the same shape with a
|
||||
terminal-output progress sink. Layer 3 runs inside any consumer of
|
||||
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
|
||||
involvement.
|
||||
|
||||
Three primitives compose the disc-side flow:
|
||||
|
||||
| Primitive | What it does |
|
||||
|---------------------------|-----------------------------------------------------------------------|
|
||||
| `Disc::copy` | disc → ISO. Writes a sidecar `.mapfile`. Opt-in skip-forward on failure. |
|
||||
| `Disc::patch` | Re-reads bad ranges from the drive. Idempotent; call N times. |
|
||||
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
|
||||
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
|
||||
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
|
||||
|
||||
## Data model
|
||||
|
|
@ -60,28 +63,46 @@ Status characters match ddrescue:
|
|||
|
||||
Position and size are hex byte offsets into the ISO.
|
||||
|
||||
### `CopyOptions` and `PatchOptions`
|
||||
### `SweepOptions` and `PatchOptions`
|
||||
|
||||
`Disc::copy()` auto-detects the pass from mapfile state:
|
||||
The library no longer dispatches between sweep and patch internally — the
|
||||
caller picks the verb explicitly per pass. The two option structs are flat
|
||||
and have no overlap:
|
||||
|
||||
```rust
|
||||
CopyOptions {
|
||||
decrypt: true, // decrypt AACS/CSS sectors
|
||||
multipass: true, // enable mapfile + skip-on-error + damage-jump
|
||||
progress: Some(&reporter), // progress callback
|
||||
halt: Some(flag), // halt flag for graceful stop
|
||||
SweepOptions {
|
||||
decrypt: true,
|
||||
resume: false,
|
||||
batch_sectors: None,
|
||||
skip_on_error: true, // damage-jump + zero-fill on read failure
|
||||
progress: Some(&reporter),
|
||||
halt: Some(flag),
|
||||
}
|
||||
|
||||
PatchOptions {
|
||||
decrypt: true,
|
||||
block_sectors: None,
|
||||
full_recovery: true,
|
||||
reverse: true, // walk bad ranges high → low LBA
|
||||
wedged_threshold: 50,
|
||||
progress: Some(&reporter),
|
||||
halt: Some(flag),
|
||||
}
|
||||
```
|
||||
|
||||
Dispatch logic:
|
||||
- No mapfile → sweep (fresh Pass 1)
|
||||
- Mapfile with NonTried (`?`) → sweep with resume
|
||||
- Mapfile covering full disc, only NonTrimmed/NonScraped/Unreadable → patch
|
||||
- Mapfile clean → no-op
|
||||
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
|
||||
|
||||
- No mapfile → `sweep` (fresh Pass 1).
|
||||
- Mapfile with `?` ranges → `sweep` with `resume: true`.
|
||||
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
|
||||
- Mapfile clean → done; no further pass needed.
|
||||
|
||||
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
|
||||
lines of `Mapfile::stats()` checks.
|
||||
|
||||
## Algorithm
|
||||
|
||||
### Pass 1 — fast sweep (`Disc::copy` → `sweep_internal`)
|
||||
### Pass 1 — fast sweep (`Disc::sweep`)
|
||||
|
||||
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
|
||||
2. On success: write data to ISO, mark `+`, advance.
|
||||
|
|
@ -95,7 +116,7 @@ Dispatch logic:
|
|||
|
||||
Pass 1 completes when every byte has been visited (either `+` or `*`).
|
||||
|
||||
### Pass 2+ — patch (`Disc::copy` → `patch_internal`)
|
||||
### Pass 2+ — patch (`Disc::patch`)
|
||||
|
||||
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
|
||||
(walks ranges from highest LBA to lowest, within each range from end to start).
|
||||
|
|
@ -177,4 +198,4 @@ scrape vs. retry with direction reversal) if there's measured benefit.
|
|||
|
||||
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
|
||||
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
|
||||
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::copy`, `Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
|
||||
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
|
||||
|
|
|
|||
|
|
@ -126,7 +126,7 @@ USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_
|
|||
|
||||
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
|
||||
|
||||
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorReader` implementation.
|
||||
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
|
||||
|
||||
### UDF Filename Encoding
|
||||
|
||||
|
|
|
|||
|
|
@ -62,7 +62,23 @@ pub fn decrypt_sectors(
|
|||
|
||||
for chunk in buf.chunks_mut(unit_len) {
|
||||
if chunk.len() == unit_len && aacs::is_unit_encrypted(chunk) {
|
||||
aacs::decrypt_unit_full(chunk, &uk, rdk);
|
||||
// `is_unit_encrypted` is a byte-0 heuristic: it fires on any
|
||||
// unit whose first byte has the top 2 bits set, which is
|
||||
// correct for m2ts source packets (where those bits are the
|
||||
// copy-control marker) but false-positives on any other binary
|
||||
// data with similarly-shaped first bytes — notably MPLS/CLPI
|
||||
// navigation files that begin with ASCII magic ('M', 'H'…)
|
||||
// and survive sweep mixed in with encrypted m2ts payloads.
|
||||
// `decrypt_unit_full` self-checks via TS-sync verification and
|
||||
// returns false on a misfire, but it has already mutated the
|
||||
// chunk by then. Snapshot and restore on verification failure
|
||||
// — same pattern `decrypt_unit_try_keys` uses for multi-key
|
||||
// discs. Real m2ts units verify and stay decrypted; nav-file
|
||||
// sectors get scrambled briefly and then put back as-was.
|
||||
let original: Vec<u8> = chunk.to_vec();
|
||||
if !aacs::decrypt_unit_full(chunk, &uk, rdk) {
|
||||
chunk.copy_from_slice(&original);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -74,3 +90,36 @@ pub fn decrypt_sectors(
|
|||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Regression for the 0.18.1 nav-file scramble bug. A non-m2ts unit whose
|
||||
/// first byte has the top 2 bits set (here: the ASCII letter 'M' that
|
||||
/// MPLS files start with, 0x4D = 0b01001101) trips `is_unit_encrypted`,
|
||||
/// gets AES-decrypted with the unit key, fails the TS-sync verification,
|
||||
/// and must be restored to its original bytes — not left scrambled.
|
||||
#[test]
|
||||
fn nav_file_unit_survives_decrypt_attempt() {
|
||||
let mut unit = vec![0u8; aacs::ALIGNED_UNIT_LEN];
|
||||
unit[0] = b'M';
|
||||
unit[1] = b'P';
|
||||
unit[2] = b'L';
|
||||
unit[3] = b'S';
|
||||
for (i, b) in unit.iter_mut().enumerate().skip(4) {
|
||||
*b = (i as u8).wrapping_mul(31);
|
||||
}
|
||||
let snapshot = unit.clone();
|
||||
|
||||
let keys = DecryptKeys::Aacs {
|
||||
unit_keys: vec![(0, [0xAB; 16])],
|
||||
read_data_key: None,
|
||||
};
|
||||
decrypt_sectors(&mut unit, &keys, 0).unwrap();
|
||||
assert_eq!(
|
||||
unit, snapshot,
|
||||
"non-m2ts unit must be restored after failed decrypt"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,11 +17,22 @@
|
|||
//!
|
||||
//! Status chars: `?` non-tried · `*` non-trimmed · `/` non-scraped · `-` unreadable · `+` finished.
|
||||
//!
|
||||
//! The mapfile is flushed to disk on every `record()` call so a crashed
|
||||
//! rip loses at most one block of recorded state.
|
||||
//! The mapfile is flushed to disk at most once per `FLUSH_INTERVAL`
|
||||
//! during `record()` calls, plus on explicit `flush()` and on `Drop`.
|
||||
//! This bounds atomic-rename RPC rate on networked staging (e.g. NFS)
|
||||
//! where per-record persists otherwise serialize the rip pipeline.
|
||||
|
||||
use std::io::{self, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
/// Minimum interval between mapfile persists. `record()` updates in-memory
|
||||
/// state every call but only writes to disk when this interval has elapsed
|
||||
/// since the last persist (or when `flush()` is called explicitly, or on
|
||||
/// `Drop`). Bounds RPC rate on NFS staging where atomic-rename per record
|
||||
/// otherwise dominates throughput. On crash the worst-case progress loss
|
||||
/// is one interval's worth of records.
|
||||
const FLUSH_INTERVAL: Duration = Duration::from_millis(1000);
|
||||
|
||||
/// Status of a byte range in the mapfile. ddrescue-compatible.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
|
|
@ -100,8 +111,11 @@ pub struct MapStats {
|
|||
pub main_lost_ms: f64,
|
||||
}
|
||||
|
||||
/// Write-through mapfile. Every `record()` persists to disk immediately
|
||||
/// so a crash during rip loses at most one block.
|
||||
/// Time-batched mapfile. `record()` keeps in-memory state up-to-date on
|
||||
/// every call; persists to disk at most once per `FLUSH_INTERVAL`.
|
||||
/// Explicit `flush()` and `Drop` guarantee state is on disk after a sweep
|
||||
/// or patch finishes. On hard crash the worst-case loss is one flush
|
||||
/// interval of records — the file's payload bytes are unaffected.
|
||||
pub struct Mapfile {
|
||||
path: PathBuf,
|
||||
entries: Vec<MapEntry>,
|
||||
|
|
@ -110,6 +124,12 @@ pub struct Mapfile {
|
|||
/// Incrementally maintained stats — updated on every `record()` call
|
||||
/// so `stats()` is O(1) instead of O(n).
|
||||
stats: MapStats,
|
||||
/// True when in-memory state has changed but `write_to_disk` has not
|
||||
/// yet captured it.
|
||||
dirty: bool,
|
||||
/// Wall-clock timestamp of the last successful `write_to_disk` (or
|
||||
/// the moment the mapfile was constructed, whichever is later).
|
||||
last_flushed: Instant,
|
||||
}
|
||||
|
||||
impl Mapfile {
|
||||
|
|
@ -117,7 +137,7 @@ impl Mapfile {
|
|||
/// Writes to disk immediately so a resume can pick up even if the caller
|
||||
/// never records anything.
|
||||
pub fn create(path: &Path, total_size: u64, version: &str) -> io::Result<Self> {
|
||||
let mf = Self {
|
||||
let mut mf = Self {
|
||||
path: path.to_path_buf(),
|
||||
entries: vec![MapEntry {
|
||||
pos: 0,
|
||||
|
|
@ -132,8 +152,13 @@ impl Mapfile {
|
|||
bytes_nontried: total_size,
|
||||
..Default::default()
|
||||
},
|
||||
dirty: false,
|
||||
last_flushed: Instant::now(),
|
||||
};
|
||||
// Eager initial persist so a resume can pick this up even if
|
||||
// `record()` is never called.
|
||||
mf.write_to_disk()?;
|
||||
mf.last_flushed = Instant::now();
|
||||
Ok(mf)
|
||||
}
|
||||
|
||||
|
|
@ -200,6 +225,8 @@ impl Mapfile {
|
|||
total_size,
|
||||
version,
|
||||
stats,
|
||||
dirty: false,
|
||||
last_flushed: Instant::now(),
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -268,7 +295,24 @@ impl Mapfile {
|
|||
// path during sweep/patch, it just returns the cached value.
|
||||
self.stats = Self::compute_stats(&merged, self.total_size);
|
||||
self.entries = merged;
|
||||
self.write_to_disk()?;
|
||||
self.dirty = true;
|
||||
if self.last_flushed.elapsed() >= FLUSH_INTERVAL {
|
||||
self.write_to_disk()?;
|
||||
self.dirty = false;
|
||||
self.last_flushed = Instant::now();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Persist any pending in-memory changes to disk. No-op if clean.
|
||||
/// Callers (sweep/patch finalisation) invoke this after their last
|
||||
/// `record()` to guarantee state is durable before returning.
|
||||
pub fn flush(&mut self) -> io::Result<()> {
|
||||
if self.dirty {
|
||||
self.write_to_disk()?;
|
||||
self.dirty = false;
|
||||
self.last_flushed = Instant::now();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
@ -363,6 +407,16 @@ impl Mapfile {
|
|||
}
|
||||
}
|
||||
|
||||
impl Drop for Mapfile {
|
||||
/// Best-effort flush on drop so a sweep / patch that returns early
|
||||
/// (or unwinds) doesn't lose its in-memory state. Errors here are
|
||||
/// swallowed because Drop has no way to surface them; explicit
|
||||
/// `flush()` on the success path gives callers proper error handling.
|
||||
fn drop(&mut self) {
|
||||
let _ = self.flush();
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_hex(s: &str) -> io::Result<u64> {
|
||||
let s = s.strip_prefix("0x").unwrap_or(s);
|
||||
u64::from_str_radix(s, 16).map_err(|_| {
|
||||
|
|
@ -463,6 +517,8 @@ mod tests {
|
|||
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
|
||||
mf.record(100, 200, SectorStatus::Finished).unwrap();
|
||||
mf.record(500, 100, SectorStatus::Unreadable).unwrap();
|
||||
// record() batches; explicit flush before reading back from disk.
|
||||
mf.flush().unwrap();
|
||||
let loaded = Mapfile::load(&p).unwrap();
|
||||
assert_eq!(loaded.entries(), mf.entries());
|
||||
let _ = std::fs::remove_file(&p);
|
||||
|
|
|
|||
1070
src/disc/mod.rs
1070
src/disc/mod.rs
File diff suppressed because it is too large
Load diff
243
src/disc/patch.rs
Normal file
243
src/disc/patch.rs
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
//! Producer / consumer split for `Disc::patch`.
|
||||
//!
|
||||
//! Background: pre-0.18 patch ran strictly serial — single-sector
|
||||
//! recovery read → seek + write recovered bytes → mapfile.record →
|
||||
//! next iteration. The drive sat idle while the previous block's
|
||||
//! recovered bytes were committed. On a damaged disc with many bad
|
||||
//! sectors that adds up: per-sector write + mapfile.record costs a
|
||||
//! handful of milliseconds each, which the drive could be using to
|
||||
//! issue the next per-sector retry.
|
||||
//!
|
||||
//! This module decouples them. A consumer thread owns the
|
||||
//! [`crate::io::WritebackFile`] (the ISO file) and the
|
||||
//! [`super::mapfile::Mapfile`]. The producer thread (`Disc::patch`)
|
||||
//! keeps the [`crate::sector::SectorReader`], the wedge / damage-window
|
||||
//! state, the per-range watchdog, decrypt — so what enters the channel
|
||||
//! is already-clean cleartext bytes (or an "Unreadable" terminal mark).
|
||||
//!
|
||||
//! Producer and consumer run concurrently; the channel uses
|
||||
//! [`crate::io::pipeline::WRITE_THROUGH_DEPTH`] (=1) so back-pressure
|
||||
//! kicks in immediately. We want the drive's per-sector retry budget
|
||||
//! to stay in lockstep with the writer — sweep's `DEFAULT_PIPELINE_DEPTH`
|
||||
//! (4) would let several sectors of recovered bytes queue up between
|
||||
//! the producer's retry decisions and the writer, and patch's recovery
|
||||
//! loop reads stats (`bytes_good`, range progress) inline to drive its
|
||||
//! skip / wedge decisions. WRITE_THROUGH_DEPTH gives "read N+1 while
|
||||
//! writing N", no further pipelining — exactly the model the producer
|
||||
//! logic was written against.
|
||||
//!
|
||||
//! Correctness invariants preserved:
|
||||
//! - Mapfile is single-writer (consumer-only). No locking on it.
|
||||
//! - All recovery state (damage window, consecutive_failures, skip
|
||||
//! escalation, range watchdog) stays on the producer thread.
|
||||
//! - `set_speed` calls happen on the producer thread (same thread that
|
||||
//! owns the `SectorReader`). No new SCSI concurrency.
|
||||
//! - Per-iteration ordering of file-write → mapfile-record is kept
|
||||
//! intact in the consumer (write before record), so the on-disk
|
||||
//! invariant "mapfile only marks Finished what the file has received"
|
||||
//! survives a crash mid-pass.
|
||||
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
|
||||
//! SCSI command in flight at a time, error-path timing identical,
|
||||
//! no new retry logic. The threading primitive only overlaps the
|
||||
//! *write* with the *next read*; the per-sector single-shot read
|
||||
//! budget that the bridge wedge concern was originally about is
|
||||
//! untouched.
|
||||
//!
|
||||
//! Per-range watchdog (`MAX_RANGE_SECS` / `RANGE_BUDGET_CAP_SECS`)
|
||||
//! checks `bytes_good` for forward progress. With work in flight on
|
||||
//! the consumer, the producer would otherwise see stale values; the
|
||||
//! sink publishes a [`SharedPatchState`] snapshot after every record
|
||||
//! so the producer's stall guards observe consumer side-effects with
|
||||
//! at most one item of lag (which is fine — the watchdog uses minute-
|
||||
//! scale budgets, not single-record latency).
|
||||
|
||||
use std::io::{Seek, SeekFrom, Write};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use crate::error::{Error, Result};
|
||||
use crate::io::pipeline::{Flow, Sink};
|
||||
|
||||
use super::mapfile::{MapStats, Mapfile, SectorStatus};
|
||||
|
||||
/// Item the producer hands to the patch consumer. One per per-sector
|
||||
/// recovery decision.
|
||||
pub(super) enum PatchItem {
|
||||
/// Sector / small batch successfully recovered (and decrypted on the
|
||||
/// producer side if `opts.decrypt` was set). Consumer seeks to
|
||||
/// `pos`, writes `buf`, records the range as `Finished`.
|
||||
Recovered { pos: u64, buf: Vec<u8> },
|
||||
|
||||
/// Producer exhausted retries on `[pos, pos+len)`. Consumer records
|
||||
/// the range as `Unreadable`. No file write — the existing zero-fill
|
||||
/// from sweep is preserved in place.
|
||||
Unreadable { pos: u64, len: u64 },
|
||||
|
||||
/// Producer hit the per-range skip limit and is leaving the
|
||||
/// remaining bytes as `NonTrimmed` for a future pass. CRITICAL:
|
||||
/// this is not the same as `Unreadable` — sectors we never tried
|
||||
/// stay hopeful. (See the comment at the skip-limit branch in
|
||||
/// `Disc::patch`: ~36% of patch-marked Unreadable sectors are
|
||||
/// actually readable on a later pass.) No file write.
|
||||
NonTrimmed { pos: u64, len: u64 },
|
||||
}
|
||||
|
||||
/// Mapfile snapshot the sink republishes after every record so the
|
||||
/// producer can drive its stall / progress logic without holding the
|
||||
/// mapfile lock for long. `bad_ranges` mirrors what
|
||||
/// `Mapfile::ranges_with(&[NonTrimmed, Unreadable, NonScraped, NonTried])`
|
||||
/// would return — same set the pre-split patch loop computed inline
|
||||
/// for the progress callback.
|
||||
pub(super) struct SharedPatchState {
|
||||
pub stats: MapStats,
|
||||
pub bad_ranges: Vec<(u64, u64)>,
|
||||
}
|
||||
|
||||
impl SharedPatchState {
|
||||
fn from_map(map: &Mapfile) -> Self {
|
||||
Self {
|
||||
stats: map.stats(),
|
||||
bad_ranges: map.ranges_with(&[
|
||||
SectorStatus::NonTrimmed,
|
||||
SectorStatus::Unreadable,
|
||||
SectorStatus::NonScraped,
|
||||
SectorStatus::NonTried,
|
||||
]),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Final summary returned by [`Sink::close`] when the consumer drains
|
||||
/// cleanly. Mirrors what the pre-split patch loop computed at the end
|
||||
/// of the function — final mapfile stats plus whether `sync_all`
|
||||
/// failed on a regular file (the only kind of fsync error patch ever
|
||||
/// surfaced; `/dev/null` and pipes always fail `sync_all`, that's not
|
||||
/// a real error).
|
||||
pub(super) struct PatchSummary {
|
||||
pub stats: MapStats,
|
||||
}
|
||||
|
||||
/// Consumer-side of the patch pipeline. Owns the ISO writeback file
|
||||
/// and the mapfile; publishes a shared snapshot after every record so
|
||||
/// the producer can read `bytes_good` for stall detection and
|
||||
/// progress reporting.
|
||||
pub(super) struct PatchSink {
|
||||
file: crate::io::WritebackFile,
|
||||
map: Mapfile,
|
||||
/// Whether the output is a regular file (so a `sync_all` failure
|
||||
/// is real). `/dev/null` etc. always fail `sync_all`; ignore those.
|
||||
is_regular: bool,
|
||||
/// Snapshot the producer reads. Updated after every successful
|
||||
/// `record()` call. `Mutex` rather than separate atomics because
|
||||
/// the producer wants stats + bad_ranges as a coherent pair.
|
||||
shared: Arc<Mutex<SharedPatchState>>,
|
||||
}
|
||||
|
||||
impl PatchSink {
|
||||
/// Open `path` as a [`crate::io::WritebackFile`] and pair it with
|
||||
/// `map` for the consumer. The producer holds onto the returned
|
||||
/// `Arc<Mutex<SharedPatchState>>` so it can poll mapfile state
|
||||
/// while the consumer is mutating it.
|
||||
pub(super) fn new(
|
||||
path: &std::path::Path,
|
||||
map: Mapfile,
|
||||
is_regular: bool,
|
||||
) -> Result<(Self, Arc<Mutex<SharedPatchState>>)> {
|
||||
let file =
|
||||
crate::io::WritebackFile::open(path).map_err(|e| Error::IoError { source: e })?;
|
||||
let shared = Arc::new(Mutex::new(SharedPatchState::from_map(&map)));
|
||||
let shared_clone = shared.clone();
|
||||
Ok((
|
||||
Self {
|
||||
file,
|
||||
map,
|
||||
is_regular,
|
||||
shared,
|
||||
},
|
||||
shared_clone,
|
||||
))
|
||||
}
|
||||
|
||||
fn republish(&self) {
|
||||
// Best-effort lock — only the producer reads, only the consumer
|
||||
// writes; contention is single-acquire so the lock is never
|
||||
// poisoned in practice. If it ever did get poisoned we'd want
|
||||
// the underlying error surfaced rather than silently swallowed,
|
||||
// so we propagate the poison panic. (Same posture as
|
||||
// `sweep_pipeline.rs` — it never recovers from a poisoned
|
||||
// mutex either.)
|
||||
let mut guard = self
|
||||
.shared
|
||||
.lock()
|
||||
.expect("PatchSink shared state mutex poisoned");
|
||||
*guard = SharedPatchState::from_map(&self.map);
|
||||
}
|
||||
}
|
||||
|
||||
impl Sink<PatchItem> for PatchSink {
|
||||
type Output = PatchSummary;
|
||||
|
||||
fn apply(&mut self, item: PatchItem) -> std::result::Result<Flow, Error> {
|
||||
match item {
|
||||
PatchItem::Recovered { pos, buf } => {
|
||||
let len = buf.len() as u64;
|
||||
self.file
|
||||
.seek(SeekFrom::Start(pos))
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.file
|
||||
.write_all(&buf)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.map
|
||||
.record(pos, len, SectorStatus::Finished)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
PatchItem::Unreadable { pos, len } => {
|
||||
self.map
|
||||
.record(pos, len, SectorStatus::Unreadable)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
PatchItem::NonTrimmed { pos, len } => {
|
||||
self.map
|
||||
.record(pos, len, SectorStatus::NonTrimmed)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
}
|
||||
self.republish();
|
||||
Ok(Flow::Continue)
|
||||
}
|
||||
|
||||
fn close(mut self) -> std::result::Result<Self::Output, Error> {
|
||||
// Drain in-flight writeback then issue a full fsync. A failure
|
||||
// here matters only on regular files — pipes / `/dev/null` etc.
|
||||
// always fail `sync_all`.
|
||||
if let Err(e) = self.file.sync_all() {
|
||||
if self.is_regular {
|
||||
tracing::warn!(
|
||||
target: "freemkv::disc",
|
||||
phase = "patch_sync_failed",
|
||||
error = %e,
|
||||
os_error = e.raw_os_error(),
|
||||
error_kind = ?e.kind(),
|
||||
"patch: sync_all failed"
|
||||
);
|
||||
return Err(Error::IoError { source: e });
|
||||
}
|
||||
tracing::debug!(
|
||||
target: "freemkv::disc",
|
||||
phase = "patch_sync_skipped",
|
||||
error = %e,
|
||||
"patch: sync_all failed for non-regular file; ignoring"
|
||||
);
|
||||
}
|
||||
self.map.flush().map_err(|e| Error::IoError { source: e })?;
|
||||
// Final republish so anyone reading the shared snapshot after
|
||||
// `Pipeline::finish` sees the post-flush state. (The producer
|
||||
// already has its own copy of the final `MapStats` in the
|
||||
// returned `PatchSummary`, but the snapshot is part of the
|
||||
// public-ish contract of the consumer: it stays current
|
||||
// through close.)
|
||||
self.republish();
|
||||
Ok(PatchSummary {
|
||||
stats: self.map.stats(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -25,11 +25,25 @@ pub struct ReadCtx {
|
|||
/// Failed reads since the last success. Resets to 0 on success.
|
||||
/// Drives long-pause escalation on persistent failure.
|
||||
pub consecutive_failures: u64,
|
||||
/// Failed OUTER batch reads since the last outer success — bisect
|
||||
/// inner-sector failures are NOT counted here. Drives the
|
||||
/// fast-entry damage-jump on Pass 1 (skip the disc-level grind
|
||||
/// once we're clearly in a damaged region; Pass N will recover
|
||||
/// the actual sectors). Reset on outer success.
|
||||
pub consecutive_outer_failures: u64,
|
||||
/// Sliding window of recent read outcomes (true=ok, false=fail).
|
||||
/// Capped at `damage_window_max`. Drives damage-jump decisions.
|
||||
pub damage_window: Vec<bool>,
|
||||
pub damage_window_max: usize,
|
||||
pub damage_threshold_pct: usize,
|
||||
/// Trigger a damage-jump after this many consecutive outer-batch
|
||||
/// failures, even when the damage_window isn't full yet. Pass 1
|
||||
/// uses a small value (4) so we don't spend ~40 minutes grinding
|
||||
/// to fill a 16-block window before the first jump on a damage
|
||||
/// zone we entered cleanly. Pass N uses a larger value (or
|
||||
/// disables this — see `bisect_on_marginal`) because Pass N's
|
||||
/// whole job IS to grind on the bad ranges.
|
||||
pub fast_jump_threshold: u64,
|
||||
/// Multiplier applied to damage-jump distance. Doubles each jump,
|
||||
/// resets to 1 after `damage_window_max` consecutive good reads.
|
||||
pub jump_multiplier: u64,
|
||||
|
|
@ -43,38 +57,73 @@ pub struct ReadCtx {
|
|||
/// failed batch, so the handler doesn't recursively request another
|
||||
/// bisect on the inner-sector failures.
|
||||
pub bisecting: bool,
|
||||
/// Whether to return `Bisect` on a marginal-media batch failure.
|
||||
/// Pass 1 sweep sets this false: a failed batch becomes
|
||||
/// SkipBlock (mark the whole 32-sector ECC block NonTrimmed,
|
||||
/// advance, let Pass N recover the salvageable sectors with
|
||||
/// proper recovery semantics). Pass N sets this true: bisection
|
||||
/// is its core job, and it has the right tools (single-sector
|
||||
/// reads, 60s recovery timeout, retry budget, escalating skip).
|
||||
pub bisect_on_marginal: bool,
|
||||
/// Count of consecutive firmware-wedge responses (HARDWARE_ERROR
|
||||
/// or ILLEGAL_REQUEST sense keys) since the last successful read.
|
||||
/// Pass 1 uses this to drive the wedge-skip path: each wedge
|
||||
/// triggers a 1 GB jump + cooldown pause. Reaching
|
||||
/// `WEDGE_ABORT_THRESHOLD` consecutive wedges with no good read
|
||||
/// in between → real AbortPass.
|
||||
pub wedge_count: u64,
|
||||
}
|
||||
|
||||
impl ReadCtx {
|
||||
/// Initial context for a Pass 1 sweep with the documented constants.
|
||||
/// Initial context for a Pass 1 sweep. The job is "fast and
|
||||
/// accurate, get the most data in the shortest time" — Pass N
|
||||
/// is the one that grinds on the bad ranges. So bisect-on-
|
||||
/// marginal is OFF (failed batches become SkipBlock; whole 32-
|
||||
/// sector blocks marked NonTrimmed for Pass N to revisit), and
|
||||
/// the damage-jump fast-path triggers after just 4 consecutive
|
||||
/// outer-batch failures.
|
||||
pub fn for_sweep(batch: u16) -> Self {
|
||||
Self {
|
||||
batch,
|
||||
consecutive_good: 0,
|
||||
consecutive_failures: 0,
|
||||
consecutive_outer_failures: 0,
|
||||
damage_window: Vec::with_capacity(16),
|
||||
damage_window_max: 16,
|
||||
damage_threshold_pct: 12,
|
||||
fast_jump_threshold: 4,
|
||||
jump_multiplier: 1,
|
||||
not_ready_retries: 0,
|
||||
bridge_degradation_count: 0,
|
||||
bisecting: false,
|
||||
bisect_on_marginal: false,
|
||||
wedge_count: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Initial context for a Pass 2-N patch.
|
||||
/// Initial context for a Pass 2-N patch. Pass N's whole reason to
|
||||
/// exist is to recover sectors Pass 1 skipped — bisection on
|
||||
/// marginal media is part of the job, and the fast-jump
|
||||
/// threshold is loose so we don't bail too early on a range that
|
||||
/// has scattered good sectors mixed in.
|
||||
pub fn for_patch(batch: u16) -> Self {
|
||||
Self {
|
||||
batch,
|
||||
consecutive_good: 0,
|
||||
consecutive_failures: 0,
|
||||
consecutive_outer_failures: 0,
|
||||
damage_window: Vec::with_capacity(16),
|
||||
damage_window_max: 16,
|
||||
damage_threshold_pct: 12,
|
||||
// Pass N is allowed to grind: window-based jump only,
|
||||
// matching the historical behaviour for patch passes.
|
||||
fast_jump_threshold: u64::MAX,
|
||||
jump_multiplier: 1,
|
||||
not_ready_retries: 0,
|
||||
bridge_degradation_count: 0,
|
||||
bisecting: false,
|
||||
bisect_on_marginal: true,
|
||||
wedge_count: 0,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -83,6 +132,16 @@ impl ReadCtx {
|
|||
self.consecutive_good += 1;
|
||||
self.consecutive_failures = 0;
|
||||
self.not_ready_retries = 0;
|
||||
// Any successful read clears the wedge-skip counter — the
|
||||
// drive recovered, so further wedges should reset the skip
|
||||
// budget instead of accumulating toward a real abort.
|
||||
self.wedge_count = 0;
|
||||
// Outer-success only: a good single-sector read inside a
|
||||
// bisect doesn't mean we've left the damaged batch. Only an
|
||||
// outer-batch success resets the outer-failure counter.
|
||||
if !self.bisecting {
|
||||
self.consecutive_outer_failures = 0;
|
||||
}
|
||||
self.damage_window.push(true);
|
||||
if self.damage_window.len() > self.damage_window_max {
|
||||
self.damage_window.remove(0);
|
||||
|
|
@ -128,6 +187,63 @@ const NOT_READY_MAX_RETRIES: u32 = 3;
|
|||
const BRIDGE_DEGRADATION_PAUSE_SECS: u64 = 15;
|
||||
const BRIDGE_DEGRADATION_MAX_RETRIES: u32 = 5;
|
||||
|
||||
/// Pass 1 inter-error pause. ZERO on the clean path (sweep zooms
|
||||
/// past damage windows) but applied to every failed read so the
|
||||
/// drive's firmware gets cool-down time between damage-cluster
|
||||
/// exposures. Wedge-avoidance change 2026-05-10: pre-fix `pause_secs
|
||||
/// = 0` on Pass 1 errors caused back-to-back exposures that
|
||||
/// accumulated firmware wedge state.
|
||||
const PASS_1_FAIL_PAUSE_SECS: u64 = 5;
|
||||
|
||||
/// Base of the damage-jump distance formula: `jump_sectors =
|
||||
/// JUMP_BASE_SECTORS × batch × jump_multiplier`. Bumped 2026-05-10
|
||||
/// from 256 → 1024 (4×) so the first damage-jump at batch=32 covers
|
||||
/// 64 MB instead of 16 MB. Empirically the BU40N's damage clusters
|
||||
/// are 100+ MB wide; 16 MB jumps landed inside the cluster and the
|
||||
/// re-read added to the firmware wedge counter. 64 MB → 128 MB
|
||||
/// (after one doubling) clears almost any single-cluster damage in
|
||||
/// 2 jumps.
|
||||
const JUMP_BASE_SECTORS: u64 = 1024;
|
||||
|
||||
// Firmware-wedge skip policy for Pass 1 sweep
|
||||
// ===========================================
|
||||
//
|
||||
// When the BU40N (or similar drives) hits a physical-damage cluster,
|
||||
// its firmware can transition into a "wedge" state where it returns
|
||||
// HARDWARE_ERROR or ILLEGAL_REQUEST for every subsequent read —
|
||||
// often for many LBAs after the actual bad sector. Per project docs
|
||||
// "Bad-sector handling" rule #2: "Recovery requires eject+reload OR
|
||||
// significant cool-down."
|
||||
//
|
||||
// Pass 1's pre-fix behavior was to immediately AbortPass on the
|
||||
// first HARDWARE_ERROR / ILLEGAL_REQUEST, killing the rip at
|
||||
// whatever percentage it had reached. That's the wrong call when:
|
||||
// - the damage zone may be small (jumping past it could resume
|
||||
// normal reads), AND
|
||||
// - even if the drive stays wedged, finishing the sweep gives us
|
||||
// an honest mapfile for Pass N to attack later.
|
||||
//
|
||||
// New policy: treat wedge sense codes the same way the damage-window
|
||||
// treats persistent failure — JumpAhead by a large distance with a
|
||||
// cooldown pause. Allow up to WEDGE_ABORT_THRESHOLD consecutive
|
||||
// wedges (no successful read in between) before declaring the drive
|
||||
// truly stuck and surfacing AbortPass to autorip.
|
||||
|
||||
/// One-gigabyte jump (1024 MiB) on each wedge. Big enough to clear
|
||||
/// almost any single-cluster damage zone we've seen.
|
||||
const WEDGE_JUMP_SECTORS: u64 = 524_288;
|
||||
/// Cooldown pause after each wedge. Per project docs the drive needs
|
||||
/// "significant cool-down"; 30 s strikes a balance between giving
|
||||
/// the drive a chance to recover and not stalling the rip if the
|
||||
/// drive is permanently stuck.
|
||||
const WEDGE_PAUSE_SECS: u64 = 30;
|
||||
/// Bail after this many consecutive wedges with no good read in
|
||||
/// between. At 1 GB jumps this lets us scan ~16 GB worth of fully
|
||||
/// wedged area before giving up — generous enough to clear most
|
||||
/// physical-damage clusters, bounded enough to not loop forever on
|
||||
/// a permanently bricked drive.
|
||||
const WEDGE_ABORT_THRESHOLD: u64 = 16;
|
||||
|
||||
/// THE single error-handling entry point. Updates `ctx`, returns the
|
||||
/// action the caller must apply.
|
||||
///
|
||||
|
|
@ -137,6 +253,13 @@ const BRIDGE_DEGRADATION_MAX_RETRIES: u32 = 5;
|
|||
pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
||||
ctx.consecutive_failures += 1;
|
||||
ctx.consecutive_good = 0;
|
||||
// Outer-failure counter — only OUTER batch failures count toward
|
||||
// the fast-jump trigger. Bisect inner failures are part of
|
||||
// recovering an already-failed batch and don't represent
|
||||
// independent damage signal.
|
||||
if !ctx.bisecting {
|
||||
ctx.consecutive_outer_failures += 1;
|
||||
}
|
||||
|
||||
tracing::warn!(
|
||||
target: "freemkv::disc",
|
||||
|
|
@ -186,11 +309,54 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
|||
ctx.not_ready_retries = 0;
|
||||
}
|
||||
|
||||
// 4. Hardware / illegal request: the drive said "no, won't do it".
|
||||
// Retrying won't change the answer. Surface to the outer layer
|
||||
// so it can eject + prompt user.
|
||||
// 4. Hardware error / illegal request — the firmware-wedge family.
|
||||
// The drive transitioned into a fast-fail state where it
|
||||
// rejects reads near the LBA. Two policies:
|
||||
//
|
||||
// Pass 1 sweep (bisect_on_marginal=false): the wedge is
|
||||
// *recoverable* by skipping. Jump a large distance ahead
|
||||
// (1 GB), pause for cooldown, mark the skipped region
|
||||
// NonTrimmed so Pass N revisits. Allow up to
|
||||
// WEDGE_ABORT_THRESHOLD consecutive wedges before truly
|
||||
// giving up. This replaces the pre-fix "abort the entire
|
||||
// rip on first wedge" behavior that caused 48%-and-die
|
||||
// failures on discs with one bad cluster.
|
||||
//
|
||||
// Pass N patch (bisect_on_marginal=true): Pass N's job IS
|
||||
// single-sector recovery; a wedge means the drive won't
|
||||
// give us the specific sectors we asked for. Skipping
|
||||
// doesn't help here. Abort and let autorip decide whether
|
||||
// to retry, eject, or surface the failure to the user.
|
||||
if sense_key == scsi::SENSE_KEY_HARDWARE_ERROR || sense_key == scsi::SENSE_KEY_ILLEGAL_REQUEST {
|
||||
return ReadAction::AbortPass;
|
||||
if ctx.bisect_on_marginal {
|
||||
return ReadAction::AbortPass;
|
||||
}
|
||||
// Pass 1 wedge-skip path.
|
||||
if !ctx.bisecting {
|
||||
ctx.wedge_count += 1;
|
||||
}
|
||||
if ctx.wedge_count >= WEDGE_ABORT_THRESHOLD {
|
||||
tracing::warn!(
|
||||
target: "freemkv::disc",
|
||||
phase = "wedge_abort",
|
||||
wedge_count = ctx.wedge_count,
|
||||
threshold = WEDGE_ABORT_THRESHOLD,
|
||||
"Pass 1 wedge-skip exhausted — drive appears permanently stuck"
|
||||
);
|
||||
return ReadAction::AbortPass;
|
||||
}
|
||||
tracing::warn!(
|
||||
target: "freemkv::disc",
|
||||
phase = "wedge_skip",
|
||||
wedge_count = ctx.wedge_count,
|
||||
jump_sectors = WEDGE_JUMP_SECTORS,
|
||||
pause_secs = WEDGE_PAUSE_SECS,
|
||||
"Pass 1 wedge detected — jumping ahead and pausing for drive cooldown"
|
||||
);
|
||||
return ReadAction::JumpAhead {
|
||||
sectors: WEDGE_JUMP_SECTORS,
|
||||
pause_secs: WEDGE_PAUSE_SECS,
|
||||
};
|
||||
}
|
||||
|
||||
// 5. Marginal media (MEDIUM_ERROR / ABORTED_COMMAND) on a multi-
|
||||
|
|
@ -198,11 +364,20 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
|||
// individually. Bisect into single-sector reads (gentler on the
|
||||
// bridge too — shorter SCSI transactions). Avoid recursive
|
||||
// bisect.
|
||||
//
|
||||
// Pass 1 sweep sets `bisect_on_marginal=false` to skip this:
|
||||
// its job is "fast and accurate, get the most data in the
|
||||
// shortest time." Pass N is purpose-built to recover
|
||||
// individual sectors with proper recovery semantics, and Pass
|
||||
// 1 grinding through 32-sector bisects costs ~2.5 min per bad
|
||||
// block AND fills the damage window slower than it should.
|
||||
// Whole-block NonTrimmed → SkipBlock → advance → Pass N
|
||||
// revisits.
|
||||
let is_marginal = matches!(
|
||||
sense_key,
|
||||
scsi::SENSE_KEY_MEDIUM_ERROR | scsi::SENSE_KEY_ABORTED_COMMAND
|
||||
);
|
||||
if is_marginal && ctx.batch > 1 && !ctx.bisecting {
|
||||
if is_marginal && ctx.batch > 1 && !ctx.bisecting && ctx.bisect_on_marginal {
|
||||
return ReadAction::Bisect;
|
||||
}
|
||||
|
||||
|
|
@ -228,22 +403,77 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
|||
bad_count * 100 / ctx.damage_window.len()
|
||||
};
|
||||
|
||||
let pause_secs = if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
||||
// Inter-error pause on Pass 1 — wedge avoidance.
|
||||
//
|
||||
// Pre-2026-05-10 Pass 1 ran `pause_secs = 0` on all errors so
|
||||
// sweep would zoom past damage zones in seconds instead of
|
||||
// minutes. Empirically on the BU40N this caused firmware-wedge
|
||||
// events: each read failure leaves residual state in the drive's
|
||||
// firmware, and back-to-back errors without cooldown accumulate
|
||||
// toward a wedge threshold. We hit wedge on Dune Pt 2 after 5
|
||||
// errors in 43 s spread over a 140 MB damage cluster — even
|
||||
// though wall-clock pacing was slow, the drive had no breathing
|
||||
// room between exposures.
|
||||
//
|
||||
// New policy: Pass 1 keeps `pause = 0` on the CLEAN path
|
||||
// (successful reads zoom past damage windows fine), but every
|
||||
// failed read takes a PASS_1_FAIL_PAUSE_SECS pause before the
|
||||
// next read. Cost: ~5 s extra per scattered failure, ~30-60 s
|
||||
// total in a damage cluster — trivial compared to the alternative
|
||||
// of crashing the whole rip at 48%. Long failure streaks still
|
||||
// escalate via CONSECUTIVE_FAIL_LONG_PAUSE.
|
||||
let pause_secs = if !ctx.bisect_on_marginal {
|
||||
if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
||||
CONSECUTIVE_FAIL_LONG_PAUSE_SECS
|
||||
} else {
|
||||
PASS_1_FAIL_PAUSE_SECS
|
||||
}
|
||||
} else if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
||||
CONSECUTIVE_FAIL_LONG_PAUSE_SECS
|
||||
} else {
|
||||
POST_FAILURE_PAUSE_SECS
|
||||
};
|
||||
|
||||
// 7. Damage-jump: too many failures in window → skip ahead by an
|
||||
// escalating gap. Multiplier capped so we can't accidentally
|
||||
// skip the entire rest of the disc (observed 2026-05-07: a
|
||||
// saturated multiplier produced a 56 GB jump). Saturating
|
||||
// arithmetic on the sector calc as defence in depth.
|
||||
const MAX_JUMP_MULTIPLIER: u64 = 64; // 64 × 256 × batch sectors
|
||||
if ctx.damage_window.len() >= ctx.damage_window_max && bad_pct >= ctx.damage_threshold_pct {
|
||||
// 7. Damage-jump: too many failures → skip ahead by an escalating
|
||||
// gap. Multiplier capped so we can't accidentally skip the
|
||||
// entire rest of the disc (observed 2026-05-07: a saturated
|
||||
// multiplier produced a 56 GB jump). Saturating arithmetic on
|
||||
// the sector calc as defence in depth.
|
||||
//
|
||||
// Jump base bumped 2026-05-10 from 256 to 1024 sectors per
|
||||
// multiplier unit (= 64 MB first jump at batch=32, up from
|
||||
// 16 MB). The smaller base routinely landed jumps back inside
|
||||
// damage clusters of 100+ MB, each landing adding to the
|
||||
// firmware's wedge counter. 64 MB initial + 128 MB second +
|
||||
// 256 MB third clears almost any single-cluster damage
|
||||
// pattern we've seen in 2 jumps.
|
||||
//
|
||||
// Two triggers, evaluated in order:
|
||||
//
|
||||
// a. **Fast-entry** — `consecutive_outer_failures >= fast_jump_threshold`.
|
||||
// Fires on Pass 1 (threshold=4) so we don't spend ~40 min
|
||||
// grinding to fill a 16-block damage window before the
|
||||
// first jump on a damage zone we entered cleanly. Doesn't
|
||||
// fire on Pass N (threshold=u64::MAX).
|
||||
//
|
||||
// b. **Window-based** — original behaviour: 12% bad in a
|
||||
// sliding window of 16 outer reads. Pass N's only path,
|
||||
// and Pass 1's fallback if the failures are scattered
|
||||
// enough that we don't hit the consecutive threshold.
|
||||
const MAX_JUMP_MULTIPLIER: u64 = 64;
|
||||
let fast_trigger = !ctx.bisecting && ctx.consecutive_outer_failures >= ctx.fast_jump_threshold;
|
||||
let window_trigger =
|
||||
ctx.damage_window.len() >= ctx.damage_window_max && bad_pct >= ctx.damage_threshold_pct;
|
||||
if fast_trigger || window_trigger {
|
||||
let mult = ctx.jump_multiplier.min(MAX_JUMP_MULTIPLIER);
|
||||
let sectors = 256u64.saturating_mul(ctx.batch as u64).saturating_mul(mult);
|
||||
let sectors = JUMP_BASE_SECTORS
|
||||
.saturating_mul(ctx.batch as u64)
|
||||
.saturating_mul(mult);
|
||||
ctx.jump_multiplier = (ctx.jump_multiplier.saturating_mul(2)).min(MAX_JUMP_MULTIPLIER);
|
||||
// Reset the outer-failure counter so a long damaged region
|
||||
// doesn't keep firing fast-jump every read after the initial
|
||||
// jump fired. The window-based trigger handles further jumps.
|
||||
ctx.consecutive_outer_failures = 0;
|
||||
return ReadAction::JumpAhead {
|
||||
sectors,
|
||||
pause_secs: pause_secs + POST_JUMP_EXTRA_PAUSE_SECS,
|
||||
|
|
@ -285,16 +515,41 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
fn illegal_request_err() -> Error {
|
||||
Error::DiscRead {
|
||||
sector: 100,
|
||||
status: Some(2),
|
||||
sense: Some(ScsiSense {
|
||||
sense_key: scsi::SENSE_KEY_ILLEGAL_REQUEST,
|
||||
asc: 0x24,
|
||||
ascq: 0x00,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn medium_error_with_batch_gt_1_bisects() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
fn pass_n_marginal_with_batch_gt_1_bisects() {
|
||||
let mut ctx = ReadCtx::for_patch(32);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
assert_eq!(action, ReadAction::Bisect);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_1_marginal_skips_instead_of_bisecting() {
|
||||
// Pass 1's job is "fast and accurate" — leave bisection to
|
||||
// Pass N. A failed batch becomes SkipBlock (whole 32-sector
|
||||
// block marked NonTrimmed for Pass N to revisit).
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::SkipBlock { .. } => {}
|
||||
other => panic!("expected SkipBlock for Pass 1, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn medium_error_with_batch_1_skips() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
let mut ctx = ReadCtx::for_patch(1);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::SkipBlock { pause_secs } => assert!(pause_secs >= 1),
|
||||
|
|
@ -304,7 +559,7 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn medium_error_while_bisecting_does_not_recurse() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let mut ctx = ReadCtx::for_patch(32);
|
||||
ctx.bisecting = true;
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
|
|
@ -314,19 +569,165 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn hardware_error_aborts() {
|
||||
fn pass_1_jumps_after_4_consecutive_outer_failures() {
|
||||
// The fast-entry trigger: Pass 1 should JumpAhead after 4
|
||||
// consecutive outer-batch failures, BEFORE the 16-block
|
||||
// damage window has filled. Otherwise we spend ~40 minutes
|
||||
// of bisecting/grinding to fill the window before the first
|
||||
// jump on a damage zone we entered cleanly.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
// First three should NOT jump (still under threshold of 4).
|
||||
for _ in 0..3 {
|
||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
||||
assert!(
|
||||
!matches!(a, ReadAction::JumpAhead { .. }),
|
||||
"should not jump until 4 consecutive outer failures"
|
||||
);
|
||||
}
|
||||
// Fourth should jump.
|
||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
||||
assert!(
|
||||
matches!(a, ReadAction::JumpAhead { .. }),
|
||||
"expected JumpAhead at 4th consecutive outer failure, got {a:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_n_does_not_fast_jump() {
|
||||
// Pass N's whole reason to exist is to grind on bad ranges.
|
||||
// It should NOT bail after 4 consecutive failures the way
|
||||
// Pass 1 does — it bisects and skips with proper recovery.
|
||||
let mut ctx = ReadCtx::for_patch(32);
|
||||
for _ in 0..4 {
|
||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
||||
assert!(
|
||||
!matches!(a, ReadAction::JumpAhead { .. }),
|
||||
"Pass N must not fast-jump; got {a:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn outer_success_resets_consecutive_outer_failures() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
for _ in 0..3 {
|
||||
handle_read_error(&medium_err(), &mut ctx);
|
||||
}
|
||||
assert_eq!(ctx.consecutive_outer_failures, 3);
|
||||
// An outer-success (bisecting=false) should reset the counter.
|
||||
ctx.bisecting = false;
|
||||
ctx.on_success();
|
||||
assert_eq!(ctx.consecutive_outer_failures, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bisect_inner_success_does_not_reset_outer_counter() {
|
||||
let mut ctx = ReadCtx::for_patch(32);
|
||||
for _ in 0..3 {
|
||||
handle_read_error(&medium_err(), &mut ctx);
|
||||
}
|
||||
assert_eq!(ctx.consecutive_outer_failures, 3);
|
||||
// A successful inner-sector read during bisect is not the
|
||||
// same as escaping the bad outer batch.
|
||||
ctx.bisecting = true;
|
||||
ctx.on_success();
|
||||
assert_eq!(
|
||||
ctx.consecutive_outer_failures, 3,
|
||||
"bisect inner success must not reset outer-failure counter"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_1_hardware_error_jumps_ahead_not_aborts() {
|
||||
// New wedge-skip policy: Pass 1 (bisect_on_marginal=false)
|
||||
// should JumpAhead with a 1 GB skip + cooldown pause instead
|
||||
// of immediately aborting. Aborting on first wedge was the
|
||||
// pre-fix behavior that killed rips at 48% on damaged discs.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::JumpAhead {
|
||||
sectors,
|
||||
pause_secs,
|
||||
} => {
|
||||
assert_eq!(sectors, WEDGE_JUMP_SECTORS);
|
||||
assert_eq!(pause_secs, WEDGE_PAUSE_SECS);
|
||||
}
|
||||
other => panic!("expected JumpAhead, got {other:?}"),
|
||||
}
|
||||
assert_eq!(ctx.wedge_count, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_1_hardware_error_aborts_after_threshold() {
|
||||
// After WEDGE_ABORT_THRESHOLD consecutive wedges with no good
|
||||
// read in between, autorip should see a real AbortPass so it
|
||||
// can surface "drive is stuck, power-cycle required" to the
|
||||
// user — rather than looping forever on a permanently bricked
|
||||
// drive.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
for i in 0..WEDGE_ABORT_THRESHOLD - 1 {
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
assert!(
|
||||
matches!(action, ReadAction::JumpAhead { .. }),
|
||||
"iter {i}: expected JumpAhead, got {action:?}"
|
||||
);
|
||||
}
|
||||
// The Nth wedge crosses the threshold.
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
assert_eq!(action, ReadAction::AbortPass);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_failure_streak_extends_pause() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
fn pass_1_good_read_resets_wedge_count() {
|
||||
// A single successful read between wedges must clear the
|
||||
// skip counter — otherwise a disc with a few scattered bad
|
||||
// zones would eventually run out of skip budget even though
|
||||
// the drive was recovering between zones.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
for _ in 0..(WEDGE_ABORT_THRESHOLD - 1) {
|
||||
handle_read_error(&hardware_err(), &mut ctx);
|
||||
}
|
||||
assert_eq!(ctx.wedge_count, WEDGE_ABORT_THRESHOLD - 1);
|
||||
ctx.on_success();
|
||||
assert_eq!(ctx.wedge_count, 0);
|
||||
// After the success, we should still get JumpAhead (not
|
||||
// AbortPass) on the next wedge.
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
assert!(matches!(action, ReadAction::JumpAhead { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_n_hardware_error_still_aborts() {
|
||||
// Pass N (bisect_on_marginal=true) keeps the original
|
||||
// AbortPass behavior — single-sector recovery can't make
|
||||
// progress through a wedge, so the right answer is to bail
|
||||
// and let the outer layer decide.
|
||||
let mut ctx = ReadCtx::for_patch(1);
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
assert_eq!(action, ReadAction::AbortPass);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_1_illegal_request_also_routes_to_wedge_skip() {
|
||||
// ILLEGAL_REQUEST is the other half of the wedge family:
|
||||
// drive saying "I won't parse your CDB" after entering the
|
||||
// fast-fail state. Same treatment as HARDWARE_ERROR.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&illegal_request_err(), &mut ctx);
|
||||
assert!(matches!(action, ReadAction::JumpAhead { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_failure_streak_extends_pause_on_pass_n() {
|
||||
// Pass N keeps the cooldown behaviour: after many consecutive
|
||||
// failures, pauses extend to give the drive time to recover.
|
||||
// Pass 1 explicitly does NOT pause — see
|
||||
// `pass_1_does_not_pause_on_skip` below.
|
||||
let mut ctx = ReadCtx::for_patch(1);
|
||||
for _ in 0..15 {
|
||||
handle_read_error(&medium_err(), &mut ctx);
|
||||
}
|
||||
// After many consecutive failures we should be in the long-pause regime
|
||||
let final_action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match final_action {
|
||||
ReadAction::SkipBlock { pause_secs } => {
|
||||
|
|
@ -339,6 +740,28 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pass_1_pauses_briefly_on_skip_for_wedge_avoidance() {
|
||||
// Pre-2026-05-10 Pass 1 ran pause_secs=0 on all errors (zoom
|
||||
// past damage zones in seconds). That caused firmware wedges
|
||||
// on the BU40N — back-to-back errors with no cooldown built
|
||||
// up firmware state until the drive entered the wedge fast-
|
||||
// fail mode. New policy: a brief inter-error pause (5 s) on
|
||||
// Pass 1 to give the drive's firmware time to settle between
|
||||
// damage-zone exposures. Successful reads remain zero-pause
|
||||
// — only errors cost time, and only a few seconds per
|
||||
// scattered failure. Trivial cost compared to crashing the
|
||||
// whole rip at 48%.
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::SkipBlock { pause_secs } => {
|
||||
assert_eq!(pause_secs, PASS_1_FAIL_PAUSE_SECS);
|
||||
}
|
||||
other => panic!("expected SkipBlock, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn damage_window_fills_then_jumps() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
|
|
@ -369,6 +792,6 @@ mod tests {
|
|||
ctx.on_success();
|
||||
assert_eq!(ctx.consecutive_good, 1);
|
||||
assert_eq!(ctx.consecutive_failures, 0);
|
||||
assert_eq!(*ctx.damage_window.last().unwrap(), true);
|
||||
assert!(*ctx.damage_window.last().unwrap());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
239
src/disc/sweep.rs
Normal file
239
src/disc/sweep.rs
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
//! `Disc::sweep`'s consumer-side `Sink<WorkItem>`.
|
||||
//!
|
||||
//! Background: the original sweep loop runs strictly serialised —
|
||||
//! SCSI read → decrypt → seek + write → mapfile.record → next iter.
|
||||
//! On a healthy disc the SCSI read costs ~5-12 ms per 64 KB batch and
|
||||
//! the post-read work (decrypt 1-3 ms + file write + mapfile fsync
|
||||
//! 5-15 ms) adds another batch's worth of latency. The drive idles
|
||||
//! during the post-read work; throughput tops out at the *sum* of
|
||||
//! both costs.
|
||||
//!
|
||||
//! 0.17.11 introduced a bespoke producer/consumer split (the now-
|
||||
//! removed `disc/sweep_pipeline.rs`) to overlap the two stages. 0.18
|
||||
//! collapses that split — together with the analogous splits patch
|
||||
//! and mux need — onto the generic [`crate::io::Pipeline`] +
|
||||
//! [`crate::io::Sink`] primitive. This module is the sweep-specific
|
||||
//! `Sink` impl; the producer-side state machine (read_error context,
|
||||
//! decrypt, set_speed, halt) stays in `Disc::sweep` in `disc/mod.rs`.
|
||||
//!
|
||||
//! Correctness invariants preserved (same as 0.17.11):
|
||||
//! - Mapfile is single-writer (consumer-only). No locking.
|
||||
//! - All `read_error::ReadCtx` state stays on the producer thread.
|
||||
//! - `set_speed` calls happen on the producer thread (same thread that
|
||||
//! owns the `SectorReader`). No new SCSI concurrency.
|
||||
//! - Per-iteration ordering of file-write → mapfile-record is kept
|
||||
//! intact in the consumer (write before record), so the on-disk
|
||||
//! invariant "mapfile only marks Finished what the file has
|
||||
//! received" survives a crash mid-pass.
|
||||
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
|
||||
//! SCSI command in flight at a time, error-path timing identical,
|
||||
//! no new retry logic.
|
||||
|
||||
use std::io::{Seek, SeekFrom, Write};
|
||||
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
|
||||
|
||||
use crate::error::Error;
|
||||
use crate::io::{Flow, Sink};
|
||||
|
||||
use super::mapfile::{MapStats, Mapfile, SectorStatus};
|
||||
|
||||
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
|
||||
/// matches the existing zero_gap chunk size used by the pre-split
|
||||
/// sweep loop.
|
||||
const ZERO_CHUNK: usize = 65 * 1024;
|
||||
|
||||
/// Producer → Consumer messages. The consumer applies these in FIFO
|
||||
/// order; ordering of file writes and mapfile records across items is
|
||||
/// preserved.
|
||||
pub(super) enum WorkItem {
|
||||
/// Successful batch read. Producer has already decrypted `buf` if
|
||||
/// `opts.decrypt` was set. Consumer writes `buf` at `pos` and
|
||||
/// records the range as `Finished`.
|
||||
Good { pos: u64, buf: Vec<u8> },
|
||||
|
||||
/// Bisect inner-loop good single sector (already decrypted by the
|
||||
/// producer). 2048 bytes.
|
||||
BisectGood { pos: u64, buf: Box<[u8; 2048]> },
|
||||
|
||||
/// Bisect inner-loop bad single sector. Consumer writes 2048
|
||||
/// zeros at `pos` and records the sector as `NonTrimmed`.
|
||||
BisectBad { pos: u64 },
|
||||
|
||||
/// Whole-batch zero-fill (failed batch on `SkipBlock`, or the
|
||||
/// failed batch portion of `JumpAhead`). Consumer streams zeros
|
||||
/// across `[pos, pos+len)` and records the range as `NonTrimmed`.
|
||||
SkipFill { pos: u64, len: u64 },
|
||||
|
||||
/// Gap fill following a `JumpAhead`. Same effect as `SkipFill`;
|
||||
/// distinguished only so future logging / instrumentation can
|
||||
/// tell them apart without parsing a flag.
|
||||
GapFill { pos: u64, len: u64 },
|
||||
|
||||
/// Producer wants the latest mapfile stats for the progress
|
||||
/// callback. Consumer responds on `prog_tx` with a fresh
|
||||
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
|
||||
/// drained the previous snapshot, the new one is silently
|
||||
/// dropped — the producer's local cache stays current enough.
|
||||
StatsRequest,
|
||||
}
|
||||
|
||||
/// Snapshot the consumer sends back to the producer for the progress
|
||||
/// callback.
|
||||
pub(super) struct ProgressSnapshot {
|
||||
pub stats: MapStats,
|
||||
pub bad_ranges: Vec<(u64, u64)>,
|
||||
}
|
||||
|
||||
/// Final summary returned by the consumer thread on shutdown — what
|
||||
/// `SweepSink::close` produces, surfaced to the producer via
|
||||
/// `Pipeline::finish`.
|
||||
pub(super) struct ConsumerSummary {
|
||||
pub stats: MapStats,
|
||||
}
|
||||
|
||||
/// Drain any pending progress snapshots from the consumer. Returns
|
||||
/// the most recent one, if any. The producer caches it and uses it
|
||||
/// for subsequent progress callbacks until a fresh one arrives.
|
||||
pub(super) fn try_recv_progress(rx: &Receiver<ProgressSnapshot>) -> Option<ProgressSnapshot> {
|
||||
let mut latest = None;
|
||||
while let Ok(snap) = rx.try_recv() {
|
||||
latest = Some(snap);
|
||||
}
|
||||
latest
|
||||
}
|
||||
|
||||
/// `Sink<WorkItem>` for sweep. Owns the writeback file + mapfile +
|
||||
/// progress back-channel. `apply` carries the file-write +
|
||||
/// mapfile.record per item; `close` drains the writeback pipeline,
|
||||
/// fsyncs the ISO, and flushes the mapfile.
|
||||
pub(super) struct SweepSink {
|
||||
file: crate::io::WritebackFile,
|
||||
map: Mapfile,
|
||||
/// `sync_all`-on-failure-is-an-error iff the output is a regular
|
||||
/// file. `/dev/null` and pipes always fail `sync_all`; that's not
|
||||
/// a real error.
|
||||
is_regular: bool,
|
||||
/// Back-channel for `StatsRequest` responses. The producer caches
|
||||
/// the latest snapshot and uses it for the progress callback;
|
||||
/// dropped sends on a full channel are by design.
|
||||
prog_tx: SyncSender<ProgressSnapshot>,
|
||||
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. Held
|
||||
/// in the sink so each apply call doesn't reallocate.
|
||||
zero: Box<[u8; ZERO_CHUNK]>,
|
||||
}
|
||||
|
||||
impl SweepSink {
|
||||
/// Construct a new `SweepSink` plus the matching progress
|
||||
/// receiver. Channel depth on the back-channel is `1` — the
|
||||
/// producer's cache is the source of truth between snapshots.
|
||||
pub(super) fn new(
|
||||
file: crate::io::WritebackFile,
|
||||
map: Mapfile,
|
||||
is_regular: bool,
|
||||
) -> (Self, Receiver<ProgressSnapshot>) {
|
||||
let (prog_tx, prog_rx) = sync_channel::<ProgressSnapshot>(1);
|
||||
let sink = SweepSink {
|
||||
file,
|
||||
map,
|
||||
is_regular,
|
||||
prog_tx,
|
||||
zero: Box::new([0u8; ZERO_CHUNK]),
|
||||
};
|
||||
(sink, prog_rx)
|
||||
}
|
||||
}
|
||||
|
||||
impl Sink<WorkItem> for SweepSink {
|
||||
type Output = ConsumerSummary;
|
||||
|
||||
fn apply(&mut self, item: WorkItem) -> Result<Flow, Error> {
|
||||
match item {
|
||||
WorkItem::Good { pos, buf } => {
|
||||
// Decrypt is on the producer; consumer assumes plaintext.
|
||||
let len = buf.len() as u64;
|
||||
self.file
|
||||
.seek(SeekFrom::Start(pos))
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.file
|
||||
.write_all(&buf)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.map
|
||||
.record(pos, len, SectorStatus::Finished)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
WorkItem::BisectGood { pos, buf } => {
|
||||
self.file
|
||||
.seek(SeekFrom::Start(pos))
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.file
|
||||
.write_all(&buf[..])
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.map
|
||||
.record(pos, 2048, SectorStatus::Finished)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
WorkItem::BisectBad { pos } => {
|
||||
self.file
|
||||
.seek(SeekFrom::Start(pos))
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.file
|
||||
.write_all(&self.zero[..2048])
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
self.map
|
||||
.record(pos, 2048, SectorStatus::NonTrimmed)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
|
||||
self.file
|
||||
.seek(SeekFrom::Start(pos))
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
// Subsequent writes are sequential; `WritebackFile`'s
|
||||
// seek-elision keeps them on the writeback pipeline path.
|
||||
let mut filled = 0u64;
|
||||
while filled < len {
|
||||
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
|
||||
self.file
|
||||
.write_all(&self.zero[..chunk])
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
filled += chunk as u64;
|
||||
}
|
||||
self.map
|
||||
.record(pos, len, SectorStatus::NonTrimmed)
|
||||
.map_err(|e| Error::IoError { source: e })?;
|
||||
}
|
||||
WorkItem::StatsRequest => {
|
||||
let stats = self.map.stats();
|
||||
let bad_ranges = self.map.ranges_with(&[
|
||||
SectorStatus::NonTrimmed,
|
||||
SectorStatus::Unreadable,
|
||||
SectorStatus::NonScraped,
|
||||
SectorStatus::NonTried,
|
||||
]);
|
||||
// Best-effort: drop on backpressure; producer's cache
|
||||
// stays current enough.
|
||||
let _ = self
|
||||
.prog_tx
|
||||
.try_send(ProgressSnapshot { stats, bad_ranges });
|
||||
}
|
||||
}
|
||||
Ok(Flow::Continue)
|
||||
}
|
||||
|
||||
fn close(mut self) -> Result<Self::Output, Error> {
|
||||
// Drain the writeback pipeline + fsync the ISO, then persist
|
||||
// any pending mapfile state. Same finalisation order as the
|
||||
// pre-Pipeline consumer loop.
|
||||
if let Err(e) = self.file.sync_all() {
|
||||
if self.is_regular {
|
||||
return Err(Error::IoError { source: e });
|
||||
}
|
||||
// Non-regular outputs (/dev/null, pipes) always fail
|
||||
// sync_all; that's not a real error.
|
||||
}
|
||||
self.map.flush().map_err(|e| Error::IoError { source: e })?;
|
||||
|
||||
Ok(ConsumerSummary {
|
||||
stats: self.map.stats(),
|
||||
})
|
||||
}
|
||||
}
|
||||
118
src/drive/mod.rs
118
src/drive/mod.rs
|
|
@ -69,6 +69,16 @@ pub struct Drive {
|
|||
halt: Arc<AtomicBool>,
|
||||
/// Event handler — fires for read errors and library-level state changes.
|
||||
event_fn: Option<Box<dyn Fn(Event) + Send>>,
|
||||
/// Linux only: raw fd for the corresponding block device (`/dev/sr*`)
|
||||
/// used as a recovery fallback when SCSI READ via `/dev/sg*` returns
|
||||
/// an error. The kernel `sr_mod` driver auto-retries failed reads
|
||||
/// (~5× per command) — historically the reason `dd if=/dev/sr0`
|
||||
/// recovers ~50% of bad sectors that single-shot `SG_IO` READ
|
||||
/// misses on the same drive. `None` when the block device couldn't
|
||||
/// be resolved or opened (no fallback in that case; SCSI read
|
||||
/// errors propagate as before).
|
||||
#[cfg(target_os = "linux")]
|
||||
block_dev_fd: Option<std::os::unix::io::RawFd>,
|
||||
}
|
||||
|
||||
impl Drive {
|
||||
|
|
@ -87,6 +97,9 @@ impl Drive {
|
|||
None => (None, None, None),
|
||||
};
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
let block_dev_fd = open_block_device_for_sg(device);
|
||||
|
||||
Ok(Drive {
|
||||
scsi: transport,
|
||||
driver,
|
||||
|
|
@ -96,6 +109,8 @@ impl Drive {
|
|||
device_path: device.to_string_lossy().to_string(),
|
||||
halt: Arc::new(AtomicBool::new(false)),
|
||||
event_fn: None,
|
||||
#[cfg(target_os = "linux")]
|
||||
block_dev_fd,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -483,6 +498,54 @@ impl Drive {
|
|||
scsi_status = status,
|
||||
"Drive::read checked_exec failed"
|
||||
);
|
||||
|
||||
// /dev/sr0 pread fallback (Linux only). The kernel
|
||||
// sr_mod driver auto-retries failed reads (~5× per
|
||||
// command). Empirically (BU40N + Dune Part 2 UHD,
|
||||
// 2026-05-08) dd via /dev/sr0 recovers ~50% of bad
|
||||
// sectors that a single-shot SG_IO READ misses.
|
||||
#[cfg(target_os = "linux")]
|
||||
if recovery {
|
||||
if let Some(fd) = self.block_dev_fd {
|
||||
let len = count as usize * 2048;
|
||||
if buf.len() >= len {
|
||||
let offset = lba as i64 * 2048;
|
||||
// Drop kernel cache for this region so we get
|
||||
// a fresh device read, not stale page-cache
|
||||
// data from a prior successful neighbour read.
|
||||
let _ = unsafe {
|
||||
libc::posix_fadvise(
|
||||
fd,
|
||||
offset,
|
||||
len as i64,
|
||||
libc::POSIX_FADV_DONTNEED,
|
||||
)
|
||||
};
|
||||
let n = unsafe {
|
||||
libc::pread(fd, buf.as_mut_ptr() as *mut libc::c_void, len, offset)
|
||||
};
|
||||
if n == len as isize {
|
||||
tracing::info!(
|
||||
target: "freemkv::drive",
|
||||
lba,
|
||||
count,
|
||||
bytes = len,
|
||||
"Drive::read recovered via /dev/sr0 pread fallback"
|
||||
);
|
||||
return Ok(len);
|
||||
}
|
||||
tracing::debug!(
|
||||
target: "freemkv::drive",
|
||||
lba,
|
||||
count,
|
||||
pread_ret = n as i64,
|
||||
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
|
||||
"/dev/sr0 pread fallback also failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Err(Error::DiscRead {
|
||||
sector: lba as u64,
|
||||
status: Some(status),
|
||||
|
|
@ -586,6 +649,61 @@ impl Drop for Drive {
|
|||
fn drop(&mut self) {
|
||||
self.cleanup();
|
||||
// SgIoTransport::drop() runs next, calling libc::close(fd)
|
||||
#[cfg(target_os = "linux")]
|
||||
if let Some(fd) = self.block_dev_fd.take() {
|
||||
unsafe { libc::close(fd) };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a `/dev/sg*` path to the corresponding `/dev/sr*` block
|
||||
/// device by walking sysfs, then open it for read (no `O_DIRECT` —
|
||||
/// `posix_fadvise(POSIX_FADV_DONTNEED)` flushes the cache before each
|
||||
/// pread, which avoids buffer-alignment requirements while still
|
||||
/// forcing fresh device reads).
|
||||
///
|
||||
/// Returns `None` on any error (sysfs not present, no matching block
|
||||
/// device, open failed). Callers treat that as "no fallback available"
|
||||
/// and propagate the original SCSI READ error.
|
||||
#[cfg(target_os = "linux")]
|
||||
fn open_block_device_for_sg(sg_path: &Path) -> Option<std::os::unix::io::RawFd> {
|
||||
let basename = sg_path.file_name()?.to_str()?;
|
||||
if !basename.starts_with("sg") {
|
||||
return None;
|
||||
}
|
||||
let sysfs_dir = format!("/sys/class/scsi_generic/{}/device/block", basename);
|
||||
let entries = std::fs::read_dir(&sysfs_dir).ok()?;
|
||||
let block_name = entries
|
||||
.flatten()
|
||||
.find_map(|e| e.file_name().into_string().ok())?;
|
||||
let block_path = format!("/dev/{}", block_name);
|
||||
|
||||
let mut bytes = block_path.as_bytes().to_vec();
|
||||
bytes.push(0);
|
||||
let fd = unsafe {
|
||||
libc::open(
|
||||
bytes.as_ptr() as *const libc::c_char,
|
||||
libc::O_RDONLY | libc::O_CLOEXEC,
|
||||
)
|
||||
};
|
||||
if fd < 0 {
|
||||
tracing::debug!(
|
||||
target: "freemkv::drive",
|
||||
sg = basename,
|
||||
block_path,
|
||||
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
|
||||
"Failed to open block device for fallback; sr0 fallback disabled"
|
||||
);
|
||||
None
|
||||
} else {
|
||||
tracing::info!(
|
||||
target: "freemkv::drive",
|
||||
sg = basename,
|
||||
block_path,
|
||||
fd,
|
||||
"Opened /dev/sr* as recovery fallback for failed SCSI reads"
|
||||
);
|
||||
Some(fd)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
166
src/halt.rs
Normal file
166
src/halt.rs
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
//! One-bit cooperative cancellation flag.
|
||||
//!
|
||||
//! `Halt` is a clonable token wrapping `Arc<AtomicBool>`. Pass clones into
|
||||
//! every long-running loop; the loop polls `is_cancelled()` and bails out
|
||||
//! cleanly. Calling `cancel()` from any clone flips the shared flag, and
|
||||
//! every other clone observes it on its next poll.
|
||||
//!
|
||||
//! Why: `Ordering::Relaxed` is sufficient on both load and store because
|
||||
//! this flag is purely advisory — no other memory operations piggyback on
|
||||
//! it for happens-before ordering. Callers that need to publish data
|
||||
//! across threads do so via channels or other synchronization, not via
|
||||
//! this bit.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
/// Clonable, infallible cooperative-cancellation token.
|
||||
///
|
||||
/// Clones share the same underlying flag. `cancel()` is one-way; there is
|
||||
/// no `reset()` by design — construct a fresh `Halt` for a fresh
|
||||
/// operation.
|
||||
///
|
||||
/// Construct with [`Halt::new`] (or [`Halt::default`]). The `Default`
|
||||
/// impl forwards to `new()` — both produce a fresh, uncancelled token.
|
||||
/// The pair exists because clippy's `new_without_default` lint requires
|
||||
/// `Default` whenever a public `new()` is present, even when the two
|
||||
/// would do exactly the same thing.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Halt(Arc<AtomicBool>);
|
||||
|
||||
impl Halt {
|
||||
/// Construct a fresh, uncancelled token.
|
||||
pub fn new() -> Self {
|
||||
Self(Arc::new(AtomicBool::new(false)))
|
||||
}
|
||||
|
||||
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. Useful as a
|
||||
/// bridge during the 0.18 deprecation window: callers that already
|
||||
/// hold an `Arc<AtomicBool>` (e.g. `Drive::halt_flag()`, the
|
||||
/// deprecated `DiscStream::set_halt`) can adopt the new token API
|
||||
/// without changing the underlying flag.
|
||||
///
|
||||
/// Cancelling either side flips the same bit — the wrapping `Halt`
|
||||
/// and the original `Arc` are two views over one shared flag.
|
||||
pub fn from_arc(flag: Arc<AtomicBool>) -> Self {
|
||||
Self(flag)
|
||||
}
|
||||
|
||||
/// Borrow the underlying `Arc<AtomicBool>`. Used at boundaries with
|
||||
/// pre-`Halt` APIs that still take an `Arc<AtomicBool>` directly
|
||||
/// (`CopyOptions::halt`, the deprecated `DiscStream::set_halt`).
|
||||
/// Round 3 deletes those boundaries and this accessor with them.
|
||||
pub fn as_arc(&self) -> &Arc<AtomicBool> {
|
||||
&self.0
|
||||
}
|
||||
|
||||
/// Flip the shared flag to cancelled. Idempotent.
|
||||
pub fn cancel(&self) {
|
||||
self.0.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Read the shared flag.
|
||||
pub fn is_cancelled(&self) -> bool {
|
||||
self.0.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for Halt {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn fresh_is_not_cancelled() {
|
||||
let h = Halt::new();
|
||||
assert!(!h.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancel_flips_state() {
|
||||
let h = Halt::new();
|
||||
assert!(!h.is_cancelled());
|
||||
h.cancel();
|
||||
assert!(h.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancel_is_idempotent() {
|
||||
let h = Halt::new();
|
||||
h.cancel();
|
||||
h.cancel();
|
||||
assert!(h.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clone_shares_state() {
|
||||
let original = Halt::new();
|
||||
let cloned = original.clone();
|
||||
assert!(!original.is_cancelled());
|
||||
assert!(!cloned.is_cancelled());
|
||||
|
||||
// Cancel via the clone; the original observes it.
|
||||
cloned.cancel();
|
||||
assert!(original.is_cancelled());
|
||||
assert!(cloned.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clone_shares_state_reverse_direction() {
|
||||
let original = Halt::new();
|
||||
let cloned = original.clone();
|
||||
|
||||
// Cancel via the original; the clone observes it.
|
||||
original.cancel();
|
||||
assert!(cloned.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clone_shares_state_across_threads() {
|
||||
let h = Halt::new();
|
||||
let h2 = h.clone();
|
||||
let handle = std::thread::spawn(move || {
|
||||
h2.cancel();
|
||||
});
|
||||
handle.join().unwrap();
|
||||
assert!(h.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn from_arc_shares_state() {
|
||||
// The 0.18 deprecation-window bridge: a Halt built from an
|
||||
// existing Arc<AtomicBool> must be a *view* over the same bit,
|
||||
// not a fresh copy. Cancelling either side flips both.
|
||||
let arc = Arc::new(AtomicBool::new(false));
|
||||
let halt = Halt::from_arc(arc.clone());
|
||||
assert!(!halt.is_cancelled());
|
||||
assert!(!arc.load(Ordering::Relaxed));
|
||||
|
||||
// Cancel via the wrapping Halt; the original Arc observes it.
|
||||
halt.cancel();
|
||||
assert!(arc.load(Ordering::Relaxed));
|
||||
|
||||
// Conversely: flip the Arc directly; the Halt view observes it.
|
||||
let arc2 = Arc::new(AtomicBool::new(false));
|
||||
let halt2 = Halt::from_arc(arc2.clone());
|
||||
arc2.store(true, Ordering::Relaxed);
|
||||
assert!(halt2.is_cancelled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_arc_returns_backing_flag() {
|
||||
// `as_arc()` must hand back the *same* Arc, not a clone of a
|
||||
// different bit. Verified by writing through the borrowed Arc
|
||||
// and observing through the Halt.
|
||||
let halt = Halt::new();
|
||||
let arc = halt.as_arc().clone();
|
||||
assert!(!halt.is_cancelled());
|
||||
arc.store(true, Ordering::Relaxed);
|
||||
assert!(halt.is_cancelled());
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue