Stop documenting the recovery API that 1.6.0 deleted
Disc::sweep, Disc::patch, Disc::copy, SweepOptions and PatchOptions have zero occurrences in src/ — recovery moved to freemkv-engine — but they were still documented in 30 places across README.md, TROUBLESHOOTING.md, six files under docs/, seven src/ doc comments and a Cargo.toml comment. README.md is the crate's GitHub front page and carried a full multi-pass code example that cannot compile. Two of the src/ references were intra-doc LINKS to deleted items ([`disc::Disc::copy`], [`disc::Disc::patch`] in scsi/mod.rs). They produced no warning on a normal `cargo doc` only because they sit on pub(crate) items; `--document-private-items` reports both, and they are gone now. The README example is deleted rather than rewritten against the engine's API: libfreemkv documenting a downstream crate's API on its own front page is the drift that produced this, and it cannot even depend on it. The src/ references become plain code spans naming freemkv_engine::recovery::* — deliberately not links, for the same reason. docs/rip-recovery.md was 202 lines about relocated code. It now documents only what this crate owns — Drive::read, SenseFamily, DiscStream's adaptive batch halving — plus the read-path design constraints, which belong with the code that enforces them, and points at freemkv-engine/src/recovery/ for the strategy. api-design.md's module tree is regenerated from the real src/disc/ and src/drive/ layouts instead of hand-patched; it had listed sweep.rs, patch.rs, mapfile.rs and read_error.rs, none of which exist. Three stale facts surfaced while rewriting and are corrected: the read timeouts are 10 s / 60 s, not the documented 1.5 s / 30 s; Drive::reset and SgIoTransport::reset no longer exist at all, so "no SCSI reset from any read path" is now stated as the stronger fact it has become; and verify_title, listed as a progress-emitting operation, was removed entirely. CHANGELOG.md keeps its references — those are the historical record of the releases that shipped the API.
This commit is contained in:
Matthew Jackson
+64
-168
@@ -1,136 +1,38 @@
|
||||
# Rip recovery — three-layer architecture
|
||||
# Rip recovery — what libfreemkv owns
|
||||
|
||||
`libfreemkv` supports a multi-stage rip model for damaged or protection-bearing
|
||||
discs: a fast forward sweep that tolerates read failures, in-loop request-size
|
||||
adaptation that survives transient drive trouble without bailing, and targeted
|
||||
retry passes against a persistent bad-range map. The stream pipeline
|
||||
(`DiscStream` + `input`/`output`) operates against the resulting ISO image, so
|
||||
the mux stage never touches the drive.
|
||||
**Recovery strategy moved OUT of this crate in 1.6.0.** The forward sweep, the
|
||||
targeted retry pass, the ddrescue mapfile, damage classification and the
|
||||
multipass loop now live in the **`freemkv-engine`** crate as
|
||||
`freemkv_engine::recovery::{copy, sweep, patch}`. The dependency runs
|
||||
engine → libfreemkv, so this crate cannot call into the engine; front-ends
|
||||
(`freemkv` CLI, autorip) get recovery from the engine directly.
|
||||
|
||||
Recovery is layered cleanly. Each layer has one responsibility and does not
|
||||
reach into the others.
|
||||
What stayed here are the two layers underneath the strategy: the single-shot
|
||||
read primitive, and the in-stream request-size adaptation that sits in front of
|
||||
it. This document covers those, plus the design constraints they encode — the
|
||||
constraints are the reason the strategy above them looks the way it does, so
|
||||
they belong with the code that enforces them.
|
||||
|
||||
For the strategy itself — damage-jump thresholds, pass ordering, mapfile status
|
||||
state machine, wedge detection — read `freemkv-engine/src/recovery/`.
|
||||
|
||||
| Layer | Where it lives | What it does |
|
||||
|-------|---------------|--------------|
|
||||
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
|
||||
| 1 — Bad-range retry | **`freemkv-engine`** (`recovery::patch`) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
|
||||
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
|
||||
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
|
||||
| 3 — In-loop request adaptation | `DiscStream::fill_extents` in `src/mux/disc.rs` | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
|
||||
|
||||
The library exposes flat verbs; the caller drives the multipass loop. Autorip
|
||||
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
|
||||
clean or the configured retry budget is exhausted, then hands the ISO off to
|
||||
the mux pipeline. The `freemkv` CLI does the same shape, but as of 1.6.0 the
|
||||
loop itself (including the multi-title rip loop) lives one layer up, in the
|
||||
shared `freemkv-engine` crate, with a terminal-output progress sink plugged
|
||||
into it as the `Sink`. Layer 3 runs inside any consumer of `DiscStream`
|
||||
(direct PES pipeline, ISO playback, etc.) without caller involvement.
|
||||
Layer 2 also translates drive facts: [`SenseFamily`](../src/scsi/mod.rs)
|
||||
classifies SCSI sense data into the categories the engine's strategy routes on
|
||||
(marginal vs. hardware vs. not-ready). Getting that classification wrong
|
||||
silently misroutes recovery, which is why it lives next to the transport rather
|
||||
than in the strategy.
|
||||
|
||||
Three primitives compose the disc-side flow:
|
||||
Layer 3 runs inside any consumer of `DiscStream` — direct PES pipeline, ISO
|
||||
playback — without caller involvement, and applies whether or not the engine's
|
||||
recovery is in play.
|
||||
|
||||
| Primitive | What it does |
|
||||
|---------------------------|-----------------------------------------------------------------------|
|
||||
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
|
||||
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
|
||||
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
|
||||
|
||||
## Data model
|
||||
|
||||
### Mapfile
|
||||
|
||||
Format: [ddrescue](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)-compatible
|
||||
plain text, greppable, tool-interoperable. Flushed to disk on every `record()`
|
||||
so a crashed rip loses at most one block.
|
||||
|
||||
```
|
||||
# Rescue Logfile. Created by libfreemkv v0.13.6
|
||||
# Current pos / status / pass / pass_time
|
||||
0x000000000 ? 1 0
|
||||
# pos size status
|
||||
0x000000000 0x12a35d000 +
|
||||
0x12a35d000 0x000003000 -
|
||||
0x12a360000 0x009c4a000 +
|
||||
0x12d00a000 0x000064000 *
|
||||
```
|
||||
|
||||
Status characters match ddrescue:
|
||||
|
||||
| Char | Meaning |
|
||||
|------|----------------------------------------------------|
|
||||
| `?` | Not yet attempted |
|
||||
| `*` | Fast-pass failed; needs edge-trim |
|
||||
| `/` | Trimmed; interior needs sector scrape |
|
||||
| `-` | Unreadable this session |
|
||||
| `+` | Finished (good) |
|
||||
|
||||
Position and size are hex byte offsets into the ISO.
|
||||
|
||||
### `SweepOptions` and `PatchOptions`
|
||||
|
||||
The library no longer dispatches between sweep and patch internally — the
|
||||
caller picks the verb explicitly per pass. The two option structs are flat
|
||||
and have no overlap:
|
||||
|
||||
```rust
|
||||
SweepOptions {
|
||||
decrypt: true,
|
||||
resume: false,
|
||||
batch_sectors: None,
|
||||
skip_on_error: true, // damage-jump + zero-fill on read failure
|
||||
progress: Some(&reporter),
|
||||
halt: Some(flag),
|
||||
}
|
||||
|
||||
PatchOptions {
|
||||
decrypt: true,
|
||||
block_sectors: None,
|
||||
full_recovery: true,
|
||||
reverse: true, // walk bad ranges high → low LBA
|
||||
wedged_threshold: 50,
|
||||
progress: Some(&reporter),
|
||||
halt: Some(flag),
|
||||
}
|
||||
```
|
||||
|
||||
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
|
||||
|
||||
- No mapfile → `sweep` (fresh Pass 1).
|
||||
- Mapfile with `?` ranges → `sweep` with `resume: true`.
|
||||
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
|
||||
- Mapfile clean → done; no further pass needed.
|
||||
|
||||
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
|
||||
lines of `Mapfile::stats()` checks.
|
||||
|
||||
## Algorithm
|
||||
|
||||
### Pass 1 — fast sweep (`Disc::sweep`)
|
||||
|
||||
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
|
||||
2. On success: write data to ISO, mark `+`, advance.
|
||||
3. On failure (with `multipass`): zero-fill, mark `*`, advance.
|
||||
4. Track a sliding window of the last 16 ECC block results. When ≥12% are failures
|
||||
→ **damage-jump**: skip ahead by `1024×batch×multiplier` sectors (64 MB base for
|
||||
UHD). Double the multiplier on each jump (64→128→256→512 MB...). Zero-fill the gap as `*`.
|
||||
5. On 16 consecutive good reads: reset jump multiplier to 1, restore max read speed.
|
||||
6. Speed control: damage zone entry → minimum speed, exit → maximum speed.
|
||||
7. Only transport failures (USB bridge crash) abort the pass.
|
||||
|
||||
Pass 1 completes when every byte has been visited (either `+` or `*`).
|
||||
|
||||
### Pass 2+ — patch (`Disc::patch`)
|
||||
|
||||
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
|
||||
(walks ranges from highest LBA to lowest, within each range from end to start).
|
||||
|
||||
1. Issue a single-sector read with 60 s timeout (`recovery=true`). Drive firmware
|
||||
does its own ECC recovery inside that window.
|
||||
2. On success: write the good bytes into the ISO, mark `+`.
|
||||
3. On failure with non-marginal SCSI sense: bail immediately (drive won't produce data).
|
||||
4. On failure with marginal sense: mark `-`, continue.
|
||||
5. Update the mapfile after every block — crash-safe resume.
|
||||
6. Wedged-drive exit: 50 consecutive failures with zero recovery → bail this pass.
|
||||
|
||||
### In-stream — adaptive batch halving (`DiscStream::fill_extents`)
|
||||
## In-stream — adaptive batch halving (`DiscStream::fill_extents`)
|
||||
|
||||
When a consumer reads a `DiscStream` directly (no ISO intermediate),
|
||||
`fill_extents` runs an adaptive sizer in front of `Drive::read`:
|
||||
@@ -144,59 +46,53 @@ When a consumer reads a `DiscStream` directly (no ISO intermediate),
|
||||
`EventKind::SectorSkipped`) when `skip_errors` is set, otherwise return
|
||||
`Err(DiscRead)`.
|
||||
|
||||
This is layer 3. It exists so a transient single-sector glitch in a 32-sector
|
||||
batch can be isolated and read individually without the caller needing to
|
||||
implement retry logic.
|
||||
This exists so a transient single-sector glitch inside a 32-sector batch can be
|
||||
isolated and read individually without the caller implementing retry logic. See
|
||||
[`src/event.rs`](../src/event.rs) for the emitted events.
|
||||
|
||||
## Design choices
|
||||
|
||||
**`Drive::read` is single-shot.** No inline retry phases, no SCSI reset,
|
||||
no eject cycle. The `recovery` flag controls only the per-CDB timeout
|
||||
(1.5 s vs. 30 s); on any failure it returns `Err(DiscRead)` immediately.
|
||||
Inline recovery (5× gentle retry → close + SCSI reset + reopen → 5× more)
|
||||
was removed in 0.13.6. See the stop-wedge postmortem (2026-04-25) for rationale:
|
||||
the inline reset on the LG BU40N (Initio USB-SATA bridge)
|
||||
wedged drive firmware below the bridge without ever recovering a sector,
|
||||
and the gentle-retry phase produced long stretches of 0 KB/s with no
|
||||
recoveries to show for it. Recovery responsibility is now layered: layer 1
|
||||
handles ranges, layer 3 handles request size, neither touches the
|
||||
wedge-prone reset path.
|
||||
These are constraints on the read path, enforced here and relied on by the
|
||||
engine's strategy.
|
||||
|
||||
**No `MODE SELECT` to disable drive retries.** Neither ddrescue
|
||||
nor any consumer ripper does this. Drive firmware has access to raw analog signal, laser
|
||||
power control, and drive-specific ECC tuning that userspace can't replicate —
|
||||
disabling it throws away recovery headroom on marginal sectors. We fail fast
|
||||
via short SG_IO timeouts in pass 1 and let the firmware work the long timeout
|
||||
in pass 2 / patch.
|
||||
**`Drive::read` is single-shot.** No inline retry phases, no SCSI reset, no
|
||||
eject cycle. The `recovery` flag controls only the per-CDB timeout (10 s vs.
|
||||
60 s); on any failure it returns `Err(DiscRead)` immediately. Inline recovery
|
||||
(5× gentle retry → close + SCSI reset + reopen → 5× more) was removed in
|
||||
0.13.6. See the stop-wedge postmortem (2026-04-25) for rationale: the inline
|
||||
reset on the LG BU40N (Initio USB-SATA bridge) wedged drive firmware below the
|
||||
bridge without ever recovering a sector, and the gentle-retry phase produced
|
||||
long stretches of 0 KB/s with nothing to show for it. Recovery responsibility
|
||||
is layered instead: layer 1 handles ranges, layer 3 handles request size,
|
||||
neither touches the wedge-prone reset path.
|
||||
|
||||
**No SCSI reset from any retry path.** `SgIoTransport::reset` (Linux) is
|
||||
trimmed to a kernel SG_IO state flush plus ALLOW MEDIUM REMOVAL — the
|
||||
`SG_SCSI_RESET` ioctl and STOP/START UNIT escalation were removed in 0.13.6.
|
||||
The macOS reset (which had been a no-op) was removed entirely. The top-level
|
||||
`scsi::reset()` / `reset_with_timeout()` / `reset_blocking()` wrappers were
|
||||
also removed (no callers). The remaining `Drive::reset()` is only invoked
|
||||
explicitly by callers that need an eject-cycle escape hatch — it is never
|
||||
reached from a read path.
|
||||
**No `MODE SELECT` to disable drive retries.** Neither ddrescue nor any
|
||||
consumer ripper does this. Drive firmware has access to raw analog signal,
|
||||
laser power control and drive-specific ECC tuning that userspace cannot
|
||||
replicate — disabling it throws away recovery headroom on marginal sectors. The
|
||||
fast pass fails quickly via short SG_IO timeouts and lets the firmware work the
|
||||
long timeout during retry.
|
||||
|
||||
**ISO intermediate, even for single-pass.** Pass 1 always writes an ISO. The
|
||||
mux stage reads the ISO via `FileSectorSource`. For single-pass (no retries),
|
||||
this adds ~2-3 min (local disk mux) but gains resumability across crashes,
|
||||
**No SCSI reset from any read path.** There is no reset escape hatch on
|
||||
`Drive` at all: the `SG_SCSI_RESET` ioctl and STOP/START UNIT escalation went in
|
||||
0.13.6, the macOS reset (always a no-op) was removed entirely, and the
|
||||
top-level `scsi::reset()` wrappers went with their last callers. The only
|
||||
remaining reset is a Windows-specific device-level helper in
|
||||
[`src/scsi/windows.rs`](../src/scsi/windows.rs), never reached from a read.
|
||||
|
||||
**ISO intermediate, even for single-pass.** The engine's Pass 1 always writes
|
||||
an ISO, and the mux stage reads it back via `FileSectorSource`. For a
|
||||
no-retry rip this costs a few minutes but buys resumability across crashes,
|
||||
re-muxability without re-ripping, and a persistent forensic artifact. Callers
|
||||
who need pure speed can bypass and use `DiscStream::new(Box::new(drive), …)`
|
||||
directly — the lib doesn't forbid it, and layer 3 (adaptive batch halving)
|
||||
still applies there.
|
||||
|
||||
**Mapfile in ddrescue format.** Plain text so users can `less` it, `diff` it,
|
||||
or feed it to ddrescue's own tooling. Crash-safe (flush-per-record). Entries
|
||||
coalesce on adjacent same-status ranges so files stay small.
|
||||
|
||||
**Patches target `-`, `*`, `/`, and `?` alike.** The status state machine is
|
||||
ddrescue's but `patch` collapses the distinction — it just tries every
|
||||
non-finished range with the long timeout. Future work can specialize (trim vs.
|
||||
scrape vs. retry with direction reversal) if there's measured benefit.
|
||||
who need pure speed can bypass it with `DiscStream::new(Box::new(drive), …)` —
|
||||
nothing forbids it, and layer 3 still applies there.
|
||||
|
||||
## References
|
||||
|
||||
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
|
||||
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
|
||||
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
|
||||
- Recovery strategy and mapfile: `freemkv-engine/src/recovery/`
|
||||
- In this crate: [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`),
|
||||
[`src/scsi/mod.rs`](../src/scsi/mod.rs) (`SenseFamily`),
|
||||
[`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`),
|
||||
[`src/event.rs`](../src/event.rs) (progress events).
|
||||
|
||||
Reference in New Issue
Block a user