mirror of
https://github.com/freemkv/libfreemkv.git
synced 2026-10-07 18:13:10 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ae2909fe8d | ||
|
|
6a3d19a453 | ||
|
|
a35596d2d1 | ||
|
|
318f654fed | ||
|
|
5e82ef65de | ||
|
|
8534607329 | ||
|
|
97e1a4cad3 | ||
|
|
3f6bf33de5 | ||
|
|
359301e8fc | ||
|
|
bfa527162a | ||
|
|
45defd47f3 | ||
|
|
fe8933dad2 |
+116
@@ -1,5 +1,121 @@
|
||||
# Changelog
|
||||
|
||||
## 0.17.10 (2026-05-09)
|
||||
|
||||
### Bounded-cache writeback for big sequential writes
|
||||
|
||||
Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
|
||||
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
|
||||
settings. Cause: the kernel's `vm.dirty_ratio` (~20 % of RAM) lets
|
||||
hundreds of MB of dirty pages accumulate, then bursts a flush at
|
||||
99 % disk utilisation that blocks app writes for ~1 s. Confirmed
|
||||
empirically on the BU40N test bed: dirty pages grew 112 → 563 MB
|
||||
between bursts; lowering `vm.dirty_bytes` to 64 MB at the host
|
||||
sysctl level eliminated the dips.
|
||||
|
||||
This release ships the equivalent inside libfreemkv so users don't
|
||||
need to tune the host kernel:
|
||||
|
||||
- New `crate::io::Writer` — drop-in `File` wrapper implementing
|
||||
`Write` + `Seek`. Wraps a per-platform `WritebackPipeline` that on
|
||||
Linux schedules `sync_file_range(WRITE)` + lagging
|
||||
`sync_file_range(WAIT_AFTER)` + `posix_fadvise(DONTNEED)` calls in
|
||||
32 MB chunks, keeping dirty cache bounded at ~64 MB. macOS and
|
||||
Windows ship a no-op stub — their default cache policies don't
|
||||
exhibit the same pathology for our access pattern.
|
||||
- Disc::sweep wraps its output `File` in `Writer`. No changes to the
|
||||
loop body — `Writer` forwards `seek`/`write_all` to `File` and
|
||||
drives the pipeline transparently.
|
||||
- Module is purpose-built for any large sequential output (sweep,
|
||||
patch, future mux) — they can adopt `crate::io::Writer` with a
|
||||
one-line wrapper and inherit the same behaviour.
|
||||
|
||||
## 0.17.7 (2026-05-08)
|
||||
|
||||
### Sync release — no functional libfreemkv changes
|
||||
|
||||
Version bump to keep the four freemkv crates at unified versioning.
|
||||
autorip shipped UI smoothness + audit fixes today (v0.17.6, v0.17.7);
|
||||
libfreemkv carries no code changes, but is bumped + republished to
|
||||
crates.io so downstream `libfreemkv = "0.17"` consumers stay aligned
|
||||
on the latest patch version. See autorip CHANGELOG for the
|
||||
user-visible work in this release window.
|
||||
|
||||
## 0.17.5 (2026-05-08)
|
||||
|
||||
### Pass N recovery — kernel block-device fallback + per-range fixes
|
||||
|
||||
Live testing on direct-SATA BU40N + AACS-encrypted UHD disc revealed that
|
||||
the v0.17.3 single-shot SCSI READ path matched 0/22 of the small bad-sector
|
||||
LBAs that `dd if=/dev/sr0` recovers on the same drive. This release closes
|
||||
that gap and fixes several adjacent bugs that were silently capping
|
||||
recovery.
|
||||
|
||||
- **`/dev/sr0` pread fallback in `Drive::read` (Linux)**: when a SCSI READ
|
||||
via `/dev/sg*` returns Err, fall back to `posix_fadvise(POSIX_FADV_DONTNEED)
|
||||
+ pread()` against the corresponding block device. The kernel `sr_mod`
|
||||
driver runs ~5 internal retries per command without the per-attempt
|
||||
error-escalation overhead that userspace SG_IO retries pay, which is the
|
||||
source of dd's recovery advantage. End-to-end byte-verification confirms
|
||||
the fallback path returns real disc data (md5-equivalent to fresh dd from
|
||||
the same drive session). The block fd is opened in `Drive::open` by
|
||||
resolving `/sys/class/scsi_generic/sgN/device/block`; best-effort, with no
|
||||
fallback if open fails.
|
||||
|
||||
- **`Disc::patch` per-range watchdog fix**: when a range hit `MAX_RANGE_SECS`,
|
||||
the old code did `wedged_exit = true; break 'outer;` — a single slow range
|
||||
killed the entire patch. Now `break;` (skip this range, advance the outer
|
||||
for loop). Pre-fix, patch was dying after 4 sectors of the first slow
|
||||
range and never reaching the other 46.
|
||||
|
||||
- **Per-sector range budget**: replaced flat `MAX_RANGE_SECS = 180` with
|
||||
`range_budget_secs = (range_sectors × SECONDS_PER_SECTOR).min(RANGE_BUDGET_CAP_SECS)`.
|
||||
Tiny ranges exit fast (1-sector range = 25 s); medium ranges get
|
||||
proportional time (51-sector range ≈ 1275 s); large ranges still bounded
|
||||
by the 1800 s cap so they cannot monopolise pass 1.
|
||||
|
||||
- **`consecutive_failures` resets per range**: the wedge-exit detector
|
||||
(`>= 50 consecutive failures`) is for "stuck on the same range"; pre-fix
|
||||
the counter persisted across ranges, so 50 small post-bisection ranges
|
||||
with one failure each falsely tripped the wedge mid-pass. Reset at every
|
||||
range boundary.
|
||||
|
||||
- **Reverted inline 5× retry in patch**: a brief experiment that was
|
||||
measurably harmful — each "2 s" SCSI timeout paid ~1.5 s kernel SCSI
|
||||
mid-layer error-escalation overhead, so 5× retry took ~17 s per LBA and
|
||||
triggered the per-range watchdog in 4 sectors. Restored
|
||||
`READ_RECOVERY_TIMEOUT_MS = 60_000` (the v0.17.3 baseline; the kernel-
|
||||
auto-retry pattern is now provided by the `/dev/sr0` fallback above).
|
||||
|
||||
### Empirical results (Dune Part Two UHD, BU40N direct SATA)
|
||||
|
||||
- Pass 1: **94.6 MB recovered** (28% of formerly-bad data, 33 sr0 fallback
|
||||
saves), **11 s of main-title content** restored. Patch completed all 47
|
||||
retryable ranges naturally (was wedging at range 1 of 47 in v0.17.3).
|
||||
- Pass 2 cumulative: 95.2 MB (+0.6 MB; diminishing returns curve).
|
||||
- Remaining ~233 MB on the test disc appears physically unrecoverable on
|
||||
this hardware (kernel auto-retry can't decode it either).
|
||||
|
||||
### Behavioural notes
|
||||
|
||||
- The `/dev/sr0` fallback is Linux only; macOS and Windows fall back to
|
||||
the existing single-shot SCSI behaviour. The fallback is gated to
|
||||
`recovery=true` reads (only fires from the patch path, not the sweep
|
||||
path) to avoid page-cache pressure during multi-GB sequential ripping.
|
||||
|
||||
## 0.17.0 (2026-05-04)
|
||||
|
||||
### Code quality: unwrap safety, clippy compliance, test coverage
|
||||
|
||||
- **Sweep() hot path**: Replaced `.err().unwrap()` in `disc/mod.rs:1553` with explicit pattern matching (`match read_result { Ok(_) => unreachable!(), Err(e) => e }`). The original unwrap was logically safe (in the else branch after `is_ok()` check) but pattern matching makes the invariant explicit and avoids panic risk if logic changes.
|
||||
- **cargo clippy --lib**: Clean build with `-D warnings` across all targets. No lint failures.
|
||||
- **cargo test --lib**: All 256 tests pass (0 failed, 1 ignored). Integration tests for sweep/patch/dev-null validate the multi-pass recovery pipeline end-to-end.
|
||||
|
||||
### Patch pass algorithm fix
|
||||
|
||||
- `patch_internal` excludes Unreadable sectors from work list (only retries NonTrimmed/NonScraped/Unscraped). Previously attempted to retry already-failed sectors, wasting time and masking real progress.
|
||||
- Exposes `bytes_bad_in_title` in patch results for accurate UI reporting of unrecoverable data vs. recovered data.
|
||||
|
||||
## 0.16.1 (2026-04-30)
|
||||
|
||||
### Unified progress display for sweep and patch
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "libfreemkv"
|
||||
version = "0.16.2"
|
||||
version = "0.17.10"
|
||||
edition = "2024"
|
||||
rust-version = "1.86"
|
||||
license = "AGPL-3.0-only"
|
||||
|
||||
@@ -18,7 +18,7 @@ Part of the [freemkv](https://github.com/freemkv) project.
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
libfreemkv = "0.13"
|
||||
libfreemkv = "0.17"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
+86
-2
@@ -91,6 +91,13 @@ pub struct MapStats {
|
||||
/// retry" UI bucket; `bytes_pending` over-counts because it folds
|
||||
/// in `bytes_nontried`.
|
||||
pub bytes_retryable: u64,
|
||||
/// Number of unreadable ranges (for UI display). Computed from
|
||||
/// `ranges_with(&[Unreadable])`.
|
||||
pub num_bad_ranges: u32,
|
||||
/// Largest gap among unreadable ranges in milliseconds. Computed as
|
||||
/// largest range size / bytes_per_sec * 1000. Set by caller (autorip)
|
||||
/// since bytes_per_sec is application-specific.
|
||||
pub main_lost_ms: f64,
|
||||
}
|
||||
|
||||
/// Write-through mapfile. Every `record()` persists to disk immediately
|
||||
@@ -100,6 +107,9 @@ pub struct Mapfile {
|
||||
entries: Vec<MapEntry>,
|
||||
total_size: u64,
|
||||
version: String,
|
||||
/// Incrementally maintained stats — updated on every `record()` call
|
||||
/// so `stats()` is O(1) instead of O(n).
|
||||
stats: MapStats,
|
||||
}
|
||||
|
||||
impl Mapfile {
|
||||
@@ -116,6 +126,12 @@ impl Mapfile {
|
||||
}],
|
||||
total_size,
|
||||
version: version.to_string(),
|
||||
stats: MapStats {
|
||||
bytes_total: total_size,
|
||||
bytes_pending: total_size,
|
||||
bytes_nontried: total_size,
|
||||
..Default::default()
|
||||
},
|
||||
};
|
||||
mf.write_to_disk()?;
|
||||
Ok(mf)
|
||||
@@ -177,11 +193,13 @@ impl Mapfile {
|
||||
}
|
||||
entries.sort_by_key(|e| e.pos);
|
||||
let total_size = entries.last().map(|e| e.pos + e.size).unwrap_or(0);
|
||||
let stats = Self::compute_stats(&entries, total_size);
|
||||
Ok(Self {
|
||||
path: path.to_path_buf(),
|
||||
entries,
|
||||
total_size,
|
||||
version,
|
||||
stats,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -244,6 +262,11 @@ impl Mapfile {
|
||||
merged.push(e);
|
||||
}
|
||||
|
||||
// Recompute stats from merged entries. record() is already O(n) due to
|
||||
// drain-and-rebuild, so this is a constant-factor overhead. The critical
|
||||
// win is that stats() is now O(1) — called millions of times in the hot
|
||||
// path during sweep/patch, it just returns the cached value.
|
||||
self.stats = Self::compute_stats(&merged, self.total_size);
|
||||
self.entries = merged;
|
||||
self.write_to_disk()?;
|
||||
Ok(())
|
||||
@@ -283,11 +306,15 @@ impl Mapfile {
|
||||
}
|
||||
|
||||
pub fn stats(&self) -> MapStats {
|
||||
self.stats
|
||||
}
|
||||
|
||||
fn compute_stats(entries: &[MapEntry], total_size: u64) -> MapStats {
|
||||
let mut s = MapStats {
|
||||
bytes_total: self.total_size,
|
||||
bytes_total: total_size,
|
||||
..Default::default()
|
||||
};
|
||||
for e in &self.entries {
|
||||
for e in entries {
|
||||
match e.status {
|
||||
SectorStatus::Finished => s.bytes_good += e.size,
|
||||
SectorStatus::Unreadable => s.bytes_unreadable += e.size,
|
||||
@@ -467,4 +494,61 @@ mod tests {
|
||||
assert_eq!(bad, vec![(100, 50), (300, 50)]);
|
||||
let _ = std::fs::remove_file(&p);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stats_consistent_after_overlapping_records() {
|
||||
let p = tmpfile("stats_consistent_after_overlapping");
|
||||
let _ = std::fs::remove_file(&p);
|
||||
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
|
||||
// Record some finished, some unreadable, some nontrimmed
|
||||
mf.record(0, 300, SectorStatus::Finished).unwrap();
|
||||
mf.record(300, 200, SectorStatus::NonTrimmed).unwrap();
|
||||
mf.record(500, 100, SectorStatus::Unreadable).unwrap();
|
||||
mf.record(600, 400, SectorStatus::Finished).unwrap();
|
||||
|
||||
// Final entries: [0..300 Finished, 300..500 NonTrimmed, 500..600 Unreadable, 600..1000 Finished]
|
||||
let s = mf.stats();
|
||||
assert_eq!(s.bytes_good, 700); // 300 + 400
|
||||
assert_eq!(s.bytes_unreadable, 100); // 100
|
||||
assert_eq!(s.bytes_pending, 200); // NonTrimmed only (NonTried=0)
|
||||
assert_eq!(s.bytes_nontried, 0);
|
||||
assert_eq!(s.bytes_retryable, 200); // NonTrimmed
|
||||
assert_eq!(s.bytes_total, 1000);
|
||||
|
||||
// Overwrite a NonTrimmed range with Finished
|
||||
mf.record(300, 100, SectorStatus::Finished).unwrap();
|
||||
// Entries: [0..400 Finished, 400..500 NonTrimmed, 500..600 Unreadable, 600..1000 Finished]
|
||||
let s2 = mf.stats();
|
||||
assert_eq!(s2.bytes_good, 800); // 400 + 400
|
||||
assert_eq!(s2.bytes_unreadable, 100);
|
||||
assert_eq!(s2.bytes_pending, 100); // NonTrimmed only
|
||||
assert_eq!(s2.bytes_retryable, 100);
|
||||
|
||||
let _ = std::fs::remove_file(&p);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stats_consistent_after_split_record() {
|
||||
let p = tmpfile("stats_consistent_after_split");
|
||||
let _ = std::fs::remove_file(&p);
|
||||
let mut mf = Mapfile::create(&p, 1000, "test").unwrap();
|
||||
// Mark middle as NonTrimmed
|
||||
mf.record(200, 400, SectorStatus::NonTrimmed).unwrap();
|
||||
// Entries: [0..200 NonTried, 200..600 NonTrimmed, 600..1000 NonTried]
|
||||
let s = mf.stats();
|
||||
assert_eq!(s.bytes_pending, 1000); // NonTried(600) + NonTrimmed(400)
|
||||
assert_eq!(s.bytes_retryable, 400); // NonTrimmed only
|
||||
assert_eq!(s.bytes_nontried, 600); // 200 + 400
|
||||
|
||||
// Overwrite the NonTrimmed with Finished (splitting the remaining NonTried)
|
||||
mf.record(200, 400, SectorStatus::Finished).unwrap();
|
||||
// Entries: [0..200 NonTried, 200..600 Finished, 600..1000 NonTried]
|
||||
let s2 = mf.stats();
|
||||
assert_eq!(s2.bytes_good, 400);
|
||||
assert_eq!(s2.bytes_pending, 600); // NonTried(200 + 400)
|
||||
assert_eq!(s2.bytes_nontried, 600);
|
||||
assert_eq!(s2.bytes_retryable, 0);
|
||||
|
||||
let _ = std::fs::remove_file(&p);
|
||||
}
|
||||
}
|
||||
|
||||
+1035
-265
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,374 @@
|
||||
//! Single source of truth for what to do when a sector read fails.
|
||||
//!
|
||||
//! Both Pass 1 (`Disc::sweep`) and Pass 2-N (`Disc::patch`) call into
|
||||
//! `handle_read_error` after every failed `read_sectors`. The handler
|
||||
//! classifies the error, updates the in-flight context (counters,
|
||||
//! damage window, retry budgets), and returns a `ReadAction` the caller
|
||||
//! dispatches on. Every read goes through the same gate — no path can
|
||||
//! silently skip pause/skip/jump/abort logic.
|
||||
//!
|
||||
//! Adding a new error class = add one arm in `handle_read_error`.
|
||||
//! Adding new logging on errors = one place.
|
||||
|
||||
use crate::error::Error;
|
||||
use crate::scsi;
|
||||
|
||||
/// In-flight bookkeeping a read loop must keep across iterations. The
|
||||
/// handler reads and mutates this. Caller owns the storage.
|
||||
pub struct ReadCtx {
|
||||
/// Number of sectors per read attempt. The handler uses this to
|
||||
/// decide whether to bisect (only worthwhile when batch > 1).
|
||||
pub batch: u16,
|
||||
/// Successful reads since the last failure. Resets to 0 on failure.
|
||||
/// Used by callers to drive damage-zone exit / speed restoration.
|
||||
pub consecutive_good: u64,
|
||||
/// Failed reads since the last success. Resets to 0 on success.
|
||||
/// Drives long-pause escalation on persistent failure.
|
||||
pub consecutive_failures: u64,
|
||||
/// Sliding window of recent read outcomes (true=ok, false=fail).
|
||||
/// Capped at `damage_window_max`. Drives damage-jump decisions.
|
||||
pub damage_window: Vec<bool>,
|
||||
pub damage_window_max: usize,
|
||||
pub damage_threshold_pct: usize,
|
||||
/// Multiplier applied to damage-jump distance. Doubles each jump,
|
||||
/// resets to 1 after `damage_window_max` consecutive good reads.
|
||||
pub jump_multiplier: u64,
|
||||
/// NOT_READY retries used so far for the current LBA. Reset to 0
|
||||
/// on any non-NOT_READY response.
|
||||
pub not_ready_retries: u32,
|
||||
/// Bridge-degradation cooldowns used so far.
|
||||
pub bridge_degradation_count: u32,
|
||||
/// Whether we're currently inside a damage-jump bisect attempt.
|
||||
/// Caller sets this true when entering single-sector mode for a
|
||||
/// failed batch, so the handler doesn't recursively request another
|
||||
/// bisect on the inner-sector failures.
|
||||
pub bisecting: bool,
|
||||
}
|
||||
|
||||
impl ReadCtx {
|
||||
/// Initial context for a Pass 1 sweep with the documented constants.
|
||||
pub fn for_sweep(batch: u16) -> Self {
|
||||
Self {
|
||||
batch,
|
||||
consecutive_good: 0,
|
||||
consecutive_failures: 0,
|
||||
damage_window: Vec::with_capacity(16),
|
||||
damage_window_max: 16,
|
||||
damage_threshold_pct: 12,
|
||||
jump_multiplier: 1,
|
||||
not_ready_retries: 0,
|
||||
bridge_degradation_count: 0,
|
||||
bisecting: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Initial context for a Pass 2-N patch.
|
||||
pub fn for_patch(batch: u16) -> Self {
|
||||
Self {
|
||||
batch,
|
||||
consecutive_good: 0,
|
||||
consecutive_failures: 0,
|
||||
damage_window: Vec::with_capacity(16),
|
||||
damage_window_max: 16,
|
||||
damage_threshold_pct: 12,
|
||||
jump_multiplier: 1,
|
||||
not_ready_retries: 0,
|
||||
bridge_degradation_count: 0,
|
||||
bisecting: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Caller calls this after every successful read.
|
||||
pub fn on_success(&mut self) {
|
||||
self.consecutive_good += 1;
|
||||
self.consecutive_failures = 0;
|
||||
self.not_ready_retries = 0;
|
||||
self.damage_window.push(true);
|
||||
if self.damage_window.len() > self.damage_window_max {
|
||||
self.damage_window.remove(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What the caller should do after a read failure. The caller owns the
|
||||
/// I/O side-effects (sleep, write zeros, advance pos) — the handler
|
||||
/// only decides which side-effects.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ReadAction {
|
||||
/// Pause `pause_secs` then retry the same LBA / batch. Used for
|
||||
/// transient conditions (NOT_READY, bridge degradation) that the
|
||||
/// drive may recover from on its own.
|
||||
Retry { pause_secs: u64 },
|
||||
/// Re-issue the failed batch as `batch` single-sector reads. Each
|
||||
/// inner read is itself dispatched through `handle_read_error` with
|
||||
/// `bisecting = true` so it cannot recurse.
|
||||
Bisect,
|
||||
/// Mark the failed range NonTrimmed (zero-fill, retry in Pass N+),
|
||||
/// then pause `pause_secs` before resuming the next LBA.
|
||||
SkipBlock { pause_secs: u64 },
|
||||
/// Mark the failed range NonTrimmed AND advance position by
|
||||
/// `sectors` (zero-filling the gap as NonTrimmed). Then pause
|
||||
/// `pause_secs`. Used when the damage-window threshold is crossed.
|
||||
JumpAhead { sectors: u64, pause_secs: u64 },
|
||||
/// Unrecoverable at this layer. Caller propagates `Err` up to the
|
||||
/// outer pass loop / autorip, which can attempt USB re-enumeration,
|
||||
/// drop session, etc.
|
||||
AbortPass,
|
||||
}
|
||||
|
||||
// Pause budget constants. Tuned from 2026-05-07 BU40N traces showing
|
||||
// bridge wedges 524 ms after a 5.4-second internal ECC retry. The
|
||||
// post-failure pauses give the drive — and the bridge — time to settle.
|
||||
const POST_FAILURE_PAUSE_SECS: u64 = 1;
|
||||
const CONSECUTIVE_FAIL_LONG_PAUSE_SECS: u64 = 5;
|
||||
const CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD: u64 = 10;
|
||||
const POST_JUMP_EXTRA_PAUSE_SECS: u64 = 2;
|
||||
const NOT_READY_PAUSE_SECS: u64 = 3;
|
||||
const NOT_READY_MAX_RETRIES: u32 = 3;
|
||||
const BRIDGE_DEGRADATION_PAUSE_SECS: u64 = 15;
|
||||
const BRIDGE_DEGRADATION_MAX_RETRIES: u32 = 5;
|
||||
|
||||
/// THE single error-handling entry point. Updates `ctx`, returns the
|
||||
/// action the caller must apply.
|
||||
///
|
||||
/// New error class = add a new arm here. New logging on errors = add
|
||||
/// it once at the top. New retry policy = adjust the constants. No
|
||||
/// other read site needs to change.
|
||||
pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
||||
ctx.consecutive_failures += 1;
|
||||
ctx.consecutive_good = 0;
|
||||
|
||||
tracing::warn!(
|
||||
target: "freemkv::disc",
|
||||
phase = "read_error",
|
||||
consecutive_failures = ctx.consecutive_failures,
|
||||
batch = ctx.batch,
|
||||
bisecting = ctx.bisecting,
|
||||
sense_key = err.scsi_sense().map(|s| s.sense_key),
|
||||
asc = err.scsi_sense().map(|s| s.asc),
|
||||
ascq = err.scsi_sense().map(|s| s.ascq),
|
||||
error = %err,
|
||||
"read failed; classifying"
|
||||
);
|
||||
|
||||
// 1. Transport failure: bridge crash / USB disconnect. The outer
|
||||
// pass loop knows how to handle this (rediscover sg path,
|
||||
// re-open drive). Inline single-sector retry here was tried in
|
||||
// pre-v0.17.0 builds and observed to make wedges worse.
|
||||
if err.is_scsi_transport_failure() {
|
||||
return ReadAction::AbortPass;
|
||||
}
|
||||
|
||||
// 2. Bridge degradation: NOT_READY with the well-known signature
|
||||
// (sense_key=2, ASC=0x04, ASCQ=0x3E). Drive's bridge is in a
|
||||
// semi-stuck state but typically recovers after a long cooldown.
|
||||
// If we've exhausted our retry budget, fall through to the
|
||||
// marginal/skip path below.
|
||||
if err.is_bridge_degradation() && ctx.bridge_degradation_count < BRIDGE_DEGRADATION_MAX_RETRIES
|
||||
{
|
||||
ctx.bridge_degradation_count += 1;
|
||||
return ReadAction::Retry {
|
||||
pause_secs: BRIDGE_DEGRADATION_PAUSE_SECS,
|
||||
};
|
||||
}
|
||||
|
||||
let sense_key = err.scsi_sense().map(|s| s.sense_key).unwrap_or(0);
|
||||
|
||||
// 3. Generic NOT_READY (other ASC codes): drive's mechanical
|
||||
// pickup may be moving. Pause and retry briefly.
|
||||
if sense_key == scsi::SENSE_KEY_NOT_READY && ctx.not_ready_retries < NOT_READY_MAX_RETRIES {
|
||||
ctx.not_ready_retries += 1;
|
||||
return ReadAction::Retry {
|
||||
pause_secs: NOT_READY_PAUSE_SECS,
|
||||
};
|
||||
}
|
||||
if sense_key != scsi::SENSE_KEY_NOT_READY {
|
||||
ctx.not_ready_retries = 0;
|
||||
}
|
||||
|
||||
// 4. Hardware / illegal request: the drive said "no, won't do it".
|
||||
// Retrying won't change the answer. Surface to the outer layer
|
||||
// so it can eject + prompt user.
|
||||
if sense_key == scsi::SENSE_KEY_HARDWARE_ERROR || sense_key == scsi::SENSE_KEY_ILLEGAL_REQUEST {
|
||||
return ReadAction::AbortPass;
|
||||
}
|
||||
|
||||
// 5. Marginal media (MEDIUM_ERROR / ABORTED_COMMAND) on a multi-
|
||||
// sector batch: the drive can often read the same sectors
|
||||
// individually. Bisect into single-sector reads (gentler on the
|
||||
// bridge too — shorter SCSI transactions). Avoid recursive
|
||||
// bisect.
|
||||
let is_marginal = matches!(
|
||||
sense_key,
|
||||
scsi::SENSE_KEY_MEDIUM_ERROR | scsi::SENSE_KEY_ABORTED_COMMAND
|
||||
);
|
||||
if is_marginal && ctx.batch > 1 && !ctx.bisecting {
|
||||
return ReadAction::Bisect;
|
||||
}
|
||||
|
||||
// 6. Single-sector failure or unbisectable error — record in
|
||||
// damage window, decide between skip-in-place vs damage-jump.
|
||||
//
|
||||
// SKIP damage-window updates while bisecting: the window
|
||||
// represents per-batch outcomes, not per-sector. Updating it
|
||||
// inside a bisect inner loop (potentially 32+ sector failures
|
||||
// per batch) would over-weight the window and cause runaway
|
||||
// JumpAhead distance via excessive multiplier doublings.
|
||||
if !ctx.bisecting {
|
||||
ctx.damage_window.push(false);
|
||||
if ctx.damage_window.len() > ctx.damage_window_max {
|
||||
ctx.damage_window.remove(0);
|
||||
}
|
||||
}
|
||||
|
||||
let bad_count = ctx.damage_window.iter().filter(|&&b| !b).count();
|
||||
let bad_pct = if ctx.damage_window.is_empty() {
|
||||
0
|
||||
} else {
|
||||
bad_count * 100 / ctx.damage_window.len()
|
||||
};
|
||||
|
||||
let pause_secs = if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
||||
CONSECUTIVE_FAIL_LONG_PAUSE_SECS
|
||||
} else {
|
||||
POST_FAILURE_PAUSE_SECS
|
||||
};
|
||||
|
||||
// 7. Damage-jump: too many failures in window → skip ahead by an
|
||||
// escalating gap. Multiplier capped so we can't accidentally
|
||||
// skip the entire rest of the disc (observed 2026-05-07: a
|
||||
// saturated multiplier produced a 56 GB jump). Saturating
|
||||
// arithmetic on the sector calc as defence in depth.
|
||||
const MAX_JUMP_MULTIPLIER: u64 = 64; // 64 × 256 × batch sectors
|
||||
if ctx.damage_window.len() >= ctx.damage_window_max && bad_pct >= ctx.damage_threshold_pct {
|
||||
let mult = ctx.jump_multiplier.min(MAX_JUMP_MULTIPLIER);
|
||||
let sectors = 256u64.saturating_mul(ctx.batch as u64).saturating_mul(mult);
|
||||
ctx.jump_multiplier = (ctx.jump_multiplier.saturating_mul(2)).min(MAX_JUMP_MULTIPLIER);
|
||||
return ReadAction::JumpAhead {
|
||||
sectors,
|
||||
pause_secs: pause_secs + POST_JUMP_EXTRA_PAUSE_SECS,
|
||||
};
|
||||
}
|
||||
|
||||
// 8. Default: zero-fill the failed batch as NonTrimmed and pause
|
||||
// before the next read.
|
||||
ReadAction::SkipBlock { pause_secs }
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::error::Error;
|
||||
use crate::scsi::ScsiSense;
|
||||
|
||||
fn medium_err() -> Error {
|
||||
Error::DiscRead {
|
||||
sector: 100,
|
||||
status: Some(2),
|
||||
sense: Some(ScsiSense {
|
||||
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
|
||||
asc: 0x11,
|
||||
ascq: 0x05,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
fn hardware_err() -> Error {
|
||||
Error::DiscRead {
|
||||
sector: 100,
|
||||
status: Some(2),
|
||||
sense: Some(ScsiSense {
|
||||
sense_key: scsi::SENSE_KEY_HARDWARE_ERROR,
|
||||
asc: 0x44,
|
||||
ascq: 0x00,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn medium_error_with_batch_gt_1_bisects() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
assert_eq!(action, ReadAction::Bisect);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn medium_error_with_batch_1_skips() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::SkipBlock { pause_secs } => assert!(pause_secs >= 1),
|
||||
other => panic!("expected SkipBlock, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn medium_error_while_bisecting_does_not_recurse() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
ctx.bisecting = true;
|
||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match action {
|
||||
ReadAction::SkipBlock { .. } => {}
|
||||
other => panic!("expected SkipBlock, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hardware_error_aborts() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
let action = handle_read_error(&hardware_err(), &mut ctx);
|
||||
assert_eq!(action, ReadAction::AbortPass);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_failure_streak_extends_pause() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
for _ in 0..15 {
|
||||
handle_read_error(&medium_err(), &mut ctx);
|
||||
}
|
||||
// After many consecutive failures we should be in the long-pause regime
|
||||
let final_action = handle_read_error(&medium_err(), &mut ctx);
|
||||
match final_action {
|
||||
ReadAction::SkipBlock { pause_secs } => {
|
||||
assert!(pause_secs >= CONSECUTIVE_FAIL_LONG_PAUSE_SECS);
|
||||
}
|
||||
ReadAction::JumpAhead { pause_secs, .. } => {
|
||||
assert!(pause_secs >= CONSECUTIVE_FAIL_LONG_PAUSE_SECS);
|
||||
}
|
||||
other => panic!("expected long-pause action, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn damage_window_fills_then_jumps() {
|
||||
let mut ctx = ReadCtx::for_sweep(1);
|
||||
ctx.damage_window_max = 4;
|
||||
ctx.damage_threshold_pct = 50;
|
||||
let mut saw_jump = false;
|
||||
for _ in 0..6 {
|
||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
||||
if matches!(a, ReadAction::JumpAhead { .. }) {
|
||||
saw_jump = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
saw_jump,
|
||||
"expected at least one JumpAhead in 6 failures with 50% threshold"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn on_success_resets_failure_counters_and_pushes_window() {
|
||||
let mut ctx = ReadCtx::for_sweep(32);
|
||||
for _ in 0..3 {
|
||||
handle_read_error(&medium_err(), &mut ctx);
|
||||
}
|
||||
assert!(ctx.consecutive_failures > 0);
|
||||
ctx.bisecting = false;
|
||||
ctx.on_success();
|
||||
assert_eq!(ctx.consecutive_good, 1);
|
||||
assert_eq!(ctx.consecutive_failures, 0);
|
||||
assert_eq!(*ctx.damage_window.last().unwrap(), true);
|
||||
}
|
||||
}
|
||||
@@ -69,6 +69,16 @@ pub struct Drive {
|
||||
halt: Arc<AtomicBool>,
|
||||
/// Event handler — fires for read errors and library-level state changes.
|
||||
event_fn: Option<Box<dyn Fn(Event) + Send>>,
|
||||
/// Linux only: raw fd for the corresponding block device (`/dev/sr*`)
|
||||
/// used as a recovery fallback when SCSI READ via `/dev/sg*` returns
|
||||
/// an error. The kernel `sr_mod` driver auto-retries failed reads
|
||||
/// (~5× per command) — historically the reason `dd if=/dev/sr0`
|
||||
/// recovers ~50% of bad sectors that single-shot `SG_IO` READ
|
||||
/// misses on the same drive. `None` when the block device couldn't
|
||||
/// be resolved or opened (no fallback in that case; SCSI read
|
||||
/// errors propagate as before).
|
||||
#[cfg(target_os = "linux")]
|
||||
block_dev_fd: Option<std::os::unix::io::RawFd>,
|
||||
}
|
||||
|
||||
impl Drive {
|
||||
@@ -87,6 +97,9 @@ impl Drive {
|
||||
None => (None, None, None),
|
||||
};
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
let block_dev_fd = open_block_device_for_sg(device);
|
||||
|
||||
Ok(Drive {
|
||||
scsi: transport,
|
||||
driver,
|
||||
@@ -96,6 +109,8 @@ impl Drive {
|
||||
device_path: device.to_string_lossy().to_string(),
|
||||
halt: Arc::new(AtomicBool::new(false)),
|
||||
event_fn: None,
|
||||
#[cfg(target_os = "linux")]
|
||||
block_dev_fd,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -483,6 +498,54 @@ impl Drive {
|
||||
scsi_status = status,
|
||||
"Drive::read checked_exec failed"
|
||||
);
|
||||
|
||||
// /dev/sr0 pread fallback (Linux only). The kernel
|
||||
// sr_mod driver auto-retries failed reads (~5× per
|
||||
// command). Empirically (BU40N + Dune Part 2 UHD,
|
||||
// 2026-05-08) dd via /dev/sr0 recovers ~50% of bad
|
||||
// sectors that a single-shot SG_IO READ misses.
|
||||
#[cfg(target_os = "linux")]
|
||||
if recovery {
|
||||
if let Some(fd) = self.block_dev_fd {
|
||||
let len = count as usize * 2048;
|
||||
if buf.len() >= len {
|
||||
let offset = lba as i64 * 2048;
|
||||
// Drop kernel cache for this region so we get
|
||||
// a fresh device read, not stale page-cache
|
||||
// data from a prior successful neighbour read.
|
||||
let _ = unsafe {
|
||||
libc::posix_fadvise(
|
||||
fd,
|
||||
offset,
|
||||
len as i64,
|
||||
libc::POSIX_FADV_DONTNEED,
|
||||
)
|
||||
};
|
||||
let n = unsafe {
|
||||
libc::pread(fd, buf.as_mut_ptr() as *mut libc::c_void, len, offset)
|
||||
};
|
||||
if n == len as isize {
|
||||
tracing::info!(
|
||||
target: "freemkv::drive",
|
||||
lba,
|
||||
count,
|
||||
bytes = len,
|
||||
"Drive::read recovered via /dev/sr0 pread fallback"
|
||||
);
|
||||
return Ok(len);
|
||||
}
|
||||
tracing::debug!(
|
||||
target: "freemkv::drive",
|
||||
lba,
|
||||
count,
|
||||
pread_ret = n as i64,
|
||||
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
|
||||
"/dev/sr0 pread fallback also failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Err(Error::DiscRead {
|
||||
sector: lba as u64,
|
||||
status: Some(status),
|
||||
@@ -586,6 +649,61 @@ impl Drop for Drive {
|
||||
fn drop(&mut self) {
|
||||
self.cleanup();
|
||||
// SgIoTransport::drop() runs next, calling libc::close(fd)
|
||||
#[cfg(target_os = "linux")]
|
||||
if let Some(fd) = self.block_dev_fd.take() {
|
||||
unsafe { libc::close(fd) };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a `/dev/sg*` path to the corresponding `/dev/sr*` block
|
||||
/// device by walking sysfs, then open it for read (no `O_DIRECT` —
|
||||
/// `posix_fadvise(POSIX_FADV_DONTNEED)` flushes the cache before each
|
||||
/// pread, which avoids buffer-alignment requirements while still
|
||||
/// forcing fresh device reads).
|
||||
///
|
||||
/// Returns `None` on any error (sysfs not present, no matching block
|
||||
/// device, open failed). Callers treat that as "no fallback available"
|
||||
/// and propagate the original SCSI READ error.
|
||||
#[cfg(target_os = "linux")]
|
||||
fn open_block_device_for_sg(sg_path: &Path) -> Option<std::os::unix::io::RawFd> {
|
||||
let basename = sg_path.file_name()?.to_str()?;
|
||||
if !basename.starts_with("sg") {
|
||||
return None;
|
||||
}
|
||||
let sysfs_dir = format!("/sys/class/scsi_generic/{}/device/block", basename);
|
||||
let entries = std::fs::read_dir(&sysfs_dir).ok()?;
|
||||
let block_name = entries
|
||||
.flatten()
|
||||
.find_map(|e| e.file_name().into_string().ok())?;
|
||||
let block_path = format!("/dev/{}", block_name);
|
||||
|
||||
let mut bytes = block_path.as_bytes().to_vec();
|
||||
bytes.push(0);
|
||||
let fd = unsafe {
|
||||
libc::open(
|
||||
bytes.as_ptr() as *const libc::c_char,
|
||||
libc::O_RDONLY | libc::O_CLOEXEC,
|
||||
)
|
||||
};
|
||||
if fd < 0 {
|
||||
tracing::debug!(
|
||||
target: "freemkv::drive",
|
||||
sg = basename,
|
||||
block_path,
|
||||
errno = std::io::Error::last_os_error().raw_os_error().unwrap_or(0),
|
||||
"Failed to open block device for fallback; sr0 fallback disabled"
|
||||
);
|
||||
None
|
||||
} else {
|
||||
tracing::info!(
|
||||
target: "freemkv::drive",
|
||||
sg = basename,
|
||||
block_path,
|
||||
fd,
|
||||
"Opened /dev/sr* as recovery fallback for failed SCSI reads"
|
||||
);
|
||||
Some(fd)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
//! File I/O helpers that bound kernel cache pressure on big writes.
|
||||
//!
|
||||
//! `Writer` is a drop-in wrapper around `std::fs::File` for any call
|
||||
//! site that performs large sequential writes (sweep, mux, etc.). It
|
||||
//! implements `Write` and `Seek` so existing code paths can swap
|
||||
//! `File` for `Writer` with no body changes. Internally it drives a
|
||||
//! `WritebackPipeline` that, on Linux, drains dirty pages continuously
|
||||
//! at 32 MB granularity to avoid the kernel's accumulate-then-burst
|
||||
//! flush behaviour. macOS and Windows use a no-op pipeline — their
|
||||
//! default cache policies have not been shown to exhibit the same
|
||||
//! pathology for this access pattern.
|
||||
|
||||
mod writeback;
|
||||
mod writer;
|
||||
|
||||
pub(crate) use writer::Writer;
|
||||
@@ -0,0 +1,17 @@
|
||||
//! Per-platform writeback pipeline. On Linux, drains dirty pages
|
||||
//! continuously at chunk granularity to keep the kernel's writeback
|
||||
//! queue bounded. On macOS and Windows, a no-op stub.
|
||||
//!
|
||||
//! The platform decision lives entirely in this file (the cfg-gated
|
||||
//! `pub use` below). Callers — and `DiskWriter` itself — are
|
||||
//! platform-independent.
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
mod linux;
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
mod noop;
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
pub(super) use linux::WritebackPipeline;
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
pub(super) use noop::WritebackPipeline;
|
||||
@@ -0,0 +1,95 @@
|
||||
//! Linux writeback pipeline using `sync_file_range` + `posix_fadvise`.
|
||||
//!
|
||||
//! Pathology this fixes: the kernel's default `vm.dirty_ratio` (~20 %
|
||||
//! of RAM) lets dirty pages accumulate to hundreds of MB during a
|
||||
//! big sequential write, then bursts a flush at 99 % disk utilisation.
|
||||
//! While the burst runs, app writes block on the writeback queue —
|
||||
//! observed empirically as instantaneous speed dropping from ~15 MB/s
|
||||
//! to ~1 MB/s every ~30 s during a Pass 1 sweep.
|
||||
//!
|
||||
//! Strategy: every `chunk_bytes` of new sequential output, kick async
|
||||
//! writeback (`SYNC_FILE_RANGE_WRITE`) on the just-completed chunk and
|
||||
//! finalise the *previous* chunk via `WAIT_AFTER` + `posix_fadvise
|
||||
//! (DONTNEED)`. By the time we finalise, that previous chunk has had
|
||||
//! a full chunk's worth of work to flush — the wait is near-instant.
|
||||
//! Dirty cache stays bounded at ~2 × `chunk_bytes` and writes drain
|
||||
//! continuously instead of in bursts.
|
||||
|
||||
use std::fs::File;
|
||||
use std::os::unix::io::{AsRawFd, RawFd};
|
||||
|
||||
pub(crate) struct WritebackPipeline {
|
||||
fd: RawFd,
|
||||
chunk_bytes: u64,
|
||||
last_flush_pos: u64,
|
||||
pending: Option<(u64, u64)>,
|
||||
}
|
||||
|
||||
impl WritebackPipeline {
|
||||
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
|
||||
Self {
|
||||
fd: file.as_raw_fd(),
|
||||
chunk_bytes,
|
||||
last_flush_pos: start_pos,
|
||||
pending: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Caller advanced the file position to `pos`. If a chunk boundary
|
||||
/// was crossed, kick async writeback for the just-completed chunk
|
||||
/// and finalise the previous one.
|
||||
pub(crate) fn note_progress(&mut self, pos: u64) {
|
||||
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
|
||||
return;
|
||||
}
|
||||
unsafe {
|
||||
let chunk_off = self.last_flush_pos as i64;
|
||||
let chunk_len = (pos - self.last_flush_pos) as i64;
|
||||
libc::sync_file_range(self.fd, chunk_off, chunk_len, libc::SYNC_FILE_RANGE_WRITE);
|
||||
if let Some((prev_off, prev_len)) = self.pending.take() {
|
||||
libc::sync_file_range(
|
||||
self.fd,
|
||||
prev_off as i64,
|
||||
prev_len as i64,
|
||||
libc::SYNC_FILE_RANGE_WAIT_AFTER,
|
||||
);
|
||||
libc::posix_fadvise(
|
||||
self.fd,
|
||||
prev_off as i64,
|
||||
prev_len as i64,
|
||||
libc::POSIX_FADV_DONTNEED,
|
||||
);
|
||||
}
|
||||
self.pending = Some((chunk_off as u64, chunk_len as u64));
|
||||
}
|
||||
self.last_flush_pos = pos;
|
||||
}
|
||||
|
||||
/// Caller is about to seek away from the current write region.
|
||||
/// Drain any in-flight chunk and reset tracking.
|
||||
pub(crate) fn handle_seek(&mut self, new_pos: u64) {
|
||||
self.finalize();
|
||||
self.last_flush_pos = new_pos;
|
||||
}
|
||||
|
||||
/// Drain any in-flight chunk. Idempotent. Call before `sync_all()`
|
||||
/// or when discarding the pipeline.
|
||||
pub(crate) fn finalize(&mut self) {
|
||||
if let Some((prev_off, prev_len)) = self.pending.take() {
|
||||
unsafe {
|
||||
libc::sync_file_range(
|
||||
self.fd,
|
||||
prev_off as i64,
|
||||
prev_len as i64,
|
||||
libc::SYNC_FILE_RANGE_WAIT_AFTER,
|
||||
);
|
||||
libc::posix_fadvise(
|
||||
self.fd,
|
||||
prev_off as i64,
|
||||
prev_len as i64,
|
||||
libc::POSIX_FADV_DONTNEED,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
//! No-op writeback pipeline for non-Linux targets. macOS and Windows
|
||||
//! page cache policies have not been shown to exhibit the Linux
|
||||
//! accumulate-then-burst flush pathology for our access pattern.
|
||||
//! If that changes, replace this stub with a real implementation
|
||||
//! (e.g. `F_NOCACHE` on macOS, `FILE_FLAG_WRITE_THROUGH` on Windows).
|
||||
|
||||
use std::fs::File;
|
||||
|
||||
pub(crate) struct WritebackPipeline;
|
||||
|
||||
impl WritebackPipeline {
|
||||
pub(crate) fn new(_file: &File, _start_pos: u64, _chunk_bytes: u64) -> Self {
|
||||
Self
|
||||
}
|
||||
pub(crate) fn note_progress(&mut self, _pos: u64) {}
|
||||
pub(crate) fn handle_seek(&mut self, _new_pos: u64) {}
|
||||
pub(crate) fn finalize(&mut self) {}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
//! `Writer` — a drop-in `File` wrapper that keeps the kernel page
|
||||
//! cache bounded during large sequential output.
|
||||
//!
|
||||
//! Implements `Write` and `Seek`, so any call site that uses `File`
|
||||
//! through those traits (sweep, mux, patch) can swap to `Writer`
|
||||
//! without touching the body of the loop. The wrapper tracks the
|
||||
//! current file position and forwards each write to a
|
||||
//! [`super::writeback::WritebackPipeline`], which on Linux schedules
|
||||
//! incremental `sync_file_range` + `posix_fadvise(DONTNEED)` calls
|
||||
//! to drain dirty pages continuously instead of letting the kernel
|
||||
//! burst-flush hundreds of MB at a time.
|
||||
//!
|
||||
//! See `super::writeback::linux` for the pathology and the strategy.
|
||||
|
||||
use std::fs::{File, Metadata};
|
||||
use std::io::{self, Seek, SeekFrom, Write};
|
||||
|
||||
use super::writeback::WritebackPipeline;
|
||||
|
||||
const CHUNK_BYTES: u64 = 32 * 1024 * 1024;
|
||||
|
||||
pub(crate) struct Writer {
|
||||
file: File,
|
||||
pipeline: WritebackPipeline,
|
||||
pos: u64,
|
||||
}
|
||||
|
||||
impl Writer {
|
||||
/// Wrap an open `File`. The current OS file position is queried
|
||||
/// once so the pipeline starts tracking from wherever the file
|
||||
/// already is (typically 0 for fresh files; non-zero for resumed
|
||||
/// or appended files).
|
||||
pub(crate) fn new(mut file: File) -> io::Result<Self> {
|
||||
let pos = file.stream_position()?;
|
||||
let pipeline = WritebackPipeline::new(&file, pos, CHUNK_BYTES);
|
||||
Ok(Self {
|
||||
file,
|
||||
pipeline,
|
||||
pos,
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn metadata(&self) -> io::Result<Metadata> {
|
||||
self.file.metadata()
|
||||
}
|
||||
|
||||
/// Drain in-flight writeback then issue a full fsync. Use this in
|
||||
/// place of `File::sync_all`.
|
||||
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
|
||||
self.pipeline.finalize();
|
||||
self.file.sync_all()
|
||||
}
|
||||
}
|
||||
|
||||
impl Write for Writer {
|
||||
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
|
||||
let n = self.file.write(buf)?;
|
||||
self.pos += n as u64;
|
||||
self.pipeline.note_progress(self.pos);
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
|
||||
self.file.write_all(buf)?;
|
||||
self.pos += buf.len() as u64;
|
||||
self.pipeline.note_progress(self.pos);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> io::Result<()> {
|
||||
self.file.flush()
|
||||
}
|
||||
}
|
||||
|
||||
impl Seek for Writer {
|
||||
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
|
||||
let p = self.file.seek(from)?;
|
||||
// Only treat seeks that actually move the position as
|
||||
// boundaries — sweep does a redundant `seek(Current(pos))`
|
||||
// before every write, and we don't want that to drain the
|
||||
// pipeline on every iteration.
|
||||
if p != self.pos {
|
||||
self.pipeline.handle_seek(p);
|
||||
self.pos = p;
|
||||
}
|
||||
Ok(p)
|
||||
}
|
||||
}
|
||||
@@ -81,6 +81,7 @@ pub mod error;
|
||||
pub mod event;
|
||||
pub(crate) mod identity;
|
||||
pub(crate) mod ifo;
|
||||
pub(crate) mod io;
|
||||
pub mod keydb;
|
||||
pub(crate) mod labels;
|
||||
pub(crate) mod mpls;
|
||||
@@ -95,6 +96,9 @@ pub(crate) mod speed;
|
||||
pub(crate) mod udf;
|
||||
pub mod verify;
|
||||
|
||||
// Re-export verify types at the crate root for ergonomic imports.
|
||||
pub use verify::{SectorRange, SectorStatus, VerifyResult, verify_title};
|
||||
|
||||
// ─── Drive lifecycle ────────────────────────────────────────────────────────
|
||||
//
|
||||
// `Drive::open(path)` → `wait_ready()` → `init()` → `Disc::scan()`. `Drive`
|
||||
|
||||
+54
-5
@@ -1,7 +1,7 @@
|
||||
//! Pipeline-progress reporting for the rip pipeline.
|
||||
//!
|
||||
//! v0.13.16 architecture rule: ONE progress signal type. Every long-running
|
||||
//! pipeline operation (`Disc::copy`, `Disc::patch`, mux) emits the same
|
||||
//! pipeline operation (`Disc::copy`, `Disc::patch`, `verify_title`) emits the same
|
||||
//! `PassProgress` shape via the `Progress` trait. Consumers (autorip) compute
|
||||
//! a single `PipelineStats` derived view and never reach into per-pass
|
||||
//! internals.
|
||||
@@ -28,6 +28,8 @@ pub enum PassKind {
|
||||
/// Demux ISO → output (MKV / M2TS / network). Single phase that runs
|
||||
/// after all rip passes complete.
|
||||
Mux,
|
||||
/// Sector verification — reads every sector and classifies health.
|
||||
Verify,
|
||||
}
|
||||
|
||||
/// One progress sample from a pipeline phase.
|
||||
@@ -36,6 +38,13 @@ pub enum PassKind {
|
||||
/// regardless of which kind of pass is running. `bytes_good_total` is the
|
||||
/// cumulative count of confirmed-clean bytes across the whole rip; useful
|
||||
/// for the "data recovered" stat the user sees.
|
||||
///
|
||||
/// For `PassKind::Verify`, the fields map as follows:
|
||||
/// - `work_done` = sectors read so far
|
||||
/// - `work_total` = total sectors in title
|
||||
/// - `bytes_good_total` = good + slow + recovered sectors × 2048
|
||||
/// - `bytes_unreadable_total` = bad sectors × 2048
|
||||
/// - `bytes_pending_total` = 0 (verify processes sequentially, nothing pending)
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct PassProgress {
|
||||
pub kind: PassKind,
|
||||
@@ -57,20 +66,60 @@ pub struct PassProgress {
|
||||
pub main_title_size_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
impl PassProgress {
|
||||
/// Percentage of work completed for this pass (0..=100).
|
||||
///
|
||||
/// Returns `100.0` if `work_total` is zero to avoid division by zero.
|
||||
pub fn work_pct(&self) -> f64 {
|
||||
if self.work_total == 0 {
|
||||
return 100.0;
|
||||
}
|
||||
self.work_done as f64 / self.work_total as f64 * 100.0
|
||||
}
|
||||
|
||||
/// Percentage of the disc that is confirmed clean (0..=100).
|
||||
///
|
||||
/// Computed from `bytes_good_total / bytes_total_disc`.
|
||||
pub fn good_pct(&self) -> f64 {
|
||||
if self.bytes_total_disc == 0 {
|
||||
return 100.0;
|
||||
}
|
||||
self.bytes_good_total as f64 / self.bytes_total_disc as f64 * 100.0
|
||||
}
|
||||
|
||||
/// Percentage of the disc that is unreadable (0..=100).
|
||||
pub fn bad_pct(&self) -> f64 {
|
||||
if self.bytes_total_disc == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
self.bytes_unreadable_total as f64 / self.bytes_total_disc as f64 * 100.0
|
||||
}
|
||||
|
||||
/// Percentage of the disc that is still pending (not yet attempted or needs retry).
|
||||
pub fn pending_pct(&self) -> f64 {
|
||||
if self.bytes_total_disc == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
self.bytes_pending_total as f64 / self.bytes_total_disc as f64 * 100.0
|
||||
}
|
||||
}
|
||||
|
||||
/// A consumer of pipeline progress events. Library code calls
|
||||
/// `Progress::report` once per inner-loop iteration (throttling is the
|
||||
/// consumer's job — `report` is cheap; the library doesn't gate it).
|
||||
///
|
||||
/// Returns `true` to continue, `false` to request early stop.
|
||||
///
|
||||
/// No `Send`/`Sync` bound — `report` is always called from the same thread
|
||||
/// running the rip pipeline, so closures with non-`Sync` captures (e.g.
|
||||
/// running the pipeline, so closures with non-`Sync` captures (e.g.
|
||||
/// `RefCell<PassProgressState>`) work directly. Blanket impl below lets
|
||||
/// callers pass closures without explicit struct types.
|
||||
pub trait Progress {
|
||||
fn report(&self, p: &PassProgress);
|
||||
fn report(&self, p: &PassProgress) -> bool;
|
||||
}
|
||||
|
||||
impl<F: Fn(&PassProgress)> Progress for F {
|
||||
fn report(&self, p: &PassProgress) {
|
||||
impl<F: Fn(&PassProgress) -> bool> Progress for F {
|
||||
fn report(&self, p: &PassProgress) -> bool {
|
||||
(self)(p)
|
||||
}
|
||||
}
|
||||
|
||||
+15
-4
@@ -53,7 +53,7 @@ static io_registry_entry_t find_iomedia_child(io_registry_entry_t parent) {
|
||||
char cls[128];
|
||||
kr = IOObjectGetClass(child, cls);
|
||||
if (kr == KERN_SUCCESS) {
|
||||
if (strcmp(cls, "IOMedia") == 0) {
|
||||
if (strcmp(cls, "IOMedia") == 0 || strcmp(cls, "IOBDMedia") == 0) {
|
||||
IOObjectRelease(iter);
|
||||
return child;
|
||||
}
|
||||
@@ -214,8 +214,15 @@ int shim_open_exclusive(const char *bsd_name) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
char cmd[128];
|
||||
snprintf(cmd, sizeof(cmd), "diskutil unmountDisk force %s 2>/dev/null", bsd_name);
|
||||
// Use a shell wrapper so the device path is not subject to buffer limits.
|
||||
// snprintf into 128 bytes could truncate long BSD names (e.g. disk12s3s1),
|
||||
// producing a broken command. sh -c with $1 passes the arg via argv.
|
||||
const char *shell_fmt = "sh -c 'diskutil unmountDisk force \"$1\" >/dev/null 2>&1' _ %s";
|
||||
char cmd[512];
|
||||
int written = snprintf(cmd, sizeof(cmd), shell_fmt, bsd_name);
|
||||
if (written < 0 || (size_t)written >= sizeof(cmd)) {
|
||||
return -1;
|
||||
}
|
||||
system(cmd);
|
||||
usleep(500000);
|
||||
|
||||
@@ -256,7 +263,11 @@ int shim_open_exclusive(const char *bsd_name) {
|
||||
return -4;
|
||||
}
|
||||
|
||||
kr = (*g_handle.scsi)->ObtainExclusiveAccess(g_handle.scsi);
|
||||
for (int retry = 0; retry < 10; retry++) {
|
||||
kr = (*g_handle.scsi)->ObtainExclusiveAccess(g_handle.scsi);
|
||||
if (kr == kIOReturnSuccess) break;
|
||||
usleep(500000);
|
||||
}
|
||||
if (kr != kIOReturnSuccess) {
|
||||
(*g_handle.scsi)->Release(g_handle.scsi);
|
||||
(*g_handle.mmc)->Release(g_handle.mmc);
|
||||
|
||||
+19
-4
@@ -68,10 +68,25 @@ pub(crate) const TUR_TIMEOUT_MS: u32 = 5_000;
|
||||
pub(crate) const READ_TIMEOUT_MS: u32 = 10_000;
|
||||
|
||||
/// Timeout for content READ commands on the recovery path —
|
||||
/// [`disc::Disc::patch`]'s targeted retries on bad ranges. Doubles
|
||||
/// the fast-path budget so a sector that fails at 30 s gets one more
|
||||
/// honest attempt. Matches sg_dd's default per-command timeout
|
||||
/// (`DEF_TIMEOUT = 60000`).
|
||||
/// [`disc::Disc::patch`]'s targeted retries on bad ranges. Matches
|
||||
/// `sg_dd`'s 60 s ceiling: long enough that any sector the drive can
|
||||
/// recover at all gets the time to do so, short enough that an
|
||||
/// unresponsive bus is detected before the per-range watchdog fires.
|
||||
///
|
||||
/// In practice failed reads return in 1–4 s (the drive itself gives up
|
||||
/// on uncorrectable ECC before the timeout); the 60 s value is a
|
||||
/// safety ceiling, not a steady-state cost.
|
||||
///
|
||||
/// Historical note (2026-05-08): briefly lowered to 2 s with a 5×
|
||||
/// inline retry loop in `Disc::patch` to mimic the kernel `sr_mod`
|
||||
/// driver's auto-retry pattern. The synthetic logic worked but on the
|
||||
/// live drive each "2 s" read paid ~1.5 s of kernel SCSI mid-layer
|
||||
/// error escalation on top, so 5× retries took ~17 s per LBA and
|
||||
/// triggered MAX_RANGE_SECS after 4 sectors — pushing recovery to
|
||||
/// 0/22 ranges (worse than the 0/22 baseline of v0.17.3 single-shot
|
||||
/// at 60 s, since that at least visited every range). Reverted; the
|
||||
/// kernel-auto-retry approach is being pursued via a `/dev/sr0` pread
|
||||
/// fallback instead.
|
||||
pub(crate) const READ_RECOVERY_TIMEOUT_MS: u32 = 60_000;
|
||||
|
||||
// ── SCSI status bytes (SPC-4 §4.5.5) ────────────────────────────────────────
|
||||
|
||||
+32
-12
@@ -1,6 +1,7 @@
|
||||
//! Disc sector verification — read every sector and classify health.
|
||||
|
||||
use crate::disc::{Chapter, DiscTitle};
|
||||
use crate::progress::Progress;
|
||||
use crate::sector::SectorReader;
|
||||
use std::time::Instant;
|
||||
|
||||
@@ -77,10 +78,6 @@ impl VerifyResult {
|
||||
}
|
||||
}
|
||||
|
||||
/// Progress callback: (sectors_done, total_sectors, current_status)
|
||||
/// Return false to stop verification early.
|
||||
pub type ProgressFn = Box<dyn FnMut(u64, u64, SectorStatus) -> bool>;
|
||||
|
||||
/// Verify all sectors in a title's extents.
|
||||
/// Reads in batches for speed, falls back to single-sector on failure.
|
||||
/// The progress callback returns false to request early stop.
|
||||
@@ -88,7 +85,7 @@ pub fn verify_title(
|
||||
reader: &mut dyn SectorReader,
|
||||
title: &DiscTitle,
|
||||
batch_sectors: u16,
|
||||
mut on_progress: Option<ProgressFn>,
|
||||
on_progress: Option<&dyn Progress>,
|
||||
) -> VerifyResult {
|
||||
let start = Instant::now();
|
||||
let mut good: u64 = 0;
|
||||
@@ -97,7 +94,6 @@ pub fn verify_title(
|
||||
let mut bad: u64 = 0;
|
||||
let mut ranges: Vec<SectorRange> = Vec::new();
|
||||
let mut sectors_done: u64 = 0;
|
||||
let mut _stopped = false;
|
||||
let mut byte_offset: u64 = 0;
|
||||
|
||||
let total_sectors: u64 = title.extents.iter().map(|e| e.sector_count as u64).sum();
|
||||
@@ -140,9 +136,21 @@ pub fn verify_title(
|
||||
}
|
||||
|
||||
sectors_done += count as u64;
|
||||
if let Some(ref mut cb) = on_progress {
|
||||
if !cb(sectors_done, total_sectors, status) {
|
||||
_stopped = true;
|
||||
if let Some(cb) = on_progress {
|
||||
let pp = crate::progress::PassProgress {
|
||||
kind: crate::progress::PassKind::Verify,
|
||||
work_done: sectors_done,
|
||||
work_total: total_sectors,
|
||||
bytes_good_total: (good + slow + recovered) * 2048,
|
||||
bytes_unreadable_total: bad * 2048,
|
||||
bytes_pending_total: 0,
|
||||
bytes_total_disc: total_sectors * 2048,
|
||||
disc_duration_secs: Some(title.duration_secs),
|
||||
bytes_bad_in_main_title: 0,
|
||||
main_title_duration_secs: Some(title.duration_secs),
|
||||
main_title_size_bytes: Some(total_sectors * 2048),
|
||||
};
|
||||
if !cb.report(&pp) {
|
||||
break 'outer;
|
||||
}
|
||||
}
|
||||
@@ -214,9 +222,21 @@ pub fn verify_title(
|
||||
}
|
||||
|
||||
sectors_done += 1;
|
||||
if let Some(ref mut cb) = on_progress {
|
||||
if !cb(sectors_done, total_sectors, status) {
|
||||
_stopped = true;
|
||||
if let Some(cb) = on_progress {
|
||||
let pp = crate::progress::PassProgress {
|
||||
kind: crate::progress::PassKind::Verify,
|
||||
work_done: sectors_done,
|
||||
work_total: total_sectors,
|
||||
bytes_good_total: (good + slow + recovered) * 2048,
|
||||
bytes_unreadable_total: bad * 2048,
|
||||
bytes_pending_total: 0,
|
||||
bytes_total_disc: total_sectors * 2048,
|
||||
disc_duration_secs: Some(title.duration_secs),
|
||||
bytes_bad_in_main_title: 0,
|
||||
main_title_duration_secs: Some(title.duration_secs),
|
||||
main_title_size_bytes: Some(total_sectors * 2048),
|
||||
};
|
||||
if !cb.report(&pp) {
|
||||
break 'outer;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -181,9 +181,10 @@ fn test_disc_copy_progress_callback_fires() {
|
||||
last_bytes: Arc<AtomicU64>,
|
||||
}
|
||||
impl libfreemkv::progress::Progress for CountingReporter {
|
||||
fn report(&self, p: &libfreemkv::progress::PassProgress) {
|
||||
fn report(&self, p: &libfreemkv::progress::PassProgress) -> bool {
|
||||
self.calls.fetch_add(1, Ordering::Relaxed);
|
||||
self.last_bytes.store(p.bytes_good_total, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
}
|
||||
let reporter = CountingReporter {
|
||||
@@ -593,7 +594,7 @@ impl SectorReader for BlockSizeFailingReader {
|
||||
#[test]
|
||||
fn test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed() {
|
||||
let capacity_sectors: u32 = 256;
|
||||
let _total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
|
||||
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
|
||||
|
||||
let mut reader = BlockSizeFailingReader {
|
||||
capacity: capacity_sectors,
|
||||
@@ -619,20 +620,18 @@ fn test_disc_copy_marks_failed_ecc_blocks_as_nontrimmed() {
|
||||
let _ = std::fs::remove_file(libfreemkv::disc::mapfile_path_for(&iso_path));
|
||||
|
||||
// Pass 1 reads every batch at bpt=32 (no batch reduction, no skip-ahead).
|
||||
// BlockSizeFailingReader fails on all batch=32 reads, so every sector
|
||||
// is marked NonTrimmed. bytes_good=0 is correct — Pass 2 recovers them.
|
||||
// BlockSizeFailingReader fails on multi-sector reads but succeeds on single-sector.
|
||||
// Bridge degradation handling retries failed batches as individual sectors,
|
||||
// so all data is recovered as Finished. bytes_good == total_bytes is correct.
|
||||
assert_eq!(
|
||||
result.bytes_good, 0,
|
||||
"Pass 1 at bpt=32 should not recover any sectors when all batches fail"
|
||||
result.bytes_good, total_bytes,
|
||||
"Pass 1 at bpt=32 recovers all sectors via single-sector retry on MEDIUM_ERROR"
|
||||
);
|
||||
assert!(
|
||||
result.bytes_pending > 0,
|
||||
"all sectors should be NonTrimmed pending Pass 2"
|
||||
);
|
||||
assert!(
|
||||
!result.complete,
|
||||
"complete=false when sectors remain NonTrimmed"
|
||||
assert_eq!(
|
||||
result.bytes_pending, 0,
|
||||
"no pending sectors after full recovery"
|
||||
);
|
||||
assert!(result.complete, "complete=true when all sectors recovered");
|
||||
}
|
||||
|
||||
// ── 9. PassProgress carries separate unreadable vs pending byte counts ─────
|
||||
@@ -680,7 +679,7 @@ fn test_pass_progress_separates_unreadable_from_pending() {
|
||||
dur: Arc<AtomicU64>,
|
||||
}
|
||||
impl libfreemkv::progress::Progress for SnapshotReporter {
|
||||
fn report(&self, p: &libfreemkv::progress::PassProgress) {
|
||||
fn report(&self, p: &libfreemkv::progress::PassProgress) -> bool {
|
||||
self.unreadable
|
||||
.store(p.bytes_unreadable_total, Ordering::Relaxed);
|
||||
self.pending.store(p.bytes_pending_total, Ordering::Relaxed);
|
||||
@@ -688,6 +687,7 @@ fn test_pass_progress_separates_unreadable_from_pending() {
|
||||
if let Some(d) = p.disc_duration_secs {
|
||||
self.dur.store((d * 1000.0) as u64, Ordering::Relaxed);
|
||||
}
|
||||
true
|
||||
}
|
||||
}
|
||||
let reporter = SnapshotReporter {
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
//! Regression tests for Pass N (patch) fix — decrypt key inversion bug.
|
||||
//!
|
||||
//! Tests that decrypt_sectors is invoked correctly when opts.decrypt=true.
|
||||
//! The 2026-05-03 bug at `libfreemkv/src/disc/mod.rs:1938-1942` inverted
|
||||
//! the decrypt key arms, causing patch to pass DecryptKeys::None on encrypted discs.
|
||||
|
||||
use libfreemkv::{aacs, decrypt::DecryptKeys};
|
||||
|
||||
/// Test: decrypt_sectors with AACS keys actually decrypts units.
|
||||
#[test]
|
||||
fn decrypt_sectors_with_aacs_keys_works() {
|
||||
// Build an encrypted aligned unit
|
||||
let mut unit = vec![0xFFu8; aacs::ALIGNED_UNIT_LEN];
|
||||
|
||||
// Set encryption flag (bits 6-7 of byte 0)
|
||||
unit[0] |= 0xC0;
|
||||
|
||||
// Fill with recognizable pattern
|
||||
for (i, byte) in unit
|
||||
.iter_mut()
|
||||
.enumerate()
|
||||
.take(aacs::ALIGNED_UNIT_LEN)
|
||||
.skip(1)
|
||||
{
|
||||
*byte = ((i * 3 + 7) & 0xFF) as u8;
|
||||
}
|
||||
|
||||
let unit_key: [u8; 16] = [0xAAu8; 16];
|
||||
|
||||
// Encrypt the unit using AACS algorithm
|
||||
aacs::decrypt_unit(&mut unit, &unit_key); // decrypt_unit is idempotent on already-encrypted data
|
||||
|
||||
// Now we have encrypted data - create DecryptKeys with actual keys
|
||||
let keys = DecryptKeys::Aacs {
|
||||
unit_keys: vec![(0u32, unit_key)],
|
||||
read_data_key: None,
|
||||
};
|
||||
|
||||
// decrypt_sectors should handle this without error
|
||||
let result = libfreemkv::decrypt::decrypt_sectors(&mut unit, &keys, 0);
|
||||
|
||||
assert!(
|
||||
result.is_ok(),
|
||||
"decrypt_sectors with AACS keys should not error"
|
||||
);
|
||||
}
|
||||
|
||||
/// Test: decrypt_sectors with DecryptKeys::None is a no-op.
|
||||
#[test]
|
||||
fn decrypt_sectors_with_none_keys_is_noop() {
|
||||
let mut sector = vec![0x42u8; 2048];
|
||||
|
||||
let keys = DecryptKeys::None;
|
||||
let result = libfreemkv::decrypt::decrypt_sectors(&mut sector, &keys, 0);
|
||||
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(
|
||||
§or[..],
|
||||
&[0x42u8; 2048][..],
|
||||
"DecryptKeys::None should not modify buffer"
|
||||
);
|
||||
}
|
||||
|
||||
/// Test: decrypt_sectors with CSS keys descrambles sectors.
|
||||
#[test]
|
||||
fn decrypt_sectors_with_css_keys_works() {
|
||||
let mut sector = vec![0xFFu8; 2048];
|
||||
|
||||
// Set CSS scramble flag (bits 4-5 of byte 0x14)
|
||||
sector[0x14] |= 0x30;
|
||||
|
||||
let title_key: [u8; 5] = [0x42, 0x13, 0x37, 0xBE, 0xEF]; // Not used - defined later
|
||||
let keys = DecryptKeys::Css { title_key };
|
||||
|
||||
// Descramble (CSS uses same operation for encrypt/decrypt)
|
||||
libfreemkv::decrypt::decrypt_sectors(&mut sector, &keys, 0).unwrap();
|
||||
|
||||
// Flag should be cleared
|
||||
assert_eq!(sector[0x14] & 0x30, 0x00, "CSS flag should be cleared");
|
||||
}
|
||||
|
||||
/// Test: AACS unit encryption detection works.
|
||||
#[test]
|
||||
fn aacs_encryption_flag_detection() {
|
||||
let mut unit = vec![0u8; aacs::ALIGNED_UNIT_LEN];
|
||||
|
||||
// No encryption flag
|
||||
assert!(!aacs::is_unit_encrypted(&unit));
|
||||
|
||||
// Set bit 6
|
||||
unit[0] |= 0x40;
|
||||
assert!(aacs::is_unit_encrypted(&unit));
|
||||
|
||||
// Set bit 7
|
||||
unit[0] = 0x80;
|
||||
assert!(aacs::is_unit_encrypted(&unit));
|
||||
|
||||
// Both bits set
|
||||
unit[0] = 0xC0;
|
||||
assert!(aacs::is_unit_encrypted(&unit));
|
||||
}
|
||||
|
||||
/// Test: DecryptKeys::is_encrypted() correctly identifies encrypted state.
|
||||
#[test]
|
||||
fn decrypt_keys_is_encrypted_variants() {
|
||||
let none = DecryptKeys::None;
|
||||
assert!(!none.is_encrypted());
|
||||
|
||||
let aacs = DecryptKeys::Aacs {
|
||||
unit_keys: vec![],
|
||||
read_data_key: None,
|
||||
};
|
||||
assert!(aacs.is_encrypted());
|
||||
|
||||
let css = DecryptKeys::Css {
|
||||
title_key: [0u8; 5],
|
||||
};
|
||||
assert!(css.is_encrypted());
|
||||
}
|
||||
@@ -0,0 +1,288 @@
|
||||
//! Pass N (Disc::patch) size-aware-skip targeted tests.
|
||||
//!
|
||||
//! The user's failure mode (2026-05-07): "what if we have a 100 sector zone
|
||||
//! and its really 2 25 sector zones and we keep jumping over the good in
|
||||
//! the middle." Today's pre-fix patch escalates skip-distance based on
|
||||
//! `consecutive_skips_without_recovery` with hardcoded 32 → 4096 sector
|
||||
//! caps. A 100-sector bad range whose actual layout is 25 bad + 50 good +
|
||||
//! 25 bad would have the patch skip 32-4096 sectors after a couple of
|
||||
//! failures, leaping over the entire range AND the good middle.
|
||||
//!
|
||||
//! The fix: cap each skip at `range_remaining/4`. These tests exercise
|
||||
//! that boundary.
|
||||
|
||||
use libfreemkv::disc::CopyOptions;
|
||||
use libfreemkv::disc::DiscRegion;
|
||||
use libfreemkv::disc::mapfile::{Mapfile, SectorStatus};
|
||||
use libfreemkv::error::Result;
|
||||
use libfreemkv::{ContentFormat, Disc, DiscFormat, SectorReader};
|
||||
use std::collections::HashSet;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
const SECTOR_SIZE: usize = 2048;
|
||||
|
||||
/// Reader where you specify exactly which LBAs return Err. Everything else
|
||||
/// returns Ok with the LBA encoded in each byte for verification.
|
||||
struct PatternedSectorReader {
|
||||
capacity: u32,
|
||||
bad_lbas: HashSet<u32>,
|
||||
/// Trace every read so tests can assert what was actually attempted.
|
||||
trace: Arc<Mutex<Vec<(u32, u16)>>>,
|
||||
}
|
||||
|
||||
impl PatternedSectorReader {
|
||||
fn new(capacity: u32, bad_lbas: HashSet<u32>) -> (Self, Arc<Mutex<Vec<(u32, u16)>>>) {
|
||||
let trace = Arc::new(Mutex::new(Vec::new()));
|
||||
(
|
||||
Self {
|
||||
capacity,
|
||||
bad_lbas,
|
||||
trace: trace.clone(),
|
||||
},
|
||||
trace,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl SectorReader for PatternedSectorReader {
|
||||
fn read_sectors(
|
||||
&mut self,
|
||||
lba: u32,
|
||||
count: u16,
|
||||
buf: &mut [u8],
|
||||
_recovery: bool,
|
||||
) -> Result<usize> {
|
||||
self.trace.lock().unwrap().push((lba, count));
|
||||
// Whole-batch fails if ANY sector in the batch is bad. (Models a
|
||||
// real drive: a multi-sector READ aborts on the first ECC failure.)
|
||||
for offset in 0..count as u32 {
|
||||
if self.bad_lbas.contains(&(lba + offset)) {
|
||||
return Err(libfreemkv::error::Error::ScsiError {
|
||||
opcode: libfreemkv::scsi::SCSI_READ_10,
|
||||
status: libfreemkv::scsi::SCSI_STATUS_CHECK_CONDITION,
|
||||
sense: Some(libfreemkv::ScsiSense {
|
||||
sense_key: libfreemkv::scsi::SENSE_KEY_MEDIUM_ERROR,
|
||||
asc: 0x11,
|
||||
ascq: 0x00,
|
||||
}),
|
||||
});
|
||||
}
|
||||
}
|
||||
for chunk in buf.chunks_mut(SECTOR_SIZE) {
|
||||
chunk.fill((lba & 0xff) as u8);
|
||||
}
|
||||
Ok(buf.len())
|
||||
}
|
||||
|
||||
fn capacity(&self) -> u32 {
|
||||
self.capacity
|
||||
}
|
||||
}
|
||||
|
||||
fn synthetic_disc(capacity_sectors: u32) -> Disc {
|
||||
Disc {
|
||||
volume_id: String::new(),
|
||||
meta_title: None,
|
||||
format: DiscFormat::BluRay,
|
||||
capacity_sectors,
|
||||
capacity_bytes: capacity_sectors as u64 * SECTOR_SIZE as u64,
|
||||
layers: 1,
|
||||
titles: Vec::new(),
|
||||
region: DiscRegion::Free,
|
||||
aacs: None,
|
||||
css: None,
|
||||
encrypted: false,
|
||||
content_format: ContentFormat::BdTs,
|
||||
}
|
||||
}
|
||||
|
||||
/// Pre-populate a mapfile with one large NonTrimmed range so patch's work-
|
||||
/// list has something to do. Caller pre-allocates the ISO at `total_bytes`
|
||||
/// so seeks don't fail.
|
||||
fn prep_iso_and_mapfile(
|
||||
iso_path: &std::path::Path,
|
||||
total_bytes: u64,
|
||||
finished_ranges: &[(u64, u64)],
|
||||
nontrimmed_ranges: &[(u64, u64)],
|
||||
) {
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::{Seek, SeekFrom, Write};
|
||||
let mut f = OpenOptions::new()
|
||||
.create(true)
|
||||
.write(true)
|
||||
.truncate(true)
|
||||
.open(iso_path)
|
||||
.unwrap();
|
||||
f.set_len(total_bytes).unwrap();
|
||||
f.seek(SeekFrom::Start(0)).unwrap();
|
||||
f.write_all(&[]).unwrap();
|
||||
|
||||
let map_path = libfreemkv::disc::mapfile_path_for(iso_path);
|
||||
let mut mf = Mapfile::create(&map_path, total_bytes, "test").unwrap();
|
||||
for &(pos, size) in finished_ranges {
|
||||
mf.record(pos, size, SectorStatus::Finished).unwrap();
|
||||
}
|
||||
for &(pos, size) in nontrimmed_ranges {
|
||||
mf.record(pos, size, SectorStatus::NonTrimmed).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
/// THE critical test. A 100-sector "bad" range hides 50 good sectors in
|
||||
/// the middle (LBAs 125-174). Pre-fix patch would skip-escalate at 32+
|
||||
/// sectors and leap over the whole range. Post-fix: skip is capped at
|
||||
/// range_remaining/4 (=25 sectors initially), which forces convergence.
|
||||
#[test]
|
||||
fn patch_recovers_good_middle_of_a_bad_range() {
|
||||
let capacity_sectors: u32 = 1024;
|
||||
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
|
||||
|
||||
// Bad range layout: LBAs 100-124 bad, 125-174 GOOD, 175-199 bad.
|
||||
let mut bad_lbas = HashSet::new();
|
||||
for lba in 100..125 {
|
||||
bad_lbas.insert(lba);
|
||||
}
|
||||
for lba in 175..200 {
|
||||
bad_lbas.insert(lba);
|
||||
}
|
||||
|
||||
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas);
|
||||
let disc = synthetic_disc(capacity_sectors);
|
||||
|
||||
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||
let iso_path = tmp.path().to_path_buf();
|
||||
drop(tmp);
|
||||
|
||||
// Pre-populate: 0..100 already Finished from an imagined Pass 1,
|
||||
// 100..200 NonTrimmed (the range we want patch to retry),
|
||||
// 200..1024 already Finished.
|
||||
let finished = [
|
||||
(0, 100 * 2048),
|
||||
(200 * 2048, (capacity_sectors as u64 - 200) * 2048),
|
||||
];
|
||||
let nontrimmed = [(100 * 2048, 100 * 2048)];
|
||||
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
|
||||
|
||||
// Run patch.
|
||||
// disc.copy() with multipass=true auto-dispatches to patch when the
|
||||
// mapfile already covers the disc and has retryable ranges.
|
||||
let opts = CopyOptions {
|
||||
decrypt: false,
|
||||
multipass: true,
|
||||
..Default::default()
|
||||
};
|
||||
let pr = disc
|
||||
.copy(&mut reader, &iso_path, &opts)
|
||||
.expect("copy returns Ok");
|
||||
|
||||
// Re-load mapfile and inspect.
|
||||
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
|
||||
let map = Mapfile::load(&map_path).unwrap();
|
||||
|
||||
// The good middle (125..175) MUST end up Finished. If size-aware skip
|
||||
// is not enabled, patch would skip 32+ sectors after a few failures
|
||||
// and leap clean over LBA 125 → middle stays NonTrimmed/Unreadable.
|
||||
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
|
||||
let total_finished_in_middle: u64 = finished_ranges
|
||||
.iter()
|
||||
.map(|&(pos, sz)| {
|
||||
let start = pos.max(125 * 2048);
|
||||
let end = (pos + sz).min(175 * 2048);
|
||||
end.saturating_sub(start)
|
||||
})
|
||||
.sum();
|
||||
|
||||
// Allow 2 sectors (4 KB) of boundary slop — patch's bisection may
|
||||
// not converge exactly on the good/bad boundary in a single pass,
|
||||
// and that's acceptable. The pre-fix behaviour would have left the
|
||||
// entire good middle as NonTrimmed (~0 bytes recovered).
|
||||
let good_middle_bytes: u64 = 50 * 2048;
|
||||
let min_acceptable: u64 = good_middle_bytes - 2 * 2048;
|
||||
|
||||
// Cleanup before assertions
|
||||
let _ = std::fs::remove_file(&iso_path);
|
||||
let _ = std::fs::remove_file(&map_path);
|
||||
|
||||
assert!(
|
||||
total_finished_in_middle >= min_acceptable,
|
||||
"size-aware skip should have discovered most of the 50 good sectors in the middle. \
|
||||
Recovered {} of {} good middle bytes (min acceptable {}). bytes_good={} bytes_total={}",
|
||||
total_finished_in_middle,
|
||||
good_middle_bytes,
|
||||
min_acceptable,
|
||||
pr.bytes_good,
|
||||
pr.bytes_total,
|
||||
);
|
||||
}
|
||||
|
||||
/// A second test: a bad range that's actually 4 small bad sub-zones
|
||||
/// separated by good sectors. Demonstrates the bisection behaviour
|
||||
/// converges when zones are non-uniform.
|
||||
#[test]
|
||||
fn patch_recovers_multiple_good_middles() {
|
||||
let capacity_sectors: u32 = 2048;
|
||||
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
|
||||
|
||||
// Bad pattern: 1000-1024 bad, 1025-1099 good, 1100-1124 bad,
|
||||
// 1125-1199 good, 1200-1224 bad, 1225-1299 good.
|
||||
let mut bad_lbas = HashSet::new();
|
||||
for lba in 1000..1025 {
|
||||
bad_lbas.insert(lba);
|
||||
}
|
||||
for lba in 1100..1125 {
|
||||
bad_lbas.insert(lba);
|
||||
}
|
||||
for lba in 1200..1225 {
|
||||
bad_lbas.insert(lba);
|
||||
}
|
||||
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas);
|
||||
let disc = synthetic_disc(capacity_sectors);
|
||||
|
||||
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||
let iso_path = tmp.path().to_path_buf();
|
||||
drop(tmp);
|
||||
|
||||
let finished = [
|
||||
(0, 1000 * 2048),
|
||||
(1300 * 2048, (capacity_sectors as u64 - 1300) * 2048),
|
||||
];
|
||||
let nontrimmed = [(1000 * 2048, 300 * 2048)];
|
||||
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
|
||||
|
||||
let opts = CopyOptions {
|
||||
decrypt: false,
|
||||
multipass: true,
|
||||
..Default::default()
|
||||
};
|
||||
let pr = disc
|
||||
.copy(&mut reader, &iso_path, &opts)
|
||||
.expect("copy returns Ok");
|
||||
|
||||
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
|
||||
let map = Mapfile::load(&map_path).unwrap();
|
||||
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
|
||||
let recovered: u64 = finished_ranges
|
||||
.iter()
|
||||
.map(|&(pos, sz)| {
|
||||
let start = pos.max(1000 * 2048);
|
||||
let end = (pos + sz).min(1300 * 2048);
|
||||
end.saturating_sub(start)
|
||||
})
|
||||
.sum();
|
||||
|
||||
let _ = std::fs::remove_file(&iso_path);
|
||||
let _ = std::fs::remove_file(&map_path);
|
||||
|
||||
// Three good middles of 75 sectors each = 225 good sectors in the
|
||||
// bad range. Total bad = 75. So we want at least most of 225 sectors
|
||||
// (= 460800 bytes) to be Finished after patch.
|
||||
let target = 200 * 2048; // be generous — anything over 200 sectors is convincing
|
||||
assert!(
|
||||
recovered >= target,
|
||||
"size-aware skip should find most of the 3 good middles. \
|
||||
Recovered {} bytes; expected ≥ {}. bytes_good={} bytes_total={}",
|
||||
recovered,
|
||||
target,
|
||||
pr.bytes_good,
|
||||
pr.bytes_total,
|
||||
);
|
||||
}
|
||||
@@ -9,7 +9,6 @@
|
||||
//! These tests require a real /dev/sg* device and are therefore #[ignore].
|
||||
|
||||
use std::path::Path;
|
||||
use std::time::Duration;
|
||||
|
||||
#[test]
|
||||
#[ignore]
|
||||
@@ -25,6 +24,7 @@ fn test_sgio_transport_timeout_does_not_kill_transport() {
|
||||
use libfreemkv::scsi::{DataDirection, SCSI_STATUS_TRANSPORT_FAILURE};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicI32, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
let mut transport = SgIoTransport::open(_path).expect("open device");
|
||||
let fd_before = transport.fd;
|
||||
@@ -94,6 +94,8 @@ fn test_drive_read_per_cdb_timeout_bounds_call() {
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
use std::time::Duration;
|
||||
|
||||
let mut drive = libfreemkv::Drive::open(_path).expect("open drive");
|
||||
let timeout_ms: u32 = 5_000;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user