disc/read_error: wedge PREVENTION — jump on first error + 30s cooldown
Rewrites the Pass 1 wedge handling from "slow skip after the drive has already wedged" to "prevent the wedge transition in the first place." Driven by 2026-05-11 empirical data: the BU40N transitioned into IllegalRequest fast-fail mode at exactly 7 medium errors in 6.5 seconds (~1 read/sec retry cadence). Once there, only physical eject + reload clears it — 30s pauses + 1 GB jumps do not. The fix is the user's mental model from that session: "We can detect bad reads, failed reads, and asking to read again fast after causes a wedge. We need to prevent the wedge in the first place." Two changes to the centralized error handler: 1. **`for_sweep().fast_jump_threshold = 1`** (was 4). Pass 1 now JumpAheads on the FIRST outer-batch failure, not the 4th. The drive never gets back-to-back retries at the same LBA in Pass 1 — every error → jump 64 MB forward + long cooldown. Pass N keeps `fast_jump_threshold = u64::MAX` because retries on already-known- bad LBAs are its whole job. 2. **`ZONE_ENTRY_COOLDOWN_SECS = 30`**. The FIRST error after a clean run (when `consecutive_outer_failures == 1` and we're not bisecting) uses this long pause instead of the standard 5 s FAIL_PAUSE_SECS. Gives the BU40N's firmware / bridge internal retry counters 30 s of breathing room before the next read, preventing the "7 errors in 6.5 s" cascade. Subsequent errors in the same zone use the standard 5 s pause (we've already jumped past the initial damage; further errors mean we landed in another bad cluster). Pass N exempt from the zone-entry cooldown — `bisect_on_marginal= true` skips the long-pause arm. Pass N's per-sector retries on known-bad LBAs would multiply uselessly with 30 s/error. Test updates: 4 tests' expected behavior changed under the new policy. Renamed `pass_1_marginal_skips_instead_of_bisecting` → `pass_1_marginal_jumps_immediately_not_bisecting`. Renamed `pass_1_jumps_after_4_consecutive_outer_failures` → `pass_1_jumps_immediately_on_first_outer_failure`. Updated `both_passes_pause_on_failed_read_for_wedge_avoidance` (now `pass_1_zone_entry_uses_long_cooldown` + `pass_n_pauses_uniformly_on_failed_read`). Cost analysis: - Clean disc (no errors): unchanged. 0% overhead. - Lightly damaged (1-2 zones): +30 s per zone = ~1 min total. Fine. - Heavily damaged (10+ zones): +5+ min total. The trade for never wedging the drive and getting a usable Pass N afterwards. Expected behavior on the next damaged-disc rip: - Pass 1 hits damage at LBA X → jumps 64 MB forward immediately, pauses 30 s - Drive's firmware never accumulates the retry pressure that triggers IllegalRequest fast-fail - bytes_maybe accumulates faster (we skip more), but Pass N picks up the slack with proper per-sector recovery — and Pass N can actually RUN because the drive isn't wedged
This commit is contained in:
+109
-51
@@ -147,8 +147,13 @@ impl ReadCtx {
|
|||||||
/// is the one that grinds on the bad ranges. So bisect-on-
|
/// is the one that grinds on the bad ranges. So bisect-on-
|
||||||
/// marginal is OFF (failed batches become SkipBlock; whole 32-
|
/// marginal is OFF (failed batches become SkipBlock; whole 32-
|
||||||
/// sector blocks marked NonTrimmed for Pass N to revisit), and
|
/// sector blocks marked NonTrimmed for Pass N to revisit), and
|
||||||
/// the damage-jump fast-path triggers after just 4 consecutive
|
/// the damage-jump fast-path triggers after just 1 consecutive
|
||||||
/// outer-batch failures.
|
/// outer-batch failure — the user's wedge-prevention principle
|
||||||
|
/// (2026-05-11): once the drive returns ANY recoverable error,
|
||||||
|
/// retrying the same LBA quickly is what triggers the firmware
|
||||||
|
/// fast-fail transition. Jump immediately, never retry in Pass 1.
|
||||||
|
/// Pass N owns retries — it gets per-sector timeouts that don't
|
||||||
|
/// hammer the firmware the same way.
|
||||||
pub fn for_sweep(batch: u16) -> Self {
|
pub fn for_sweep(batch: u16) -> Self {
|
||||||
Self {
|
Self {
|
||||||
batch,
|
batch,
|
||||||
@@ -158,7 +163,7 @@ impl ReadCtx {
|
|||||||
damage_window: Vec::with_capacity(16),
|
damage_window: Vec::with_capacity(16),
|
||||||
damage_window_max: 16,
|
damage_window_max: 16,
|
||||||
damage_threshold_pct: 12,
|
damage_threshold_pct: 12,
|
||||||
fast_jump_threshold: 4,
|
fast_jump_threshold: 1,
|
||||||
jump_multiplier: 1,
|
jump_multiplier: 1,
|
||||||
not_ready_retries: 0,
|
not_ready_retries: 0,
|
||||||
bridge_degradation_count: 0,
|
bridge_degradation_count: 0,
|
||||||
@@ -309,6 +314,22 @@ pub enum ReadAction {
|
|||||||
/// internal) → return → cooldown pause → next read. Same shape
|
/// internal) → return → cooldown pause → next read. Same shape
|
||||||
/// everywhere reads can fail.
|
/// everywhere reads can fail.
|
||||||
const FAIL_PAUSE_SECS: u64 = 5;
|
const FAIL_PAUSE_SECS: u64 = 5;
|
||||||
|
/// Long cooldown applied when a damage zone is first entered (the
|
||||||
|
/// FIRST read failure after a clean run, before the drive has had a
|
||||||
|
/// chance to cycle in retries that push it toward fast-fail).
|
||||||
|
///
|
||||||
|
/// Empirical: 2026-05-11 Dune Pt 2 wedge incident showed 7 medium
|
||||||
|
/// errors in 6.5 seconds (~1s per attempt + ~1s pause) push the
|
||||||
|
/// BU40N's firmware into IllegalRequest fast-fail mode permanently.
|
||||||
|
/// Once there, only physical eject + reload clears it. Giving the
|
||||||
|
/// drive 30s of breathing room after the FIRST error in a zone —
|
||||||
|
/// before we start adding more error counts in the firmware's
|
||||||
|
/// internal window — prevents the transition.
|
||||||
|
///
|
||||||
|
/// Cost on clean discs: zero (first-error path doesn't trigger).
|
||||||
|
/// Cost on damaged discs: ~30s × N damage zones; on a 5-zone disc
|
||||||
|
/// that's 2.5 min extra. Trade for never wedging the drive.
|
||||||
|
const ZONE_ENTRY_COOLDOWN_SECS: u64 = 30;
|
||||||
/// Cooldown when a long streak of failures suggests the drive is
|
/// Cooldown when a long streak of failures suggests the drive is
|
||||||
/// stuck in a damage zone and needs MORE breathing room than the
|
/// stuck in a damage zone and needs MORE breathing room than the
|
||||||
/// standard inter-error pause. Same value as `FAIL_PAUSE_SECS`
|
/// standard inter-error pause. Same value as `FAIL_PAUSE_SECS`
|
||||||
@@ -599,12 +620,29 @@ pub fn handle_read_error(err: &Error, ctx: &mut ReadCtx) -> ReadAction {
|
|||||||
bad_count * 100 / ctx.damage_window.len()
|
bad_count * 100 / ctx.damage_window.len()
|
||||||
};
|
};
|
||||||
|
|
||||||
// Inter-error pause — wedge avoidance, applied uniformly across
|
// Inter-error pause — wedge prevention via pacing.
|
||||||
// Pass 1 and Pass N. A failed read is a failed read; same
|
//
|
||||||
// cool-down regardless of which pass is calling. The escalation
|
// Zone-entry case (first error after a clean run): apply the
|
||||||
// arm (LONG_PAUSE after a long streak) currently resolves to the
|
// long ZONE_ENTRY_COOLDOWN_SECS pause. The empirical wedge
|
||||||
// same value but is kept as a separate branch for future tuning.
|
// observed 2026-05-11 happened ~7 errors into a damage zone,
|
||||||
let pause_secs = if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
// each retry adding to the firmware's internal counter. A 30s
|
||||||
|
// pause at zone entry lets the drive's bridge / firmware
|
||||||
|
// counters reset before we issue the next read.
|
||||||
|
//
|
||||||
|
// Subsequent errors in the same zone: the standard 5s pause.
|
||||||
|
// (We've already jumped past the initial damage; further errors
|
||||||
|
// mean we landed in another bad cluster — same pacing applies.)
|
||||||
|
//
|
||||||
|
// Long-streak escalation: same 5s currently; kept as a separate
|
||||||
|
// branch for future tuning. Pass N (bisect_on_marginal=true)
|
||||||
|
// uses the standard pauses — it's running single-sector retries
|
||||||
|
// on already-known-bad LBAs by design.
|
||||||
|
let is_zone_entry = ctx.consecutive_outer_failures == 1
|
||||||
|
&& !ctx.bisecting
|
||||||
|
&& !ctx.bisect_on_marginal;
|
||||||
|
let pause_secs = if is_zone_entry {
|
||||||
|
ZONE_ENTRY_COOLDOWN_SECS
|
||||||
|
} else if ctx.consecutive_failures >= CONSECUTIVE_FAIL_LONG_PAUSE_THRESHOLD {
|
||||||
CONSECUTIVE_FAIL_LONG_PAUSE_SECS
|
CONSECUTIVE_FAIL_LONG_PAUSE_SECS
|
||||||
} else {
|
} else {
|
||||||
FAIL_PAUSE_SECS
|
FAIL_PAUSE_SECS
|
||||||
@@ -712,15 +750,18 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pass_1_marginal_skips_instead_of_bisecting() {
|
fn pass_1_marginal_jumps_immediately_not_bisecting() {
|
||||||
// Pass 1's job is "fast and accurate" — leave bisection to
|
// 2026-05-11 wedge-prevention rewrite: Pass 1 jumps on the
|
||||||
// Pass N. A failed batch becomes SkipBlock (whole 32-sector
|
// FIRST marginal error (fast_jump_threshold=1) rather than
|
||||||
// block marked NonTrimmed for Pass N to revisit).
|
// SkipBlock. Retrying the same LBA quickly is what triggers
|
||||||
|
// the BU40N's firmware fast-fail transition; immediate jump
|
||||||
|
// prevents the cascade. Pass N still bisects (its job is
|
||||||
|
// per-sector recovery on already-known-bad LBAs).
|
||||||
let mut ctx = ReadCtx::for_sweep(32);
|
let mut ctx = ReadCtx::for_sweep(32);
|
||||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||||
match action {
|
match action {
|
||||||
ReadAction::SkipBlock { .. } => {}
|
ReadAction::JumpAhead { .. } => {}
|
||||||
other => panic!("expected SkipBlock for Pass 1, got {other:?}"),
|
other => panic!("expected JumpAhead on first Pass 1 marginal error, got {other:?}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -746,26 +787,18 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pass_1_jumps_after_4_consecutive_outer_failures() {
|
fn pass_1_jumps_immediately_on_first_outer_failure() {
|
||||||
// The fast-entry trigger: Pass 1 should JumpAhead after 4
|
// 2026-05-11 rewrite: fast_jump_threshold is 1 on Pass 1, not
|
||||||
// consecutive outer-batch failures, BEFORE the 16-block
|
// 4. Even ONE error triggers a jump because BU40N's firmware
|
||||||
// damage window has filled. Otherwise we spend ~40 minutes
|
// fast-fail mode is sensitive to retry cadence. The wedge
|
||||||
// of bisecting/grinding to fill the window before the first
|
// observed 2026-05-11 happened at 7 errors / 6.5s — by then
|
||||||
// jump on a damage zone we entered cleanly.
|
// we were already wedged. Jumping on error #1 means we
|
||||||
|
// physically can't reach the cascade.
|
||||||
let mut ctx = ReadCtx::for_sweep(32);
|
let mut ctx = ReadCtx::for_sweep(32);
|
||||||
// First three should NOT jump (still under threshold of 4).
|
|
||||||
for _ in 0..3 {
|
|
||||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
|
||||||
assert!(
|
|
||||||
!matches!(a, ReadAction::JumpAhead { .. }),
|
|
||||||
"should not jump until 4 consecutive outer failures"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
// Fourth should jump.
|
|
||||||
let a = handle_read_error(&medium_err(), &mut ctx);
|
let a = handle_read_error(&medium_err(), &mut ctx);
|
||||||
assert!(
|
assert!(
|
||||||
matches!(a, ReadAction::JumpAhead { .. }),
|
matches!(a, ReadAction::JumpAhead { .. }),
|
||||||
"expected JumpAhead at 4th consecutive outer failure, got {a:?}"
|
"expected JumpAhead on first outer failure (fast_jump_threshold=1), got {a:?}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -786,12 +819,15 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn outer_success_resets_consecutive_outer_failures() {
|
fn outer_success_resets_consecutive_outer_failures() {
|
||||||
|
// With fast_jump_threshold=1 each Pass 1 error fires a jump
|
||||||
|
// and resets `consecutive_outer_failures` to 0 inside the
|
||||||
|
// handler. So we can't accumulate "3" the old way — instead,
|
||||||
|
// verify the counter goes back to 0 after on_success too.
|
||||||
let mut ctx = ReadCtx::for_sweep(32);
|
let mut ctx = ReadCtx::for_sweep(32);
|
||||||
for _ in 0..3 {
|
handle_read_error(&medium_err(), &mut ctx);
|
||||||
handle_read_error(&medium_err(), &mut ctx);
|
// After fast-jump, consecutive_outer_failures already 0.
|
||||||
}
|
assert_eq!(ctx.consecutive_outer_failures, 0);
|
||||||
assert_eq!(ctx.consecutive_outer_failures, 3);
|
// on_success keeps it at 0 (defensive).
|
||||||
// An outer-success (bisecting=false) should reset the counter.
|
|
||||||
ctx.bisecting = false;
|
ctx.bisecting = false;
|
||||||
ctx.on_success();
|
ctx.on_success();
|
||||||
assert_eq!(ctx.consecutive_outer_failures, 0);
|
assert_eq!(ctx.consecutive_outer_failures, 0);
|
||||||
@@ -945,21 +981,43 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn both_passes_pause_on_failed_read_for_wedge_avoidance() {
|
fn pass_1_zone_entry_uses_long_cooldown() {
|
||||||
// 2026-05-11 reframe: a failed read is a failed read. Same
|
// 2026-05-11 wedge-prevention rewrite: Pass 1's FIRST error
|
||||||
// FAIL_PAUSE_SECS for both Pass 1 (sweep) and Pass N (patch).
|
// (zone entry) gets a 30 s ZONE_ENTRY_COOLDOWN_SECS pause +
|
||||||
// Pre-reframe Pass 1 had its own PASS_1_FAIL_PAUSE_SECS=5
|
// a 2 s POST_JUMP_EXTRA on top (since we're also jumping).
|
||||||
// and Pass N had POST_FAILURE_PAUSE_SECS=1 — the asymmetry
|
// The long pause prevents the retry cadence that triggers
|
||||||
// is gone. The wedge avoidance is a centralized policy now.
|
// firmware fast-fail. Subsequent errors in the same zone fall
|
||||||
for mut ctx in [ReadCtx::for_sweep(32), ReadCtx::for_patch(1)] {
|
// back to the standard 5 s FAIL_PAUSE_SECS.
|
||||||
let action = handle_read_error(&medium_err(), &mut ctx);
|
let mut ctx = ReadCtx::for_sweep(32);
|
||||||
let pause = match action {
|
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||||
ReadAction::SkipBlock { pause_secs } => pause_secs,
|
match action {
|
||||||
ReadAction::JumpAhead { pause_secs, .. } => pause_secs,
|
ReadAction::JumpAhead { pause_secs, .. } => {
|
||||||
ReadAction::Bisect => continue,
|
assert_eq!(
|
||||||
other => panic!("expected pausing action, got {other:?}"),
|
pause_secs,
|
||||||
};
|
ZONE_ENTRY_COOLDOWN_SECS + POST_JUMP_EXTRA_PAUSE_SECS,
|
||||||
assert_eq!(pause, FAIL_PAUSE_SECS);
|
"first-error pause should be 30 + 2 = 32 s"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
other => panic!("expected JumpAhead on first Pass 1 error, got {other:?}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pass_n_pauses_uniformly_on_failed_read() {
|
||||||
|
// Pass N (bisect_on_marginal=true) is exempt from the
|
||||||
|
// zone-entry long pause — its whole job is to retry single
|
||||||
|
// sectors on already-known-bad LBAs, and the 30 s pause every
|
||||||
|
// single-sector failure would multiply slow recovery
|
||||||
|
// pointlessly. Pass N keeps the standard 5 s FAIL_PAUSE_SECS.
|
||||||
|
let mut ctx = ReadCtx::for_patch(1);
|
||||||
|
let action = handle_read_error(&medium_err(), &mut ctx);
|
||||||
|
match action {
|
||||||
|
ReadAction::SkipBlock { pause_secs } => assert_eq!(pause_secs, FAIL_PAUSE_SECS),
|
||||||
|
ReadAction::JumpAhead { pause_secs, .. } => {
|
||||||
|
assert_eq!(pause_secs, FAIL_PAUSE_SECS + POST_JUMP_EXTRA_PAUSE_SECS)
|
||||||
|
}
|
||||||
|
ReadAction::Bisect => {}
|
||||||
|
other => panic!("expected pausing action, got {other:?}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user