v0.13.9: Disc::copy stall guard + SgIoTransport no-reopen-on-timeout

Fixes the silent Pass 1 hang observed on Dune 2 with v0.13.8 (drive
grinding through bad sectors at 0 KB/s, errs=0, no error surfaced).

Root cause: SgIoTransport::execute's reopen-after-poll-timeout opened
a fresh /dev/sg* fd on the main thread, which serialized against the
spawned close() of the old fd via the kernel's per-device state lock.
The userspace 1.5s timeout still fired, but the abandon-and-reopen
recovery itself blocked the main thread for as long as close() took.
Net: reads returned slowly, skip-forward fired on every iteration,
bytes_good never advanced.

- SgIoTransport::execute: on poll timeout, spawn close, set fd=-1,
  return Err. No reopen on the main thread. The transport is now
  invalidated until the consumer creates a fresh Drive.
- Disc::copy: add stall guard. CopyOptions.stall_secs (default 120s).
  If bytes_good doesn't advance for the threshold, break 'outer
  cleanly with complete=false, bytes_pending > 0 so Pass 2 retries
  pick up the NonTrimmed ranges with recovery=true 30s timeouts.
- New regression test: test_disc_copy_stall_detection_triggers_
  skip_forward in tests/integration_progress_and_halt.rs.
This commit is contained in:
2026-04-25 08:12:01 -07:00
parent af5a705972
commit 3f98671ae0
5 changed files with 266 additions and 16 deletions
+38
View File
@@ -1,5 +1,43 @@
# Changelog
## 0.13.9 (2026-04-25)
### Fix: Disc::copy silent stall + SgIoTransport reopen-after-timeout serialization
Two correlated fixes for a hang observed live on the LG BU40N during a
v0.13.8 rip of Dune: Part Two. At ~30 % progress through Pass 1
(disc → ISO), `bytes_good` froze for 10+ minutes with `errs=0`,
no error surfaced, drive not wedged.
Root cause: `SgIoTransport::execute` (linux.rs) attempted to recover from
a `poll()` timeout by spawning a background `close()` of the old fd and
opening a fresh `/dev/sg*` fd on the main thread. On Linux, opening the
SAME device while a prior fd is mid-close serializes via the kernel's
per-device state lock — so the fresh `open()` blocks for as long as the
close does (until the kernel completes the in-flight CDB). This undid
the userspace 1.5 s timeout: each timed-out read added 60+ s to the
next iteration. From `Disc::copy`'s perspective, reads kept returning
Err slowly, the skip-forward path advanced `pos` but never `bytes_good`.
Fixes:
- `SgIoTransport::execute` no longer reopens on timeout. Spawns the
close, sets `self.fd = -1`, returns Err immediately. Subsequent
calls fail with `DeviceNotFound` (already gated at line 248).
Caller (Drive) is invalidated until reopened. Pass 2's
`Disc::patch` would need a fresh Drive; that's a v0.14 follow-up.
- `Disc::copy` adds a stall guard. New `CopyOptions::stall_secs:
Option<u64>` (default 120 s). If `bytes_good` doesn't advance for
the threshold, breaks the outer loop with `complete: false,
bytes_pending > 0` so the caller's retry path picks up.
Tests: new `test_disc_copy_stall_detection_triggers_skip_forward` in
`tests/integration_progress_and_halt.rs` proves the guard fires within
the configured threshold.
Other:
- Cosmetic: warning text "rip thread did not drain within 35s" updated
to 60s (matches the v0.13.8 timeout bump).
## 0.13.8 (2026-04-25)
### Version sync — no functional changes
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "0.13.8"
version = "0.13.9"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
+36
View File
@@ -1286,6 +1286,9 @@ impl Disc {
let mut buf = vec![0u8; batch as usize * 2048];
let mut bytes_done = 0u64;
let mut halt_requested = false;
let stall_threshold = std::time::Duration::from_secs(opts.stall_secs.unwrap_or(120));
let mut last_good_advance = std::time::Instant::now();
let mut last_observed_good: u64 = 0;
// Iterate over not-yet-finished regions from the mapfile. We re-read the
// mapfile after each block because record() mutates the region list.
@@ -1315,6 +1318,30 @@ impl Disc {
break 'outer;
}
}
// Stall guard. The signal is "bytes_good (Finished sectors)
// hasn't advanced for stall_threshold." pos may still be
// advancing via the skip_on_error branch; that's not real
// progress because skip-forward only marks ranges
// NonTrimmed for Pass 2 to retry. If we go stall_threshold
// without ANY successful read, the drive is grinding
// unproductively (or kernel is silently stalling reads
// past their per-CDB timeout — observed live on Dell with
// SgIoTransport's reopen-after-timeout serializing
// against close). Bail Pass 1; Pass 2 (Disc::patch with
// recovery=true, 30 s timeouts) will retry the
// NonTrimmed ranges.
let cur_good = map.stats().bytes_good;
if cur_good != last_observed_good {
last_observed_good = cur_good;
last_good_advance = std::time::Instant::now();
} else if last_good_advance.elapsed() > stall_threshold {
// Stall: bail Pass 1. Return cleanly with
// bytes_pending > 0 and complete = false so the
// caller's retry path (Disc::patch with
// recovery=true, 30s timeouts) gets a shot at the
// NonTrimmed ranges.
break 'outer;
}
let block_bytes = (region_end - pos).min(batch as u64 * 2048);
let lba = (pos / 2048) as u32;
let count = (block_bytes / 2048) as u16;
@@ -1404,6 +1431,15 @@ pub struct CopyOptions<'a> {
pub skip_forward: bool,
pub on_progress: Option<&'a dyn Fn(u64, u64)>,
pub halt: Option<std::sync::Arc<std::sync::atomic::AtomicBool>>,
/// Wall-clock stall threshold in seconds. If the inner copy loop runs
/// this long without `pos` advancing, the current block is treated as
/// a read failure (skip-forward in `skip_on_error` mode, else
/// `Err(DiscRead)`). Defaults to 120 s. Defensive guard for
/// kernel-level hangs that bypass the per-CDB SCSI timeout — e.g. the
/// v0.13.8 case where SgIoTransport's reopen-after-timeout serialized
/// against the in-flight close, blocking the main thread for tens of
/// seconds per read.
pub stall_secs: Option<u64>,
}
/// Result of `Disc::copy`. `complete=true` means every byte reached a terminal
+26 -14
View File
@@ -321,26 +321,38 @@ impl ScsiTransport for SgIoTransport {
if pr <= 0 {
// Timeout (0) or fatal poll error (-1).
// Command is still pending in the kernel. Abandon this fd and
// open a fresh one. The old fd is closed in a background thread
// because close() blocks until the kernel completes/aborts the
// pending command.
// Command is still pending in the kernel. Abandon the fd by
// spawning a background close (which will block until the
// kernel completes/aborts the pending command), and mark
// this transport invalid by setting `self.fd = -1`.
//
// Why we no longer reopen on the main thread: opening the
// SAME /dev/sg* device while the prior fd is mid-close
// serializes via the kernel's per-device state lock, so
// `libc::open()` on the main thread blocks for the same
// duration that close() does — defeating the userspace
// timeout. Observed in v0.13.8 live test on Dune 2: each
// timed-out read added 60+ s to the next iteration of
// Disc::copy, leaving the rip stuck without surfacing an
// error or wedging the drive.
//
// Net effect of the fix: a single read timeout invalidates
// the SgIoTransport. The Drive is now "dead" until the
// consumer (autorip's rip thread) catches the failure and
// reopens. Disc::copy's `skip_on_error=true` path will see
// the Err and skip-forward, advancing pos, and the next
// read on this fd returns Err(DeviceNotFound) immediately —
// which Disc::copy continues to skip-forward through until
// the NonTried region is exhausted. Pass 1 then ends with
// bytes_pending > 0 and the rip thread reopens the Drive
// for Pass 2 (Disc::patch with recovery=true and 30 s
// timeouts).
let old_fd = self.fd;
self.fd = -1;
std::thread::spawn(move || {
unsafe { libc::close(old_fd) };
});
let c_path = Self::to_c_path(&self.device_path);
let new_fd = unsafe {
libc::open(
c_path.as_ptr() as *const libc::c_char,
libc::O_RDWR | libc::O_NONBLOCK | libc::O_CLOEXEC,
)
};
self.fd = if new_fd >= 0 { new_fd } else { -1 };
return Err(Error::ScsiError {
opcode: cdb[0],
status: 0xFF,
+165 -1
View File
@@ -9,8 +9,8 @@ use libfreemkv::{
SectorReader,
};
use std::io::Write;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
const SECTOR_SIZE: usize = 2048;
@@ -338,3 +338,167 @@ fn test_file_sector_reader_round_trip() {
assert_eq!(n, N_SECTORS * SECTOR_SIZE);
assert_eq!(all, data, "bulk read mismatch");
}
// ── 6. Disc::copy stall detection triggers skip-forward (TDD red) ─────────
//
// Regression guard for the Dell-host hang where `read_sectors` blocked inside
// a kernel-level USB stall and `Disc::copy` sat frozen for 10+ minutes with
// no progress and no error. The fix introduces `CopyOptions::stall_secs:
// Option<u64>` — when elapsed-since-last-`bytes_good`-advance exceeds the
// threshold, `Disc::copy` treats the current block as a read failure and
// triggers the skip-forward path so the rip can advance.
//
// THIS TEST IS EXPECTED TO FAIL UNTIL THE PARALLEL FIX LANDS.
// - Until `stall_secs` exists on `CopyOptions`, the test will not compile.
// - Once the field exists but the stall guard isn't wired, the spawned copy
// thread will never exit (test fails on the 5s join bound).
// - Once the guard is wired, copy returns within ~stall_secs with
// `complete=false` and `bytes_pending>0`.
/// Reader that returns Ok for sectors `< block_after`, then returns Err for
/// any sector `>= block_after` after a small per-call delay. Models the
/// realistic Dell-host symptom: reads keep returning Err (skip-forward fires)
/// but no `bytes_good` ever accrues; without a stall guard, Pass 1 grinds
/// silently for tens of minutes.
struct StallingSectorReader {
capacity: u32,
block_after: u32,
/// Per-call delay for sectors >= block_after (simulates slow reads).
err_delay_ms: u64,
release: Arc<AtomicBool>,
/// Retained so callers can release the reader; unused now that the
/// reader returns Err instead of blocking, but kept so the test's
/// existing release plumbing compiles.
park: Arc<(Mutex<()>, std::sync::Condvar)>,
}
impl StallingSectorReader {
fn new(capacity: u32, block_after: u32) -> Self {
Self {
capacity,
block_after,
err_delay_ms: 100,
release: Arc::new(AtomicBool::new(false)),
park: Arc::new((Mutex::new(()), std::sync::Condvar::new())),
}
}
fn release_handle(&self) -> (Arc<AtomicBool>, Arc<(Mutex<()>, std::sync::Condvar)>) {
(self.release.clone(), self.park.clone())
}
}
impl SectorReader for StallingSectorReader {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
if lba >= self.block_after {
// Realistic stall model: read takes err_delay_ms then returns
// Err. With skip_on_error+skip_forward, Disc::copy will keep
// skip-forwarding through this region — no bytes_good accrues.
// The stall guard fires when bytes_good is unchanged for
// stall_secs.
std::thread::sleep(Duration::from_millis(self.err_delay_ms));
return Err(libfreemkv::error::Error::DiscRead { sector: lba as u64 });
}
let bytes = count as usize * SECTOR_SIZE;
buf[..bytes].fill(0);
Ok(bytes)
}
fn capacity(&self) -> u32 {
self.capacity
}
}
#[test]
fn test_disc_copy_stall_detection_triggers_skip_forward() {
// 1024 sectors total. Reader serves the first 64 sectors instantly, then
// every later read blocks forever. With stall_secs=2, copy should bail
// out of the stalled block within ~2s and either skip forward or finish
// with bytes_pending > 0 / complete=false.
let capacity_sectors: u32 = 1024;
let block_after: u32 = 64;
let reader = StallingSectorReader::new(capacity_sectors, block_after);
let (release_flag, park) = reader.release_handle();
let mut reader = reader;
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().expect("tempfile create");
let iso_path = tmp.path().to_path_buf();
drop(tmp);
let iso_path_for_thread = iso_path.clone();
let join = std::thread::spawn(move || {
let opts = CopyOptions {
decrypt: false,
skip_on_error: true,
skip_forward: true,
// ASSUMPTION: parallel fix adds `pub stall_secs: Option<u64>` to
// CopyOptions. If the field name differs, update here.
stall_secs: Some(2),
..Default::default()
};
let t0 = Instant::now();
let res = disc.copy(&mut reader, &iso_path_for_thread, &opts);
(res, t0.elapsed())
});
// Bound the join to ~5s. With stall_secs=2 the copy should exit well
// within this window. If it doesn't, the stall guard isn't working.
let started = Instant::now();
let mut joined = None;
while started.elapsed() < Duration::from_millis(5000) {
if join.is_finished() {
joined = Some(join.join().expect("thread join"));
break;
}
std::thread::sleep(Duration::from_millis(50));
}
// Whether or not the join succeeded, release the parked reader thread so
// it can exit (its &mut reader is owned by the spawned thread; releasing
// lets that thread unwind cleanly).
release_flag.store(true, Ordering::Relaxed);
park.1.notify_all();
let (result, elapsed) = match joined {
Some(v) => v,
None => {
// Wait a bit longer for the thread to drain after release so we
// don't leave it dangling, then fail the test.
std::thread::sleep(Duration::from_millis(500));
panic!(
"Disc::copy did not return within 5s of stall_secs=2 — \
stall guard not wired (TDD red until fix lands)"
);
}
};
// Cleanup
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(libfreemkv::disc::mapfile_path_for(&iso_path));
let copy_result = result.expect("copy returns Ok with stall handling");
assert!(
elapsed < Duration::from_millis(5000),
"copy elapsed {elapsed:?} exceeded 5s bound (stall_secs=2)"
);
assert!(
copy_result.bytes_pending > 0,
"expected bytes_pending > 0 after stall-triggered skip; got {}",
copy_result.bytes_pending
);
assert!(
!copy_result.complete,
"expected complete=false after stall-triggered skip"
);
}