v0.13.1: scsi::reset() bounded by wallclock timeout

Production incident: autorip's poll loop called scsi::reset() on a
wedged BU40N USB drive. The Linux SG_SCSI_RESET ioctl blocked
indefinitely (kernel SCSI subsystem waiting for a bus-wedged device to
ack a reset that will never come). Caller's poll loop hung for 60+
seconds before manual intervention.

scsi::reset() now spawns a detached worker for the platform-specific
reset and bounds the caller's wait via mpsc::recv_timeout. Default
30 s (DEFAULT_RESET_TIMEOUT_SECS); reset_with_timeout(device, dur)
exposes the bound for callers that want a different value. Returns
DeviceResetFailed on timeout. Worker thread keeps running until the
kernel eventually unblocks — leaks one OS thread per hard wedge, but
the daemon stays responsive instead of hanging forever.

Follow-up flagged for 0.13.2: USB-attached drives wedge at the USB
Mass Storage layer below SCSI; SG_SCSI_RESET doesn't help. A
scsi::usb_reset(path) using USBDEVFS_RESET is the proper escalation.
This commit is contained in:
MattJackson
2026-04-24 16:58:13 -07:00
parent d1f09439a5
commit 8af47e4c19
3 changed files with 105 additions and 3 deletions
+67 -2
View File
@@ -88,9 +88,74 @@ pub fn open(device: &Path) -> Result<Box<dyn ScsiTransport>> {
}
}
/// Reset a SCSI device to a known good state. Platform-specific.
/// On Linux: open/close fd cycle + TUR + SG_SCSI_RESET escalation.
/// Default upper bound on `scsi::reset()`. The platform-specific reset
/// sequence does ~15 s of bounded sleeps + ioctls in the happy path, so
/// 30 s is roughly 2× the worst-case happy time — long enough that a
/// healthy-but-slow drive isn't false-positively timed out, short enough
/// that a kernel-wedged ioctl doesn't take down the caller's poll loop
/// for minutes.
pub const DEFAULT_RESET_TIMEOUT_SECS: u64 = 30;
/// Reset a SCSI device to a known good state, with a hard wallclock
/// bound (`DEFAULT_RESET_TIMEOUT_SECS`). On Linux: open/close fd cycle +
/// TUR + SG_SCSI_RESET escalation.
///
/// **Why the timeout matters.** SG_SCSI_RESET is an ioctl that can block
/// indefinitely on a kernel-wedged USB target (the kernel waits for the
/// SCSI subsystem to ack the reset, which never comes from a dead-bus
/// device). Without an outer bound, the call hangs the caller's thread
/// forever — observed in production on a wedged BU40N where the autorip
/// poll loop sat in the ioctl for 60+ s. The bounded version returns
/// `DeviceResetFailed` after `DEFAULT_RESET_TIMEOUT_SECS`; the inner
/// thread keeps running until the kernel eventually unblocks it (we
/// can't cancel a Linux ioctl from userspace), so this leaks one OS
/// thread per wedge — acceptable cost for a daemon that recovers
/// instead of hanging.
pub fn reset(device: &Path) -> Result<()> {
reset_with_timeout(device, std::time::Duration::from_secs(DEFAULT_RESET_TIMEOUT_SECS))
}
/// Reset with a caller-specified timeout. See [`reset`] for the full
/// rationale on why an outer wallclock bound is required.
pub fn reset_with_timeout(device: &Path, timeout: std::time::Duration) -> Result<()> {
let device_owned = device.to_path_buf();
let (tx, rx) = std::sync::mpsc::channel();
// Detach a worker thread for the actual reset. We never `join` it —
// if the kernel ioctl is wedged, the join would block forever, which
// is the very thing we're protecting the caller from. The thread will
// exit on its own when the kernel eventually returns from the ioctl
// (or never, if the device is permanently dead — process exit cleans
// it up).
std::thread::Builder::new()
.name("scsi-reset".into())
.spawn(move || {
let r = reset_blocking(&device_owned);
let _ = tx.send(r);
})
.map_err(|_| Error::DeviceResetFailed {
path: device.display().to_string(),
})?;
match rx.recv_timeout(timeout) {
Ok(result) => result,
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => Err(Error::DeviceResetFailed {
path: device.display().to_string(),
}),
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
// The worker thread panicked before sending. Surface as a
// reset failure rather than a generic error.
Err(Error::DeviceResetFailed {
path: device.display().to_string(),
})
}
}
}
/// Inner reset — runs on the worker thread. May block indefinitely if
/// the kernel's SCSI subsystem is wedged. Callers must use the bounded
/// `reset()` wrapper above; this raw function isn't exposed.
fn reset_blocking(device: &Path) -> Result<()> {
#[cfg(target_os = "linux")]
{
linux::SgIoTransport::reset(device)