v0.13.1: scsi::reset() bounded by wallclock timeout
Production incident: autorip's poll loop called scsi::reset() on a wedged BU40N USB drive. The Linux SG_SCSI_RESET ioctl blocked indefinitely (kernel SCSI subsystem waiting for a bus-wedged device to ack a reset that will never come). Caller's poll loop hung for 60+ seconds before manual intervention. scsi::reset() now spawns a detached worker for the platform-specific reset and bounds the caller's wait via mpsc::recv_timeout. Default 30 s (DEFAULT_RESET_TIMEOUT_SECS); reset_with_timeout(device, dur) exposes the bound for callers that want a different value. Returns DeviceResetFailed on timeout. Worker thread keeps running until the kernel eventually unblocks — leaks one OS thread per hard wedge, but the daemon stays responsive instead of hanging forever. Follow-up flagged for 0.13.2: USB-attached drives wedge at the USB Mass Storage layer below SCSI; SG_SCSI_RESET doesn't help. A scsi::usb_reset(path) using USBDEVFS_RESET is the proper escalation.
This commit is contained in:
@@ -1,5 +1,42 @@
|
||||
# Changelog
|
||||
|
||||
## 0.13.1 (2026-04-24)
|
||||
|
||||
### `scsi::reset()` now has a hard wallclock timeout
|
||||
|
||||
Production incident on a wedged BU40N USB drive: autorip's poll loop
|
||||
called `scsi::reset()` and the call hung for 60+ seconds before the
|
||||
operator manually intervened. Root cause: the Linux `SG_SCSI_RESET`
|
||||
ioctl can block indefinitely when the kernel SCSI subsystem is waiting
|
||||
on a bus-wedged device that will never ack — there's no kernel-side
|
||||
timeout on this ioctl. Without an outer wallclock bound the caller's
|
||||
thread is stuck in the kernel until the device unwedges (which, for a
|
||||
permanently-dead USB target, may be never).
|
||||
|
||||
`scsi::reset()` is now a wrapper that runs the platform-specific reset
|
||||
on a detached worker thread and bounds the caller's wait via
|
||||
`mpsc::recv_timeout(DEFAULT_RESET_TIMEOUT_SECS)` (30 s). Returns
|
||||
`DeviceResetFailed` on timeout. The worker thread keeps running until
|
||||
the kernel eventually unblocks (we can't cancel a Linux ioctl from
|
||||
userspace) — this leaks one OS thread per hard wedge, an acceptable
|
||||
cost for a daemon that recovers vs. one that hangs.
|
||||
|
||||
- New `pub const DEFAULT_RESET_TIMEOUT_SECS: u64 = 30;`
|
||||
- New `pub fn reset_with_timeout(device, Duration) -> Result<()>` for
|
||||
callers that want a different bound.
|
||||
- Existing `pub fn reset(device) -> Result<()>` keeps the same
|
||||
signature; behaviour change is the timeout, not the API.
|
||||
|
||||
### Follow-up flagged
|
||||
|
||||
`SG_SCSI_RESET` only resets at the SCSI layer. For USB-attached drives
|
||||
(the BU40N case), the wedge is often in the USB Mass Storage layer
|
||||
*below* SCSI — `SG_SCSI_RESET` doesn't help. The proper escalation is
|
||||
`USBDEVFS_RESET` (the `usbreset.c` ioctl), which re-enumerates the
|
||||
device at the USB layer. Tracked for 0.13.2: a `scsi::usb_reset(path)`
|
||||
that resolves sg → USB device and issues `USBDEVFS_RESET`. That would
|
||||
have recovered tonight's BU40N without operator intervention.
|
||||
|
||||
## 0.13.0 (2026-04-24)
|
||||
|
||||
### Zero English in library — typed variants for every error path
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "libfreemkv"
|
||||
version = "0.13.0"
|
||||
version = "0.13.1"
|
||||
edition = "2024"
|
||||
rust-version = "1.86"
|
||||
license = "AGPL-3.0-only"
|
||||
|
||||
+67
-2
@@ -88,9 +88,74 @@ pub fn open(device: &Path) -> Result<Box<dyn ScsiTransport>> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Reset a SCSI device to a known good state. Platform-specific.
|
||||
/// On Linux: open/close fd cycle + TUR + SG_SCSI_RESET escalation.
|
||||
/// Default upper bound on `scsi::reset()`. The platform-specific reset
|
||||
/// sequence does ~15 s of bounded sleeps + ioctls in the happy path, so
|
||||
/// 30 s is roughly 2× the worst-case happy time — long enough that a
|
||||
/// healthy-but-slow drive isn't false-positively timed out, short enough
|
||||
/// that a kernel-wedged ioctl doesn't take down the caller's poll loop
|
||||
/// for minutes.
|
||||
pub const DEFAULT_RESET_TIMEOUT_SECS: u64 = 30;
|
||||
|
||||
/// Reset a SCSI device to a known good state, with a hard wallclock
|
||||
/// bound (`DEFAULT_RESET_TIMEOUT_SECS`). On Linux: open/close fd cycle +
|
||||
/// TUR + SG_SCSI_RESET escalation.
|
||||
///
|
||||
/// **Why the timeout matters.** SG_SCSI_RESET is an ioctl that can block
|
||||
/// indefinitely on a kernel-wedged USB target (the kernel waits for the
|
||||
/// SCSI subsystem to ack the reset, which never comes from a dead-bus
|
||||
/// device). Without an outer bound, the call hangs the caller's thread
|
||||
/// forever — observed in production on a wedged BU40N where the autorip
|
||||
/// poll loop sat in the ioctl for 60+ s. The bounded version returns
|
||||
/// `DeviceResetFailed` after `DEFAULT_RESET_TIMEOUT_SECS`; the inner
|
||||
/// thread keeps running until the kernel eventually unblocks it (we
|
||||
/// can't cancel a Linux ioctl from userspace), so this leaks one OS
|
||||
/// thread per wedge — acceptable cost for a daemon that recovers
|
||||
/// instead of hanging.
|
||||
pub fn reset(device: &Path) -> Result<()> {
|
||||
reset_with_timeout(device, std::time::Duration::from_secs(DEFAULT_RESET_TIMEOUT_SECS))
|
||||
}
|
||||
|
||||
/// Reset with a caller-specified timeout. See [`reset`] for the full
|
||||
/// rationale on why an outer wallclock bound is required.
|
||||
pub fn reset_with_timeout(device: &Path, timeout: std::time::Duration) -> Result<()> {
|
||||
let device_owned = device.to_path_buf();
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
|
||||
// Detach a worker thread for the actual reset. We never `join` it —
|
||||
// if the kernel ioctl is wedged, the join would block forever, which
|
||||
// is the very thing we're protecting the caller from. The thread will
|
||||
// exit on its own when the kernel eventually returns from the ioctl
|
||||
// (or never, if the device is permanently dead — process exit cleans
|
||||
// it up).
|
||||
std::thread::Builder::new()
|
||||
.name("scsi-reset".into())
|
||||
.spawn(move || {
|
||||
let r = reset_blocking(&device_owned);
|
||||
let _ = tx.send(r);
|
||||
})
|
||||
.map_err(|_| Error::DeviceResetFailed {
|
||||
path: device.display().to_string(),
|
||||
})?;
|
||||
|
||||
match rx.recv_timeout(timeout) {
|
||||
Ok(result) => result,
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => Err(Error::DeviceResetFailed {
|
||||
path: device.display().to_string(),
|
||||
}),
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
|
||||
// The worker thread panicked before sending. Surface as a
|
||||
// reset failure rather than a generic error.
|
||||
Err(Error::DeviceResetFailed {
|
||||
path: device.display().to_string(),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Inner reset — runs on the worker thread. May block indefinitely if
|
||||
/// the kernel's SCSI subsystem is wedged. Callers must use the bounded
|
||||
/// `reset()` wrapper above; this raw function isn't exposed.
|
||||
fn reset_blocking(device: &Path) -> Result<()> {
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
linux::SgIoTransport::reset(device)
|
||||
|
||||
Reference in New Issue
Block a user