io/writeback/linux: drop is_nfs skip — bounded cache works on every medium

The WritebackPipeline's WAIT_AFTER + posix_fadvise(DONTNEED) dance
keeps dirty pages bounded at ~2 × chunk_bytes by waiting for each
chunk's writeback to commit before issuing the DONTNEED hint to drop
it from cache. The original 0.18-era design unconditionally skipped
this on NFS on the premise that "NFS clients have their own buffering
and commit semantics that handle dirty-page bounds without us forcing
the issue."

Empirically wrong. On unraid-1 NFS the kernel client buffers dirty
pages up to vm.dirty_ratio (default 20% of RAM = ~6.6 GB on the rip1
host) before the kernel forces writeback and throttles app writes.
Result on 0.21.11 mux measured 2026-05-15: mux throughput cycled
between ~45 MB/s (cache absorbing) and ~7 MB/s (cache draining under
throttle) on a ~100 s period — exactly the burst-flush pathology this
pipeline was built to fix, but disabled on the medium it actually
runs on. /proc/meminfo Dirty: column climbed lockstep with mux
write rate during the slow half of every cycle, confirming the cause.

The original safety concern — `sync_file_range(WAIT_AFTER)` hanging
indefinitely on a wedged NFS server — is already handled by
`wait_after_with_timeout`'s `bounded_syscall` wrapper (30 s deadline).
If a real WAIT_AFTER call exceeds the deadline the pipeline flips to
the `degraded` state and skips WAIT_AFTER + DONTNEED for the rest of
its life — same effect as the old NFS branch, but only triggered when
something is genuinely broken rather than as a blanket exception.

This change is medium-agnostic: every medium goes through the same
path now, every medium gets the same safety net, and the
ADAPTIVE_WINDOW chunk-size autotuner (lines 226-255 — measures p95 of
WAIT_AFTER and resizes between 4 MiB and 256 MiB) finally activates
on NFS where previously it was dead code. Slow medium auto-grows
chunks to amortise per-chunk overhead; fast medium auto-shrinks to
keep cache pressure tight; nothing in the code special-cases the
filesystem type.

`is_nfs` is still detected (for logging + observability) but no
longer keys `skip_wait`. Module doc + startup log line updated to
match.
This commit is contained in:
MattJackson
2026-05-15 09:55:38 -07:00
parent d8f27cdee0
commit dffee56102
+25 -18
View File
@@ -22,21 +22,24 @@
//! fast storage (NVMe) sees smaller chunks to keep cache pressure //! fast storage (NVMe) sees smaller chunks to keep cache pressure
//! tight. Bounds: [4 MiB, 256 MiB]. //! tight. Bounds: [4 MiB, 256 MiB].
//! //!
//! ## NFS escape hatch //! ## NFS — same path, same safety net
//! //!
//! `sync_file_range(WAIT_AFTER)` on an NFS-mounted file can block //! Earlier revisions of this code unconditionally skipped WAIT_AFTER +
//! indefinitely waiting for the server's commit ack. If the server //! `posix_fadvise(DONTNEED)` on NFS-mounted files, on the premise that
//! never acks (network partition, server-side hang, slow commit), the //! "NFS clients have their own buffering and commit semantics that
//! syscall never returns and the consumer thread is stuck inside the //! handle dirty-page bounds without us forcing the issue." That premise
//! kernel — `/api/stop` can't reach it because halt is cooperative. //! was empirically wrong: on Linux NFS clients dirty pages still
//! accumulate up to `vm.dirty_ratio` (default 20% of RAM, ~6.6 GB on a
//! 32 GB box) before the kernel forces writeback and throttles app
//! writes. Mux throughput on NFS therefore cycled between ~50 MB/s
//! (cache absorbing) and ~10 MB/s (cache draining under throttle) on
//! a ~100 s period — exactly the pathology this pipeline was built to
//! fix, but disabled on the medium where it actually mattered.
//! //!
//! When `fstatfs` reports the file lives on an NFS mount //! `is_nfs` is still detected (for logging and observability) but
//! (`f_type == NFS_SUPER_MAGIC`), the pipeline skips the WAIT_AFTER + //! `skip_wait` no longer keys off it. WAIT_AFTER + DONTNEED run on NFS
//! `posix_fadvise(DONTNEED)` dance entirely. NFS clients have their //! exactly as on local storage. The safety net described below
//! own buffering and commit semantics that handle dirty-page bounds //! (`WAIT_AFTER_TIMEOUT`) catches the original NFS-hang concern.
//! without us forcing the issue. The async `SYNC_FILE_RANGE_WRITE`
//! kickoff still runs (non-blocking by spec) so writeback still gets
//! a nudge.
//! //!
//! ## Defence in depth: WAIT_AFTER timeout //! ## Defence in depth: WAIT_AFTER timeout
//! //!
@@ -113,8 +116,8 @@ impl WritebackPipeline {
let is_nfs = detect_nfs(fd); let is_nfs = detect_nfs(fd);
tracing::info!( tracing::info!(
target: "mux", target: "mux",
"WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy={}", "WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy=wait+dontneed (falls back to skip if WAIT_AFTER timeouts past {}s)",
if is_nfs { "nfs-skip-wait" } else { "wait+dontneed" } WAIT_AFTER_TIMEOUT.as_secs(),
); );
Self { Self {
fd, fd,
@@ -129,11 +132,15 @@ impl WritebackPipeline {
} }
/// True if we should bypass the WAIT_AFTER + DONTNEED finalisation /// True if we should bypass the WAIT_AFTER + DONTNEED finalisation
/// step. NFS always bypasses; local storage bypasses once the /// step. The pipeline starts in the normal path on every medium
/// pipeline has flipped to degraded after a WAIT_AFTER timeout. /// (NFS, local, etc.) and flips here only if a real
/// `WAIT_AFTER` call exceeds [`WAIT_AFTER_TIMEOUT`] — at which
/// point we conclude this particular FS/server combination cannot
/// safely service WAIT_AFTER and fall back to skip mode for the
/// rest of the pipeline's life. See module-level comment.
#[inline] #[inline]
fn skip_wait(&self) -> bool { fn skip_wait(&self) -> bool {
self.is_nfs || self.degraded.load(Ordering::Relaxed) self.degraded.load(Ordering::Relaxed)
} }
/// Caller advanced the file position to `pos`. If a chunk boundary /// Caller advanced the file position to `pos`. If a chunk boundary