v0.17.10: bounded-cache writeback pipeline for big sequential writes

Pass 1 sweep speed on a healthy disc previously dipped from ~15 MB/s
to ~1 MB/s every ~30 s on a host with default Linux dirty-page
settings. Empirical cause: the kernel's vm.dirty_ratio (~20% of RAM)
lets hundreds of MB of dirty pages accumulate, then bursts a flush at
99% disk utilisation that blocks app writes for ~1 s. Confirmed on
the BU40N test bed — dirty pages grew 112 → 563 MB between bursts;
lowering vm.dirty_bytes to 64 MB at the host sysctl level eliminated
the dips. Shipping the equivalent inside libfreemkv so users do not
need to tune the host kernel.

- New crate::io::Writer: drop-in File wrapper (impl Write + Seek).
  Wraps a per-platform WritebackPipeline that on Linux schedules
  sync_file_range(WRITE) + lagging sync_file_range(WAIT_AFTER) +
  posix_fadvise(DONTNEED) in 32 MB chunks, bounding dirty cache at
  ~64 MB. macOS and Windows ship a no-op stub.
- Disc::sweep wraps its output File in Writer. Loop body unchanged.
- Module is purpose-built so any large sequential output (patch,
  mux) can adopt the same wrapper as a one-line change later.
This commit is contained in:
2026-05-08 19:54:25 -07:00
parent 75f6df529c
commit 1085eb1e39
9 changed files with 271 additions and 2 deletions
+16
View File
@@ -0,0 +1,16 @@
//! File I/O helpers that bound kernel cache pressure on big writes.
//!
//! `Writer` is a drop-in wrapper around `std::fs::File` for any call
//! site that performs large sequential writes (sweep, mux, etc.). It
//! implements `Write` and `Seek` so existing code paths can swap
//! `File` for `Writer` with no body changes. Internally it drives a
//! `WritebackPipeline` that, on Linux, drains dirty pages continuously
//! at 32 MB granularity to avoid the kernel's accumulate-then-burst
//! flush behaviour. macOS and Windows use a no-op pipeline — their
//! default cache policies have not been shown to exhibit the same
//! pathology for this access pattern.
mod writeback;
mod writer;
pub(crate) use writer::Writer;