io/file_sector_source: throttle readahead for NFS bidirectional workloads

Empirical regression observed during 0.21.1 mux test on rip1/unraid-1:
historical 0.20.7 baseline averaged ~18 MB/s mux throughput; 0.21.1
dropped to ~7-8 MB/s flat. Same NFS source + destination, same disc.

Suspect: 32 MiB FileSectorSource readahead + posix_fadvise(SEQUENTIAL)
together saturate the TCP connection on read bursts, starving the
writer thread's concurrent NFS writes (mux reads the source ISO and
writes the MKV over the same connection).

- READAHEAD_BUF_BYTES: 32 MiB -> 4 MiB. Matches NFS rsize=1 MiB * 4
  round-trips per refill, interleaves cleanly with writes.
- linux/hint_sequential: now no-op. Kernel's default ~128 KiB
  readahead is what we want on NFS-backed ISOs (the dominant case).
  Per-OS file stays so we can re-enable a hint cleanly later if a
  different path benefits.
This commit is contained in:
MattJackson
2026-05-13 22:27:57 -07:00
parent e110e80e6e
commit b179846f5d
2 changed files with 22 additions and 12 deletions
+14 -11
View File
@@ -1,15 +1,18 @@
//! Linux: hint the kernel that this fd will be read sequentially so //! Linux: kernel readahead hint for the ISO file.
//! readahead widens. `posix_fadvise(POSIX_FADV_SEQUENTIAL)` is a hint, //!
//! not a guarantee — the kernel still owns the policy decision. //! Originally `POSIX_FADV_SEQUENTIAL` to widen the readahead window.
//! On NFS that turned out to cause aggressive multi-MB readahead bursts
//! that saturated the TCP connection and starved concurrent writes
//! during mux — observed empirically as a ~3× drop in mux throughput
//! on the rip1/unraid-1 setup (0.21.0 vs 0.20.7 baseline). The kernel's
//! default readahead (~128 KiB on Linux) interleaves more naturally
//! with the muxer's concurrent NFS writes, so we no longer issue any
//! hint here. The per-OS file stays so the convention is honoured and
//! we can re-enable a hint cleanly if a different storage path benefits.
use std::fs::File; use std::fs::File;
use std::os::unix::io::AsRawFd;
pub(super) fn hint_sequential(file: &File, _len_bytes: u64) { pub(super) fn hint_sequential(_file: &File, _len_bytes: u64) {
// Best-effort: return value ignored. A fadvise failure has no // No-op: see module-level comment. Kernel default readahead is
// user-observable consequence (reads still work, just without the // what we want on NFS-backed ISOs, which is the dominant case.
// widened readahead window).
unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, 0, libc::POSIX_FADV_SEQUENTIAL);
}
} }
+8 -1
View File
@@ -65,8 +65,15 @@ use crate::sector::SectorSource;
/// than per-sector pread, and large enough to coast through a typical /// than per-sector pread, and large enough to coast through a typical
/// NFS server commit blip. /// NFS server commit blip.
/// ///
/// 0.21.2: shrunk from 32 MiB → 4 MiB. On NFS-backed ISOs with
/// concurrent NFS writes (the mux phase), a 32 MiB refill bursts the
/// TCP connection hard enough to starve the writer thread, observed
/// empirically as a ~3× drop in sustained mux throughput on the
/// rip1/unraid-1 setup. 4 MiB matches `rsize=1 MiB` × 4 round-trips
/// and interleaves cleanly with writes.
///
/// Tweakable. Named const, not a magic number. /// Tweakable. Named const, not a magic number.
pub const READAHEAD_BUF_BYTES: usize = 32 * 1024 * 1024; pub const READAHEAD_BUF_BYTES: usize = 4 * 1024 * 1024;
const SECTOR_SIZE: usize = 2048; const SECTOR_SIZE: usize = 2048;
/// Sectors per refill: [`READAHEAD_BUF_BYTES`] / [`SECTOR_SIZE`]. The /// Sectors per refill: [`READAHEAD_BUF_BYTES`] / [`SECTOR_SIZE`]. The