v0.17.12: mapfile time-batched persistence — unblock NFS staging

Pre-0.17.12 every Mapfile::record() persisted the full mapfile via
tempfile-create + write + atomic-rename. On local LVM that's
microseconds; on NFS each rename is multiple RPCs through the
unraid user-share fuse layer, dragging a Black Mass UHD rip from
~11 MB/s on local to ~1.5 MB/s on NFS — the mapfile path alone burned
multiple seconds of wall time per real-world second of work.

Mapfile now batches the rename to once per second:

- record() always updates in-memory state and stats; only fires
  write_to_disk when last_flushed.elapsed() >= FLUSH_INTERVAL (1 s).
- New flush() API forces a persist; called by sweep_pipeline's
  consumer at end-of-sweep and by Disc::patch at end-of-patch,
  after the file's sync_all.
- Drop impl best-effort flushes so an early-return / unwind doesn't
  silently lose pending state.

Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are unaffected;
only the mapfile's authority over which sectors are already-good is at
risk, and a resume re-reads anything Pass 1 had already covered.

Measured on the BU40N test bed against Black Mass UHD inner zone:
- NFS staging: 1.5 MB/s → 16.48 MB/s (10.9× recovery)
- Local LVM staging: 11.09 MB/s → 11.83 MB/s (+6.7 % bonus)

Internal round_trip_load test now flushes before reading back from
disk. External patch / copy tests are unaffected: patch and
sweep_pipeline flush at completion before returning.
This commit is contained in:
MattJackson
2026-05-08 23:00:52 -07:00
parent 1ba3264747
commit 3a6c1aa5a3
5 changed files with 108 additions and 8 deletions
+39
View File
@@ -1,5 +1,44 @@
# Changelog
## 0.17.12 (2026-05-09)
### Mapfile time-batched persistence — unblock NFS staging
Pre-0.17.12 every `Mapfile::record()` wrote the entire mapfile to disk
via tempfile-create + buffered-write + atomic-rename. On local LVM
that's effectively free (page cache + microsecond-scale renames). On
NFS (autorip's intended staging path for centralised media) each
record() became three RPCs through the unraid user share's shfs-fuse
layer — measured end-to-end at multiple ms each. With ~170 record()
calls per second of sustained sweep, the mapfile path alone burned
multiple seconds of wall time per real-world second of work, dragging
the rip from ~11 MB/s on local to ~1.5 MB/s on NFS.
Fix: time-batch the persistence inside `Mapfile`.
- `record()` always updates in-memory state and stats (so callers'
`stats()` reads stay coherent in the same process).
- The `write_to_disk()` rename only fires when ≥ `FLUSH_INTERVAL`
(1 s) has elapsed since the last persist.
- New `flush()` method forces an out-of-band persist; called by
`sweep_pipeline`'s consumer at end-of-sweep and by `Disc::patch` at
end-of-patch, after the file's `sync_all()`.
- `Drop` impl best-effort flushes so an early-return / unwind doesn't
silently lose pending state.
Crash-safety changes from "lose at most one block" to "lose at most
1 s of recorded progress" — the ISO file's payload bytes are
unaffected; only the mapfile's authority over which sectors are
already-good is at risk, and a resume re-reads anything Pass 1 had
already covered. Acceptable for a 7× throughput recovery on the
target deployment.
The internal `round_trip_load` test now calls `flush()` before
`Mapfile::load` to read back what the in-memory state asserts.
External patch / copy tests are unaffected: `patch` and
`sweep_pipeline` flush at completion before returning, so any
`Mapfile::load` at the call-site sees fully persisted state.
## 0.17.11 (2026-05-09)
### Sweep producer/consumer split — overlap drive read with file write