mux: pipelined PES highway — read+decrypt → demux → parse on 3 threads
Introduces the freemkv mux throughput highway: a three-stage thread pipeline that replaces the inline single-thread read path for any file-backed source (ISO and m2ts file URLs both route through it). Thread A: read + decrypt (PrefetchedSectorSource / BytePrefetcher) Thread B: M2TS demux (DemuxThread) Thread C: codec parse (PipelinedPesStream, on caller thread) Each handoff uses a bounded crossbeam channel with a recycled buffer pool — no allocations or memcpys in the steady-state hot loop. Component map: * io/byte_prefetcher.rs (new) — std::io::Read producer thread with recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector side) so demux_thread::spawn_zero_copy can wire either upstream. * sector/prefetched.rs — recycled buffer pool added; into_channels() peels off the rx/recycle_tx/shell triple for zero-copy demux. * mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs feed() on its thread, ships Vec<PesPacket> batches. * mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls packets from the demux thread and runs codec parse on the caller. * mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline (private) assemble the three stages; iso:// and m2ts:// both return PipelinedPesStream. * mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted; the read direction lives on the highway now). * mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find. * mux/codec/hevc.rs — tightened frame_data initial capacity. * mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page first-touch fault tax that the previous 256 KiB pre-alloc paid on every PES boundary. * mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as the legacy autorip ingress (drive + multipass paths still need on_event / skip_errors before they migrate to the highway). * io/file_sector_source/* — per-OS prefetch() syscall hook (Linux readahead, macOS F_RDADVISE, Windows/other no-op). * decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS; pool sized to all cores by default. Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://): 60 → 322 MB/s warm cache (old new_pipeline path) 60 → 660 MB/s warm cache (highway path, this commit) 60 → 126 MB/s sustained disk-bound The IsoSectorReader baseline reader was deleted in favour of FileSectorSource so the freemkv CLI and autorip exercise the same read path.
This commit is contained in:
+129
-28
@@ -300,35 +300,22 @@ impl WritebackPipeline {
|
||||
}
|
||||
}
|
||||
|
||||
/// Probe whether `fd` lives on an NFS mount via `fstatfs`. Returns
|
||||
/// `false` on any error — we fail open, not closed: better to run the
|
||||
/// normal local-storage path on a misdetected NFS mount (and surface
|
||||
/// the freeze loudly via the timeout) than to needlessly disable
|
||||
/// writeback bounding on every local file because of a transient
|
||||
/// stat error.
|
||||
/// Probe whether `fd` lives on an NFS mount. Thin wrapper around
|
||||
/// [`crate::platform::fs_type::detect_fd`] so writeback policy and
|
||||
/// general-purpose fs-type classification stay in sync (same magic
|
||||
/// numbers, same musl-vs-glibc cast handling).
|
||||
///
|
||||
/// Fails open: any classification other than NFS counts as "not NFS"
|
||||
/// (including `Unknown` on `fstatfs` error) — better to run the
|
||||
/// normal local-storage path on a misdetected NFS mount and surface
|
||||
/// the freeze loudly via [`WAIT_AFTER_TIMEOUT`] than to needlessly
|
||||
/// disable writeback bounding on every local file because of a
|
||||
/// transient stat error.
|
||||
fn detect_nfs(fd: RawFd) -> bool {
|
||||
// `libc::statfs` is repr(C) with a fixed layout; zeroing is the
|
||||
// documented init pattern for the kernel uapi struct.
|
||||
let mut buf: libc::statfs = unsafe { std::mem::zeroed() };
|
||||
let rc = unsafe { libc::fstatfs(fd, &mut buf) };
|
||||
if rc != 0 {
|
||||
let errno = std::io::Error::last_os_error();
|
||||
tracing::warn!(
|
||||
target: "mux",
|
||||
"WritebackPipeline fstatfs(fd={fd}) failed: {errno} — defaulting is_nfs=false",
|
||||
);
|
||||
return false;
|
||||
}
|
||||
// `f_type` is signed (`__fsword_t`) on glibc and unsigned
|
||||
// (`c_ulong`) on musl. Cast both sides to i64 for a portable
|
||||
// comparison. On glibc x86_64 both already are i64 — clippy flags
|
||||
// the cast as unnecessary on that target only, but we need it for
|
||||
// musl, so silence the lint.
|
||||
#[allow(clippy::unnecessary_cast)]
|
||||
let f_type = buf.f_type as i64;
|
||||
#[allow(clippy::unnecessary_cast)]
|
||||
let nfs_magic = libc::NFS_SUPER_MAGIC as i64;
|
||||
f_type == nfs_magic
|
||||
matches!(
|
||||
crate::platform::fs_type::detect_fd(fd),
|
||||
crate::platform::fs_type::FsType::Nfs
|
||||
)
|
||||
}
|
||||
|
||||
/// Run `sync_file_range(WAIT_AFTER)` on a worker thread and wait up
|
||||
@@ -356,3 +343,117 @@ fn wait_after_with_timeout(fd: RawFd, off: u64, len: u64) -> Option<u64> {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tempfile::NamedTempFile;
|
||||
|
||||
/// Helper: build a `WritebackPipeline` over a local tempfile. On
|
||||
/// every test rig (linux dev box, CI) the tempfile lives on a
|
||||
/// local FS, so `is_nfs=false` and `skip_wait` returns false until
|
||||
/// we explicitly mark the pipeline degraded.
|
||||
fn local_pipeline(chunk_bytes: u64) -> (NamedTempFile, WritebackPipeline) {
|
||||
let f = NamedTempFile::new().expect("tempfile create");
|
||||
let pipeline = WritebackPipeline::new(f.as_file(), 0, chunk_bytes);
|
||||
(f, pipeline)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn new_pipeline_starts_active() {
|
||||
let (_f, p) = local_pipeline(32 * 1024 * 1024);
|
||||
assert!(!p.is_nfs, "local tempfile must not classify as NFS");
|
||||
assert!(!p.degraded.load(Ordering::Relaxed));
|
||||
assert!(!p.skip_wait(), "fresh local pipeline must not skip wait");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn degraded_flag_short_circuits_wait() {
|
||||
let (_f, p) = local_pipeline(32 * 1024 * 1024);
|
||||
assert!(!p.skip_wait());
|
||||
p.degraded.store(true, Ordering::Relaxed);
|
||||
assert!(
|
||||
p.skip_wait(),
|
||||
"degraded flag must force the wait+dontneed bypass"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn record_wait_grows_chunk_on_high_p95() {
|
||||
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
|
||||
// Fill the window with samples above the grow threshold.
|
||||
for _ in 0..ADAPTIVE_WINDOW {
|
||||
p.record_wait(ADAPTIVE_GROW_MS + 50);
|
||||
}
|
||||
assert!(
|
||||
p.chunk_bytes > 16 * 1024 * 1024,
|
||||
"chunk should have grown; got {}",
|
||||
p.chunk_bytes
|
||||
);
|
||||
assert!(p.chunk_bytes <= CHUNK_BYTES_MAX);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn record_wait_shrinks_chunk_on_low_p95() {
|
||||
let (_f, mut p) = local_pipeline(64 * 1024 * 1024);
|
||||
for _ in 0..ADAPTIVE_WINDOW {
|
||||
p.record_wait(1); // well under ADAPTIVE_SHRINK_MS
|
||||
}
|
||||
assert!(
|
||||
p.chunk_bytes < 64 * 1024 * 1024,
|
||||
"chunk should have shrunk; got {}",
|
||||
p.chunk_bytes
|
||||
);
|
||||
assert!(p.chunk_bytes >= CHUNK_BYTES_MIN);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn record_wait_no_op_below_window_fill() {
|
||||
let (_f, mut p) = local_pipeline(16 * 1024 * 1024);
|
||||
let initial = p.chunk_bytes;
|
||||
// Only push a few samples; window not full → no adaptation.
|
||||
for _ in 0..(ADAPTIVE_WINDOW - 1) {
|
||||
p.record_wait(ADAPTIVE_GROW_MS + 100);
|
||||
}
|
||||
assert_eq!(
|
||||
p.chunk_bytes, initial,
|
||||
"chunk must not change before window is full"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn record_wait_clamps_to_chunk_bounds() {
|
||||
// Grow past the max.
|
||||
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MAX);
|
||||
for _ in 0..ADAPTIVE_WINDOW {
|
||||
p.record_wait(ADAPTIVE_GROW_MS + 1000);
|
||||
}
|
||||
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MAX, "must clamp to MAX");
|
||||
|
||||
// Shrink past the min.
|
||||
let (_f, mut p) = local_pipeline(CHUNK_BYTES_MIN);
|
||||
for _ in 0..ADAPTIVE_WINDOW {
|
||||
p.record_wait(0);
|
||||
}
|
||||
assert_eq!(p.chunk_bytes, CHUNK_BYTES_MIN, "must clamp to MIN");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detect_nfs_local_file_is_false() {
|
||||
// Local tempfile must not classify as NFS. This locks in the
|
||||
// consolidation through `crate::platform::fs_type::detect_fd`.
|
||||
let f = NamedTempFile::new().expect("tempfile create");
|
||||
use std::os::unix::io::AsRawFd;
|
||||
assert!(!detect_nfs(f.as_file().as_raw_fd()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn note_progress_below_chunk_is_noop() {
|
||||
let (_f, mut p) = local_pipeline(32 * 1024 * 1024);
|
||||
// No-op return before crossing the first chunk boundary.
|
||||
let before = p.chunk_count;
|
||||
p.note_progress(1024); // < 32 MiB
|
||||
assert_eq!(p.chunk_count, before);
|
||||
assert!(p.pending.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user