mux: pipelined PES highway — read+decrypt → demux → parse on 3 threads
Introduces the freemkv mux throughput highway: a three-stage thread pipeline that replaces the inline single-thread read path for any file-backed source (ISO and m2ts file URLs both route through it). Thread A: read + decrypt (PrefetchedSectorSource / BytePrefetcher) Thread B: M2TS demux (DemuxThread) Thread C: codec parse (PipelinedPesStream, on caller thread) Each handoff uses a bounded crossbeam channel with a recycled buffer pool — no allocations or memcpys in the steady-state hot loop. Component map: * io/byte_prefetcher.rs (new) — std::io::Read producer thread with recycled Vec<u8> pool. Pairs with PrefetchedSectorSource (sector side) so demux_thread::spawn_zero_copy can wire either upstream. * sector/prefetched.rs — recycled buffer pool added; into_channels() peels off the rx/recycle_tx/shell triple for zero-copy demux. * mux/demux_thread.rs (new) — owns the TsDemuxer/PsDemuxer, runs feed() on its thread, ships Vec<PesPacket> batches. * mux/pipelined_stream.rs (new) — the read-side Stream impl. Pulls packets from the demux thread and runs codec parse on the caller. * mux/resolve.rs — build_iso_pipeline (public) / build_m2ts_pipeline (private) assemble the three stages; iso:// and m2ts:// both return PipelinedPesStream. * mux/m2ts.rs — collapsed to a write-only sink (Mode::Read deleted; the read direction lives on the highway now). * mux/codec/h264.rs — find_start_code uses memchr SIMD memmem::find. * mux/codec/hevc.rs — tightened frame_data initial capacity. * mux/ts.rs — boundary-packet handling avoids the per-batch 16 MiB remainder copy; PesAssembler starts at 16 KiB to dodge the 64-page first-touch fault tax that the previous 256 KiB pre-alloc paid on every PES boundary. * mux/disc.rs — gains DiscStream::new_pipeline + read_pipeline as the legacy autorip ingress (drive + multipass paths still need on_event / skip_errors before they migrate to the highway). * io/file_sector_source/* — per-OS prefetch() syscall hook (Linux readahead, macOS F_RDADVISE, Windows/other no-op). * decrypt.rs — FREEMKV_DECRYPT_THREADS renamed to FREEMKV_THREADS; pool sized to all cores by default. Measured on rip1 testbed (Civil War UHD, 62 GiB ISO → null://): 60 → 322 MB/s warm cache (old new_pipeline path) 60 → 660 MB/s warm cache (highway path, this commit) 60 → 126 MB/s sustained disk-bound The IsoSectorReader baseline reader was deleted in favour of FileSectorSource so the freemkv CLI and autorip exercise the same read path.
This commit is contained in:
+326
-8
@@ -1,7 +1,7 @@
|
||||
//! DiscStream — read any disc (physical drive or ISO file) → PES frames.
|
||||
//!
|
||||
//! One stream type for all disc sources. The source is a SectorSource —
|
||||
//! Drive (hardware) or IsoSectorReader (file). DiscStream doesn't care.
|
||||
//! Drive (hardware) or FileSectorSource (file). DiscStream doesn't care.
|
||||
//!
|
||||
//! Read-only. For disc→ISO (raw sector copy), use `Disc::copy()`.
|
||||
|
||||
@@ -149,12 +149,28 @@ pub struct DiscStream {
|
||||
// a percent without a separate API call.
|
||||
bytes_total_extents: u64,
|
||||
|
||||
// PES output
|
||||
// PES output. `ts_demuxer` and `ps_demuxer` are `None` when the
|
||||
// stream is in pipeline mode — the demux state lives inside a
|
||||
// [`super::demux_thread::DemuxThread`] and PesPackets arrive
|
||||
// through `demux_rx` already parsed.
|
||||
ts_demuxer: Option<super::ts::TsDemuxer>,
|
||||
ps_demuxer: Option<super::ps::PsDemuxer>,
|
||||
parsers: Vec<(u16, Box<dyn super::codec::CodecParser>)>,
|
||||
pending_frames: std::collections::VecDeque<crate::pes::PesFrame>,
|
||||
pid_to_track: Vec<(u16, usize)>,
|
||||
|
||||
// Pipeline mode: when `Some`, the read+decrypt+demux pipeline
|
||||
// runs on a dedicated thread; this stream's `read()` just pulls
|
||||
// PesPacket batches from `demux_rx` and runs codec parse on the
|
||||
// caller thread. See [`super::demux_thread`].
|
||||
//
|
||||
// `demux_thread` is kept solely so that `Drop` joins the worker
|
||||
// before this stream is dropped — direct reads happen through
|
||||
// `demux_rx`. The `allow(dead_code)` keeps the optimizer happy
|
||||
// since the field is only used at drop time.
|
||||
#[allow(dead_code)]
|
||||
demux_thread: Option<super::demux_thread::DemuxThread>,
|
||||
demux_rx: Option<crossbeam_channel::Receiver<super::demux_thread::DemuxBatch>>,
|
||||
}
|
||||
|
||||
impl DiscStream {
|
||||
@@ -235,6 +251,118 @@ impl DiscStream {
|
||||
parsers,
|
||||
pending_frames: std::collections::VecDeque::new(),
|
||||
pid_to_track,
|
||||
demux_thread: None,
|
||||
demux_rx: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Pipeline-mode constructor. Moves the read+decrypt+demux work
|
||||
/// onto a [`super::demux_thread::DemuxThread`] so the caller's
|
||||
/// `read()` thread only does codec parse + frame emission.
|
||||
///
|
||||
/// `reader` is a [`crate::sector::PrefetchedSectorSource`] — the
|
||||
/// prefetched producer thread already runs read+decrypt on its
|
||||
/// own thread; this constructor peels off its channels for the
|
||||
/// demux thread to consume in zero-copy mode (no buffer memcpy
|
||||
/// across thread boundary, recycled-pool of two buffers, no
|
||||
/// allocator activity in the hot loop).
|
||||
///
|
||||
/// Pipeline mode is the preferred wiring for ISO file mux on a
|
||||
/// multi-core host; it gives a ~2× consumer throughput in the
|
||||
/// `null://` benchmark vs the single-thread inline path.
|
||||
pub fn new_pipeline(
|
||||
reader: crate::sector::PrefetchedSectorSource,
|
||||
title: DiscTitle,
|
||||
decrypt_keys: crate::decrypt::DecryptKeys,
|
||||
batch_sectors: u16,
|
||||
content_format: crate::disc::ContentFormat,
|
||||
halt: Option<Halt>,
|
||||
) -> Self {
|
||||
let extents = title.extents.clone();
|
||||
let bytes_total_extents: u64 = extents.iter().map(|e| e.sector_count as u64 * 2048).sum();
|
||||
|
||||
let mut pids = Vec::new();
|
||||
let mut parsers = Vec::new();
|
||||
let mut pid_to_track = Vec::new();
|
||||
for (idx, s) in title.streams.iter().enumerate() {
|
||||
let (pid, codec) = match s {
|
||||
crate::disc::Stream::Video(v) => (v.pid, v.codec),
|
||||
crate::disc::Stream::Audio(a) => (a.pid, a.codec),
|
||||
crate::disc::Stream::Subtitle(s) => (s.pid, s.codec),
|
||||
};
|
||||
pids.push(pid);
|
||||
pid_to_track.push((pid, idx));
|
||||
parsers.push((pid, super::codec::parser_for_codec(codec, None)));
|
||||
}
|
||||
|
||||
let (ts, ps) = match content_format {
|
||||
crate::disc::ContentFormat::MpegPs => (None, Some(super::ps::PsDemuxer::new())),
|
||||
crate::disc::ContentFormat::BdTs => {
|
||||
let ts_pids: Vec<u16> = pids.clone();
|
||||
if ts_pids.is_empty() {
|
||||
(None, None)
|
||||
} else {
|
||||
(Some(super::ts::TsDemuxer::new(&ts_pids)), None)
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
let (prefetch_rx, recycle_tx, shell) = reader.into_channels();
|
||||
let (handle, rx) = super::demux_thread::DemuxThread::spawn_zero_copy(
|
||||
prefetch_rx,
|
||||
recycle_tx,
|
||||
shell,
|
||||
halt.clone(),
|
||||
ts,
|
||||
ps,
|
||||
);
|
||||
|
||||
// The DiscStream's own reader is a no-op pass-through — the
|
||||
// real reader lives inside the demux thread. We need *some*
|
||||
// Box<dyn SectorSource> to satisfy the field type; use a
|
||||
// tiny stub. fill_extents won't be called in pipeline mode.
|
||||
struct NullSource;
|
||||
impl SectorSource for NullSource {
|
||||
fn capacity_sectors(&self) -> u32 {
|
||||
0
|
||||
}
|
||||
fn read_sectors(
|
||||
&mut self,
|
||||
_: u32,
|
||||
_: u16,
|
||||
_: &mut [u8],
|
||||
_: bool,
|
||||
) -> crate::error::Result<usize> {
|
||||
Ok(0)
|
||||
}
|
||||
}
|
||||
let dummy: Box<dyn SectorSource> = Box::new(NullSource);
|
||||
|
||||
Self {
|
||||
reader: DecryptingSectorSource::new(dummy, crate::decrypt::DecryptKeys::None),
|
||||
title,
|
||||
disc: None,
|
||||
decrypt_keys,
|
||||
extents,
|
||||
current_extent: 0,
|
||||
current_offset: 0,
|
||||
read_buf: Vec::new(),
|
||||
buf_valid: 0,
|
||||
adaptive: AdaptiveBatch::new(batch_sectors),
|
||||
errors: 0,
|
||||
skip_errors: false,
|
||||
halt,
|
||||
event_fn: None,
|
||||
eof: false,
|
||||
bytes_read_total: 0,
|
||||
bytes_total_extents,
|
||||
ts_demuxer: None,
|
||||
ps_demuxer: None,
|
||||
parsers,
|
||||
pending_frames: std::collections::VecDeque::new(),
|
||||
pid_to_track,
|
||||
demux_thread: Some(handle),
|
||||
demux_rx: Some(rx),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -403,6 +531,153 @@ impl DiscStream {
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-stage profiling state — populated only when `FREEMKV_PROFILE`
|
||||
/// is set. Dumps a percentage breakdown to stderr every
|
||||
/// [`PROFILE_INTERVAL`]. Zero overhead in normal runs (Option check
|
||||
/// is the only added cost).
|
||||
struct StageProf {
|
||||
started: std::time::Instant,
|
||||
last_dump: std::time::Instant,
|
||||
fill_ns: u128,
|
||||
feed_ns: u128,
|
||||
consume_ns: u128,
|
||||
bytes_in: u64,
|
||||
}
|
||||
|
||||
const PROFILE_INTERVAL: std::time::Duration = std::time::Duration::from_secs(5);
|
||||
|
||||
thread_local! {
|
||||
static STAGE_PROF: std::cell::RefCell<Option<StageProf>> = const { std::cell::RefCell::new(None) };
|
||||
}
|
||||
|
||||
fn prof_active() -> bool {
|
||||
std::env::var_os("FREEMKV_PROFILE").is_some()
|
||||
}
|
||||
|
||||
fn prof_tick(stage: &str, ns: u128, bytes: u64) {
|
||||
STAGE_PROF.with(|cell| {
|
||||
let mut slot = cell.borrow_mut();
|
||||
if slot.is_none() {
|
||||
if !prof_active() {
|
||||
return;
|
||||
}
|
||||
let now = std::time::Instant::now();
|
||||
*slot = Some(StageProf {
|
||||
started: now,
|
||||
last_dump: now,
|
||||
fill_ns: 0,
|
||||
feed_ns: 0,
|
||||
consume_ns: 0,
|
||||
bytes_in: 0,
|
||||
});
|
||||
}
|
||||
let p = slot.as_mut().unwrap();
|
||||
match stage {
|
||||
"fill" => p.fill_ns += ns,
|
||||
"feed" => p.feed_ns += ns,
|
||||
"consume" => p.consume_ns += ns,
|
||||
_ => {}
|
||||
}
|
||||
p.bytes_in += bytes;
|
||||
let now = std::time::Instant::now();
|
||||
if now.duration_since(p.last_dump) < PROFILE_INTERVAL {
|
||||
return;
|
||||
}
|
||||
let elapsed_ms = now.duration_since(p.started).as_millis().max(1);
|
||||
let fill_pct = p.fill_ns / 10_000 / elapsed_ms;
|
||||
let feed_pct = p.feed_ns / 10_000 / elapsed_ms;
|
||||
let consume_pct = p.consume_ns / 10_000 / elapsed_ms;
|
||||
let mbps = p.bytes_in as u128 * 1000 / 1_000_000 / elapsed_ms;
|
||||
eprintln!(
|
||||
"[profile] elapsed={}ms in={}MB/s fill={}% feed={}% consume={}%",
|
||||
elapsed_ms, mbps, fill_pct, feed_pct, consume_pct,
|
||||
);
|
||||
p.last_dump = now;
|
||||
});
|
||||
}
|
||||
|
||||
impl DiscStream {
|
||||
/// Pipeline-mode `read()` helper: pull one PesPacket batch from
|
||||
/// the demux thread, run codec parse on each PES, enqueue the
|
||||
/// resulting PesFrames, return the first one.
|
||||
fn read_pipeline(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
|
||||
use super::demux_thread::DemuxBatch;
|
||||
let rx = self.demux_rx.as_ref().expect("read_pipeline without rx");
|
||||
match rx.recv() {
|
||||
Ok(DemuxBatch::Ts(packets)) => {
|
||||
let skip_parse = std::env::var_os("FREEMKV_SKIP_PARSE").is_some();
|
||||
for pes in packets {
|
||||
if let Some((_, track)) = self
|
||||
.pid_to_track
|
||||
.iter()
|
||||
.find(|(pid, _)| *pid == pes.pid)
|
||||
.copied()
|
||||
{
|
||||
if skip_parse {
|
||||
self.pending_frames.push_back(crate::pes::PesFrame {
|
||||
track,
|
||||
pts: pes.pts.map(super::codec::pts_to_ns).unwrap_or(0),
|
||||
keyframe: false,
|
||||
data: pes.data,
|
||||
});
|
||||
} else if let Some((_, parser)) =
|
||||
self.parsers.iter_mut().find(|(pid, _)| *pid == pes.pid)
|
||||
{
|
||||
for frame in parser.parse(&pes) {
|
||||
self.pending_frames.push_back(
|
||||
crate::pes::PesFrame::from_codec_frame(track, frame),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(self.pending_frames.pop_front())
|
||||
}
|
||||
Ok(DemuxBatch::Ps(packets)) => {
|
||||
for ps in packets {
|
||||
let track = match ps.stream_id {
|
||||
0xE0..=0xEF => 0,
|
||||
0xC0..=0xDF => 1,
|
||||
0xBD => ps
|
||||
.sub_stream_id
|
||||
.map(|s| (s & 0x1F) as usize + 1)
|
||||
.unwrap_or(1),
|
||||
_ => continue,
|
||||
};
|
||||
if track >= self.title.streams.len() {
|
||||
continue;
|
||||
}
|
||||
let pid = self
|
||||
.pid_to_track
|
||||
.iter()
|
||||
.find(|(_, idx)| *idx == track)
|
||||
.map(|(p, _)| *p)
|
||||
.unwrap_or(0);
|
||||
let pes = super::ts::PesPacket {
|
||||
pid,
|
||||
pts: ps.pts.map(|p| p as i64),
|
||||
dts: ps.dts.map(|d| d as i64),
|
||||
data: ps.data,
|
||||
};
|
||||
if let Some((_, parser)) = self.parsers.iter_mut().find(|(p, _)| *p == pid) {
|
||||
for frame in parser.parse(&pes) {
|
||||
self.pending_frames
|
||||
.push_back(crate::pes::PesFrame::from_codec_frame(track, frame));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(self.pending_frames.pop_front())
|
||||
}
|
||||
Ok(DemuxBatch::Err(e)) => Err(e),
|
||||
Err(_) => {
|
||||
// Channel closed → demux thread finished. EOF.
|
||||
self.eof = true;
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl crate::pes::Stream for DiscStream {
|
||||
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
|
||||
if let Some(frame) = self.pending_frames.pop_front() {
|
||||
@@ -413,7 +688,21 @@ impl crate::pes::Stream for DiscStream {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
// Pipeline mode: read+decrypt+demux all happen on the demux
|
||||
// thread. Pull a batch of PesPackets and run codec parse on
|
||||
// this thread; loop until we have at least one frame OR EOF.
|
||||
if self.demux_rx.is_some() {
|
||||
loop {
|
||||
match self.read_pipeline()? {
|
||||
Some(f) => return Ok(Some(f)),
|
||||
None if self.eof => return Ok(None),
|
||||
None => continue,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
loop {
|
||||
let t0 = std::time::Instant::now();
|
||||
if !self.fill_extents()? {
|
||||
self.eof = true;
|
||||
// Flush demuxer — last PES packet may still be in the assembler
|
||||
@@ -475,6 +764,8 @@ impl crate::pes::Stream for DiscStream {
|
||||
}
|
||||
|
||||
let bytes = self.buf_valid;
|
||||
let t1 = std::time::Instant::now();
|
||||
prof_tick("fill", t1.duration_since(t0).as_nanos(), bytes as u64);
|
||||
// Plaintext: the wrapped reader (DecryptingSectorSource)
|
||||
// applied AACS / CSS in-place during fill_extents'
|
||||
// read_sectors call. The pre-0.18 inline decrypt step
|
||||
@@ -482,21 +773,41 @@ impl crate::pes::Stream for DiscStream {
|
||||
|
||||
if let Some(ref mut demuxer) = self.ts_demuxer {
|
||||
let packets = demuxer.feed(&self.read_buf[..bytes]);
|
||||
for pes in &packets {
|
||||
if let Some((_, track)) =
|
||||
self.pid_to_track.iter().find(|(pid, _)| *pid == pes.pid)
|
||||
let t2 = std::time::Instant::now();
|
||||
prof_tick("feed", t2.duration_since(t1).as_nanos(), 0);
|
||||
let skip_parse = std::env::var_os("FREEMKV_SKIP_PARSE").is_some();
|
||||
for pes in packets {
|
||||
if let Some((_, track)) = self
|
||||
.pid_to_track
|
||||
.iter()
|
||||
.find(|(pid, _)| *pid == pes.pid)
|
||||
.copied()
|
||||
{
|
||||
if let Some((_, parser)) =
|
||||
if skip_parse {
|
||||
// Profiling escape hatch — bypass the codec
|
||||
// parser and pass the raw PES bytes straight
|
||||
// through as a single PesFrame. Lets us
|
||||
// attribute consumer-thread time to
|
||||
// "demux + framing" vs "codec parse".
|
||||
self.pending_frames.push_back(crate::pes::PesFrame {
|
||||
track,
|
||||
pts: pes.pts.map(super::codec::pts_to_ns).unwrap_or(0),
|
||||
keyframe: false,
|
||||
data: pes.data,
|
||||
});
|
||||
} else if let Some((_, parser)) =
|
||||
self.parsers.iter_mut().find(|(pid, _)| *pid == pes.pid)
|
||||
{
|
||||
for frame in parser.parse(pes) {
|
||||
for frame in parser.parse(&pes) {
|
||||
self.pending_frames.push_back(
|
||||
crate::pes::PesFrame::from_codec_frame(*track, frame),
|
||||
crate::pes::PesFrame::from_codec_frame(track, frame),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let t3 = std::time::Instant::now();
|
||||
prof_tick("consume", t3.duration_since(t2).as_nanos(), 0);
|
||||
} else if let Some(ref mut demuxer) = self.ps_demuxer {
|
||||
let packets = demuxer.feed(&self.read_buf[..bytes]);
|
||||
for ps in &packets {
|
||||
@@ -570,6 +881,13 @@ impl crate::pes::Stream for DiscStream {
|
||||
}
|
||||
|
||||
fn headers_ready(&self) -> bool {
|
||||
// FREEMKV_SKIP_PARSE bypasses codec parsers entirely for
|
||||
// bottleneck profiling, so codec_private is never populated.
|
||||
// Pretend headers are ready immediately in that mode so the
|
||||
// CLI loop doesn't hang waiting for them.
|
||||
if std::env::var_os("FREEMKV_SKIP_PARSE").is_some() {
|
||||
return true;
|
||||
}
|
||||
for (idx, s) in self.title.streams.iter().enumerate() {
|
||||
if let crate::disc::Stream::Video(v) = s {
|
||||
if !v.secondary && self.codec_private(idx).is_none() {
|
||||
|
||||
Reference in New Issue
Block a user