Compare commits

...
28 Commits
Author SHA1 Message Date
MattJackson 5b98c13e47 v0.20.8: bump version 2026-05-13 19:18:51 -07:00
MattJackson 8f8f1a62a2 io+disc: bundle 0.20.8 dev work
- io/pipeline.rs: add send_with_halt + finish_with_halt for cooperative
  halt during blocking producer-consumer handoffs; 5 new tests
- disc/patch.rs: split Disc::patch body (1168 -> 316 LOC) into named
  helpers (compute_initial_state, prime_cache, check_range_watchdog,
  handle_skip_limit, compute_damage_skip, handle_read_success,
  handle_read_failure, report_patch_progress, build_outcome) with
  PatchLoopState / RangeFrame structs; references shared
  PATCH_DAMAGE_THRESHOLD_PCT constant
- disc/read_error.rs: add pub const PATCH_DAMAGE_THRESHOLD_PCT = 6;
  ReadCtx::for_patch() now references the shared constant (was a
  latent 12 / 6 inconsistency)
- tests/passn_handler_ab.rs: 8-profile A/B fixture locking current
  patch-side recovery behavior (clean / all-medium / alternating /
  edge-bad-good-middle / single-bad / deep-pit / medium-then-good /
  batch-fail). Goldens captured pre-unification; will catch any
  future refactor that breaks the size-aware skip cap.
2026-05-13 19:15:48 -07:00
MattJackson e3cfd27942 v0.20.7: version bump for unified release (no source changes)
The 0.20.7 work lives downstream in autorip — process-level safety net
(hard watchdog 5-min mux escalation + std::process::exit, restart-loop
counter + auto-quarantine after 3 attempts, partial-state preservation,
failure_reason surfaced in /api/state).

Bumping libfreemkv to 0.20.7 keeps the 4 crates on the unified release
train per project docs.
2026-05-13 14:29:40 -07:00
MattJackson aa12bdad62 v0.20.6: io::bounded — halt-safe wrapper for blocking syscalls
Generalizes 0.20.5's hand-written wait_after_with_timeout into a
reusable primitive. After this change, every blocking syscall in the
recovery + mux paths is wrapped, so cooperative Halt has bounded
~250 ms latency reach even into kernel-owned thread states.

New module src/io/bounded.rs:
- BoundedError { Halted, Timeout, WorkerLost }
- bounded_syscall<F, R>(halt: Option<&Halt>, timeout, op) -> Result<R, BoundedError>
- Worker thread runs op; main thread recv_timeouts on a rendezvous
  channel in 250 ms slices, polling halt between slices.
- Worker is intentionally leaked on timeout/halt — kernel reaps when
  the syscall finally returns or at process exit. Calling thread is
  NEVER trapped inside a kernel call.
- 6 unit tests cover the happy path + each error variant.

Refactored callsites:
- src/io/writeback/linux.rs::wait_after_with_timeout now delegates
  to bounded_syscall. ~30 LOC of duplicated channel/thread plumbing
  deleted. Same semantics, cleaner.
- src/io/writeback_file.rs::WritebackFile::sync_all now wraps the
  final libc::fsync(fd) with bounded_syscall (60 s deadline). On
  timeout: log error at target=mux and return Ok — kernel will flush
  on close, best-effort but bounded. Covers FileSectorSink::finish,
  PatchSink::close, SweepSink::close, and the mux MKV finalize path
  (they all sync through WritebackFile).

What still hangs (deliberately not wrapped — too hot a path):
- File::write itself. Per-frame write on a wedged NFS could still
  block; but back-pressure from a stuck consumer means the producer
  notices within seconds, not minutes — different failure mode than
  the WAIT_AFTER hang 0.20.5/0.20.6 fix.
2026-05-13 14:22:24 -07:00
MattJackson ef3895cdc5 v0.20.5: NFS-aware writeback + bounded sync_file_range timeout
Targets the recurring mux hang on NFS dest where the consumer thread
sits indefinitely inside libc::sync_file_range(SYNC_FILE_RANGE_WAIT_AFTER)
because the NFS server never returns a commit ack. The whole rip
wedges; halt is cooperative and can't reach inside a kernel syscall.

A. NFS detection at WritebackPipeline construction (fstatfs f_type ==
NFS_SUPER_MAGIC 0x6969). When NFS:
- Skip SYNC_FILE_RANGE_WAIT_AFTER entirely.
- Skip posix_fadvise(DONTNEED) — NFS client handles its own buffering.
- Still issue async SYNC_FILE_RANGE_WRITE (harmless hint).
Cannot hang on a syscall not made. fstatfs failure fails open (assume
local). Logged at info on construction so operators see which strategy
is active. The whole hang vector is removed for NFS deployments.

B. Hard timeout on WAIT_AFTER for non-NFS (defense in depth, since
even a degraded local disk could in principle hang the syscall).
Each WAIT_AFTER runs on a worker thread; main thread waits on a
sync_channel rendezvous with 30s deadline. On timeout: log error,
set per-pipeline 'degraded' Arc<AtomicBool>, downgrade to NFS-style
skip for the rest of the pipeline's life. Worker thread leaks
intentionally — it'll unwind when the syscall eventually returns or
the process exits. Converts indefinite freeze into 'log loud +
downgrade + keep ripping'.

C. Diagnostic logging for the 73%-of-this-movie reproduction:
- WritebackFile::seek logs every non-trivial seek (from, to, signed
  delta) at target=mux so we can see if MkvMuxer seeks back before
  a stall.
- WritebackPipeline::finalize logs the chunk being finalised before
  any WAIT_AFTER call, so a hung chunk is identifiable by offset.

No new dependencies. macOS / Windows noop stubs unchanged. Net
+198 LOC libfreemkv (mostly writeback/linux.rs).
2026-05-13 14:09:55 -07:00
MattJackson 2dcf969ac8 v0.20.4: mux performance + observability — universal across storage
Four targeted changes to maximize mux throughput regardless of storage
backend (local SSD, local HDD, NFS, network share) and surface enough
log data to diagnose 'mux slow' reports without a re-rip:

1. POSIX_FADV_SEQUENTIAL on FileSectorSource::open (Linux only).
   Widens the kernel readahead window for sequential ISO reads. One
   syscall at open, free on every storage type.

2. POSIX_FADV_DONTNEED on the ISO read side after every 32 MiB chunk.
   Mirrors the writeback DONTNEED that already runs on the write
   side. Keeps the read-side page cache bounded during multi-GB ISO
   reads — eliminates the OOM-pressure / eviction-storm risk on
   long mux runs. Linux only; per-drop trace at target="mux".

3. WritebackFile::create_with_size_hint(path, size_bytes) calls
   fallocate(FALLOC_FL_KEEP_SIZE) on Linux to pre-reserve extents
   for the output. Reported file size stays 0 (writes grow it
   naturally) but the on-disk extent allocation is contiguous —
   reduces extent fragmentation for big sequential muxes. Wired
   into mkv:// and m2ts:// output paths via DiscTitle::size_bytes.
   No-op on macOS/Windows; old create() kept with #[allow(dead_code)]
   for callers without a size hint.

4. Adaptive WRITEBACK_CHUNK_BYTES in the Linux writeback pipeline.
   Tracks sync_file_range(WAIT_AFTER) elapsed_ms in a rolling
   16-sample window. p95 > 200 ms → double chunk size (cap 256 MiB).
   p95 < 20 ms → halve (floor 4 MiB). One algorithm, both
   fast-storage (small chunks, responsive) and slow-storage (big
   chunks, fewer commit round-trips) optimized. Per-chunk trace +
   per-32-chunk debug snapshot + info-on-resize so an operator can
   see where the autoscaler settled.

All four are universal — no storage-type detection, no env vars to
flip, no per-deploy tuning required. Total +201/-6 across four files.
2026-05-13 13:50:44 -07:00
MattJackson 005f887bf9 v0.20.3: add halt check to Disc::patch backtrack inner loop
WO-5 (partial): the patch backtrack inner loop ('while bt_pos <
backtrack_end' in disc/patch.rs) issues per-sector reads to fill the
gap created by a damage-window skip. A long backtrack span can run
minutes; without an inline halt poll, the outer halt only takes
effect when control returns to the per-range loop. Adds a halt poll
at the top of each iteration so cancellation propagates inside the
backtrack span.

Per-sector read failures inside the backtrack already drop through
to the main fail path; this only changes the cancellation latency
between an /api/stop call and the producer actually unwinding. Drops
worst-case unwind from 'whole backtrack span × per-sector recovery
timeout' (minutes) to 'one in-flight SCSI command' (seconds).

The broader Arc<AtomicBool> → Halt migration on CopyOptions /
SweepOptions / PatchOptions / Drive::halt and Pipeline::send halt-
awareness is deferred — separate cycle, larger API impact.
2026-05-13 11:55:52 -07:00
MattJackson d7243a6044 v0.20.2: delete dead retry block + wedge-family cooldown in Disc::patch
WO-3c: Delete dead non-NOT_READY retry block (~100 LOC). The block
declared retry_count = 0 inside the per-iteration Err arm, so the
'MAX_NON_NOT_READY_RETRIES=3' budget actually fired exactly once
(1s pause + 1 retry) before falling through to NonTrimmed. The
'exponential backoff: 2s, 4s, 8s' comment was wrong by construction.
Cross-pass NonTrimmed retry (each pass gives the same sectors another
shot) already covers the recovery case, and gives the drive minutes
between attempts instead of 1-8 seconds — empirically what stochastic
recovery on the BU40N actually needs.

WO-4 (targeted slice): Add wedge-family cooldown on HARDWARE_ERROR /
ILLEGAL_REQUEST senses. These are what the BU40N's firmware fast-fail
state returns; every subsequent read in that state comes back in
<100ms. Pre-fix patch hammered the drive: mark NonTrimmed, sleep 1s,
advance, hit next wedge, mark, sleep 1s — exactly the rapid-retry
cadence the firmware is sensitive to. Now a wedge-family sense triggers
WEDGE_FAMILY_COOLDOWN_SECS=30 cooldown (matches read_error.rs's
ZONE_ENTRY_COOLDOWN_SECS), and WEDGE_ABORT_THRESHOLD=16 consecutive
wedges aborts the pass for autorip eject+reload. Any non-wedge read
clears the counter.

Also drops the duplicate NonTrimmed dispatch (Mapfile::record is
idempotent so it wasn't a correctness bug, but it doubled per-failure
consumer work).
2026-05-13 11:53:57 -07:00
MattJackson f1926c38dc v0.20.1: delete SectorReader, extract Disc::patch, doc/stub cleanup
WO-2 (delete SectorReader trait):
- The 0.18 trait split into SectorSource (read-only) and SectorSink
  (write-only) is final; the legacy SectorReader alias was a bridge.
- Renames every internal &mut dyn SectorReader (~25 sites) to
  &mut dyn SectorSource. The trait method capacity() becomes
  capacity_sectors() with a default of 0 (preserves SectorReader's
  default-0 behavior).
- Deletes the SectorReader trait, its blanket-to-Source bridge, and
  the FileSectorReader type alias. Adds explicit forwarding impls
  for Box<dyn SectorSource> and &mut dyn SectorSource so generic
  decorators like DecryptingSectorSource<S: SectorSource> compose.

WO-3a (extract Disc::patch):
- Moves Disc::patch (1230 lines) and bytes_bad_in_title from
  disc/mod.rs into disc/patch.rs as a split inherent impl. Zero
  behavior change — pure mechanical relocation. disc/mod.rs drops
  from 3,945 to 2,714 LOC.

WO-6 (partial):
- Deletes src/labels/png_filenames.rs — was a 72-LOC stub with
  detect() returning false, never wired into the PARSERS registry.

project docs doc drift fixes (audited 2026-05-13):
- JUMP_BASE_SECTORS: 256→1024 (64 MB base for UHD, not 8 MB)
- PASSN_DAMAGE_THRESHOLD_PCT: 12→6
- PASSN_SKIP_SECTORS_BASE: 64→32
- MAX_RANGE_SECS=180: replaced by proportional range_sectors × 25,
  capped at RANGE_BUDGET_CAP_SECS=1800.
2026-05-13 11:36:55 -07:00
MattJackson 4709a73c80 v0.20.0: delete FrameSource/FrameSink, keep single Stream trait
The 0.18 trait split into FrameSource (read-only) and FrameSink
(write-only) was an over-engineered API. Consumers don't think
"frame source backed by MKV" — they think "open MKV for reading".
The split paid a real API-complexity cost (two trait names, two
re-exports, dual impls per bidirectional type, deprecation bridge)
for one marginal property: compile-time direction-safety at the
trait-object boundary. The runtime error path on a wrong-direction
call (StreamReadOnly / StreamWriteOnly) is unambiguous and rare in
practice.

Deletions:
- pes::Stream is no longer #[deprecated]
- pes::FrameSource trait + its blanket-from-Stream bridge
- pes::FrameSink trait + the trampoline impls on every concrete type
- The compile-time-direction-safety test scaffolding
- Crate-root FrameSource / FrameSink re-exports

Additions:
- Stream is now Send-bounded (Stream: Send supertrait). Every
  concrete impl was already Send-compliant — Box<dyn Read + Send>
  and Box<dyn Write + Send> were already in place on the trait
  objects MkvStream / M2tsStream / etc hold internally. Promoting
  Send into the trait makes Box<dyn Stream> Send too, which lets
  autorip drop its SendStream unsafe newtype.

The public API is now: one Stream trait, one concrete type per
format, two constructors (open/create or input/output). Bidirectional
types route through internal Mode { Read | Write } discriminants.

Net: -347 lines libfreemkv, -38 lines autorip, -5 lines freemkv.
2026-05-13 08:42:14 -07:00
MattJackson 55bd1ee868 v0.19.1: repair Cargo.toml after botched 0.19.0 bump
v0.19.0 was tagged with a search-and-replace gone wrong:
rust-version, serde, and zip all had their version strings
replaced with "0.19.0". Edition 2024 rejected rust-version
0.19.0 (< 1.85), failing every CI build. No artifacts shipped
to crates.io.

Repair:
- rust-version: 0.19.0 → 1.86 (CI pin)
- serde: 0.19.0 → 1
- zip: 0.19.0 → 2

Also drops an unused start_lba binding in mux/disc.rs that
clippy 1.86 catches.
2026-05-12 21:29:07 -07:00
MattJackson 5285e8b6ec v0.19.0: bump version 2026-05-12 19:24:06 -07:00
MattJackson c5af61e46b v0.18.25: bump version 2026-05-12 16:27:28 -07:00
MattJackson ed73c1ab68 v0.18.25: add mux debug logging for reader type and stall detection 2026-05-12 16:27:16 -07:00
MattJackson 5ce2df0888 v0.18.24: bump version 2026-05-12 15:46:47 -07:00
MattJackson ea591a36a5 v0.18.23: bump version 2026-05-12 10:59:45 -07:00
MattJackson 368e10486e v0.18.22: bump version 2026-05-12 09:33:00 -07:00
MattJackson a99c4f8487 v0.18.21: format code 2026-05-12 09:18:52 -07:00
MattJackson 4df470e572 v0.18.21: fix borrow checker, make io public 2026-05-12 09:01:52 -07:00
MattJackson a0b13941fe v0.18.21: fix borrow checker error in debug logging 2026-05-12 08:49:33 -07:00
MattJackson 275c9da009 v0.18.21: verbose debug logging for pipeline stalls (fixed) 2026-05-12 08:37:42 -07:00
MattJackson da183722fa v0.18.20: bump version 2026-05-11 22:33:15 -07:00
MattJackson 4a5424eabe v0.18.20: bump version 2026-05-11 22:28:57 -07:00
MattJackson 293d89c586 v0.18.20: separate read/write pipeline depths 2026-05-11 22:18:56 -07:00
MattJackson 4840f5134d libfreemkv: increase DEFAULT_PIPELINE_DEPTH to 32 for smoother mux speed 2026-05-11 22:01:27 -07:00
MattJackson 11c8211605 v0.18.18: bump version 2026-05-11 20:16:30 -07:00
MattJackson 1038beaf06 v0.18.17: bump version 2026-05-11 19:58:36 -07:00
MattJackson f1df57196c libfreemkv: add Clone derive to MuxAtomics 2026-05-11 19:54:50 -07:00
48 changed files with 3786 additions and 2084 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "0.18.16"
version = "0.20.8"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
+2 -2
View File
@@ -16,7 +16,7 @@ pub mod lfsr;
pub(crate) mod tables;
use crate::disc::Extent;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
/// CSS decryption state for a DVD title.
#[derive(Debug, Clone)]
@@ -34,7 +34,7 @@ pub struct CssState {
/// 0x80 (start of the encrypted region). This only happens when a new PES
/// packet begins at exactly sector offset 128. We scan up to 50000
/// scrambled sectors sequentially across all extents.
pub fn crack_key(reader: &mut dyn SectorReader, extents: &[Extent]) -> Option<CssState> {
pub fn crack_key(reader: &mut dyn SectorSource, extents: &[Extent]) -> Option<CssState> {
let mut tried = 0u32;
let max_tries = 50_000;
+4 -4
View File
@@ -3,13 +3,13 @@
use super::*;
use crate::clpi;
use crate::mpls;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
impl Disc {
/// Scan Blu-ray titles from MPLS playlists.
pub(super) fn scan_bluray_titles(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Vec<DiscTitle> {
let mut titles = Vec::new();
@@ -31,7 +31,7 @@ impl Disc {
}
pub(super) fn parse_playlist(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
filename: &str,
data: &[u8],
@@ -199,7 +199,7 @@ impl Disc {
/// Prefers English, falls back to first available language.
/// Returns None if META directory is empty or XML has no usable title.
pub(super) fn read_meta_title(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Option<String> {
let meta_dir = udf_fs.find_dir("/BDMV/META")?;
+2 -2
View File
@@ -2,13 +2,13 @@
use super::*;
use crate::ifo;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
impl Disc {
/// Scan DVD titles from IFO files (VIDEO_TS.IFO + VTS_XX_0.IFO).
pub(super) fn scan_dvd_titles(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf_fs: &udf::UdfFs,
) -> Vec<DiscTitle> {
let dvd_info = match ifo::parse_vmg(reader, udf_fs) {
+3 -3
View File
@@ -2,7 +2,7 @@
use super::*;
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf;
/// Result of SCSI AACS handshake (ECDH authentication).
@@ -113,12 +113,12 @@ impl Disc {
/// Resolve disc encryption — AACS 1.0, AACS 2.0, CSS, or none.
///
/// Reads AACS files from UDF (via SectorReader), resolves keys through
/// Reads AACS files from UDF (via SectorSource), resolves keys through
/// whatever path works: KEYDB VUK lookup, media key derivation, processing
/// keys, device keys. Uses handshake result (volume ID, bus key) if available.
pub(super) fn resolve_encryption(
udf_fs: &udf::UdfFs,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
keydb_path: &std::path::Path,
handshake: Option<&HandshakeResult>,
) -> Result<AacsState> {
+14 -1246
View File
File diff suppressed because it is too large Load Diff
+1528 -3
View File
File diff suppressed because it is too large Load Diff
+29 -1
View File
@@ -186,6 +186,22 @@ impl ReadCtx {
/// marginal media is part of the job, and the fast-jump
/// threshold is loose so we don't bail too early on a range that
/// has scattered good sectors mixed in.
///
/// `damage_threshold_pct = 6` mirrors `disc/patch.rs`'s
/// `PASSN_DAMAGE_THRESHOLD_PCT`. Pass N triggers the damage-skip
/// at half the density Pass 1 uses (Pass 1 = 12%) because the
/// patch loop's whole job is to chip away at bad ranges — being
/// more eager to skip clustered bad sectors converges faster on
/// the recoverable good sectors inside a range. The patch-side
/// `compute_damage_skip` reads its threshold from
/// `PASSN_DAMAGE_THRESHOLD_PCT`; keep the two in sync until the
/// patch loop's damage-skip is unified with `handle_read_error`'s
/// jump path. (v0.20.8 unification attempt found the unification
/// itself blocked on the size-aware `range_remaining/4` cap that
/// lives in `compute_damage_skip` but not in
/// `handle_read_error::JumpAhead` — see
/// `tests/passn_handler_ab.rs` for the A/B fixture that pins
/// the divergence point.)
pub fn for_patch(batch: u16) -> Self {
Self {
batch,
@@ -194,7 +210,7 @@ impl ReadCtx {
consecutive_outer_failures: 0,
damage_window: Vec::with_capacity(16),
damage_window_max: 16,
damage_threshold_pct: 12,
damage_threshold_pct: PATCH_DAMAGE_THRESHOLD_PCT,
// Pass N is allowed to grind: window-based jump only,
// matching the historical behaviour for patch passes.
fast_jump_threshold: u64::MAX,
@@ -401,6 +417,18 @@ const WEDGE_ABORT_THRESHOLD: u64 = 16;
/// next range.
const WEDGE_PASS_N_SKIP_SECTORS: u64 = 64;
/// Single source of truth for the Pass-N damage-window threshold.
/// Both [`ReadCtx::for_patch`] and `disc::patch::compute_damage_skip`
/// reference this constant so the two damage-skip paths cannot drift.
///
/// 6% means: with a 16-entry sliding window, the damage-skip fires
/// once 1 out of 16 recent reads has failed. Pass 1 uses a 12%
/// threshold via `damage_threshold_pct` on `for_sweep`; Pass N is
/// twice as eager because patch's whole job is to converge on the
/// bad sub-zones inside a NonTrimmed range — a faster trigger
/// produces tighter convergence in fewer iterations.
pub const PATCH_DAMAGE_THRESHOLD_PCT: usize = 6;
/// THE single error-handling entry point. Updates `ctx`, returns the
/// action the caller must apply.
///
+1 -1
View File
@@ -20,7 +20,7 @@
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorReader`). No new SCSI concurrency.
//! owns the `SectorSource`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
+2 -2
View File
@@ -29,7 +29,7 @@ use crate::platform::PlatformDriver;
use crate::platform::mt1959::Mt1959;
use crate::profile::{self, DriveProfile};
use crate::scsi::ScsiTransport;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use std::path::Path;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -707,7 +707,7 @@ fn open_block_device_for_sg(sg_path: &Path) -> Option<std::os::unix::io::RawFd>
}
}
impl SectorReader for Drive {
impl SectorSource for Drive {
fn read_sectors(
&mut self,
lba: u32,
+3 -3
View File
@@ -9,7 +9,7 @@
use crate::disc::{Codec, Resolution};
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
// ── Public types ────────────────────────────────────────────────────────────
@@ -184,7 +184,7 @@ fn bcd_byte(b: u8) -> u32 {
///
/// Reads the VMG (Video Manager) to discover title sets, then reads each
/// VTS IFO to extract PGC chains, cell addresses, and stream attributes.
pub fn parse_vmg(reader: &mut dyn SectorReader, udf: &UdfFs) -> Result<DvdInfo> {
pub fn parse_vmg(reader: &mut dyn SectorSource, udf: &UdfFs) -> Result<DvdInfo> {
let vmg_data = udf.read_file(reader, "/VIDEO_TS/VIDEO_TS.IFO")?;
// Validate VMG magic
@@ -263,7 +263,7 @@ pub fn parse_vmg(reader: &mut dyn SectorReader, udf: &UdfFs) -> Result<DvdInfo>
///
/// `titles_info` is a list of (chapter_count, vts_title_number) from TT_SRPT.
fn parse_vts(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf: &UdfFs,
vts_number: u8,
titles_info: &[(u16, u8)],
+246
View File
@@ -0,0 +1,246 @@
//! Bounded-syscall primitive: run a (potentially-blocking) operation
//! on a worker thread, with a hard wall-clock deadline and an optional
//! cooperative [`Halt`] poll. The calling thread is never trapped
//! inside a kernel call.
//!
//! ## Why this exists
//!
//! [`crate::halt::Halt`] is cooperative: callers poll
//! `is_cancelled()`. It cannot reach inside a syscall the kernel
//! currently owns the thread for — `libc::sync_file_range`,
//! `libc::fsync`, `File::write` on NFS, and so on. `/api/stop` from
//! autorip therefore can't unstick a thread sitting in such a syscall.
//!
//! [`bounded_syscall`] is the escape hatch: it runs `op` on a fresh
//! worker thread, then `recv_timeout`s on a rendezvous channel for the
//! result. The wait is broken into ~250 ms slices so the calling
//! thread can poll the supplied [`Halt`] in between. If the deadline
//! elapses or the halt fires, the worker is intentionally leaked — the
//! syscall will unwind whenever the kernel decides, or at process
//! exit, but the caller is free to fall back to a degraded code path
//! (skip the sync, log loudly, etc.).
//!
//! ## Trade-offs
//!
//! - **Thread per call.** Cheap (`std::thread::spawn` is < 100 µs on
//! Linux/macOS), but not free. Use on coarse-grained finalisation
//! syscalls (`sync_all`, `sync_file_range(WAIT_AFTER)`), not on hot
//! inner-loop writes.
//! - **Leak on timeout.** A wedged syscall keeps a kernel slot and a
//! user-space thread around for the rest of the process's life.
//! Bounded by the number of independent rip/mux sessions, which is
//! one per disc. The alternative — trapping the caller forever —
//! defeats the entire purpose of `/api/stop`.
//! - **Halt granularity ~250 ms.** Halt observation is not instant;
//! it's the worst-case latency of the `recv_timeout` slice. Good
//! enough for human-driven stop requests; not suitable for hard
//! real-time deadlines.
//!
//! ## Single source of truth
//!
//! Do NOT inline this pattern. Every blocking-syscall wrapper in the
//! rip + mux pipeline calls this helper, so changes (e.g. swapping the
//! channel impl, adjusting the poll slice, adding metrics) land in one
//! place.
//!
//! ## Platform
//!
//! Pure `std::thread` + `std::sync::mpsc`. No `cfg(target_os)` needed
//! here — the helper itself is platform-agnostic. Callers that wrap
//! Linux-only syscalls (`sync_file_range`) still need their own
//! `#[cfg(target_os = "linux")]` gates; this helper does not.
use std::sync::mpsc::{RecvTimeoutError, sync_channel};
use std::thread;
use std::time::{Duration, Instant};
use crate::halt::Halt;
/// Granularity of the halt poll. The receive loop wakes every
/// [`POLL_INTERVAL`] to (a) check the [`Halt`] token, then (b) check
/// the overall deadline, then go back to waiting. 250 ms is a
/// pragmatic balance: short enough that human-driven `/api/stop` feels
/// responsive (< 0.5 s p99), long enough that the polling overhead is
/// negligible against multi-second syscalls.
const POLL_INTERVAL: Duration = Duration::from_millis(250);
/// Failure outcome from a bounded syscall wrapper.
#[derive(Debug)]
pub(crate) enum BoundedError {
/// The user-visible halt token fired during the wait. The worker
/// thread is intentionally leaked — the caller should fall back to
/// a degraded code path rather than waiting on the syscall to
/// return.
Halted,
/// The deadline elapsed before the syscall returned. Same leak
/// semantics as `Halted`.
Timeout,
/// The worker thread panicked, or its sender disconnected before
/// sending a result. Treat as a benign no-op (callers usually
/// log and continue) rather than a hard error — by definition no
/// syscall observably ran to completion in this case.
WorkerLost,
}
/// Run a (potentially-blocking) operation on a worker thread with a
/// deadline and an optional cooperative halt-token poll. Returns the
/// operation's result if it completes within `timeout`; otherwise one
/// of [`BoundedError::Halted`] / [`BoundedError::Timeout`] /
/// [`BoundedError::WorkerLost`].
///
/// On `Halted` / `Timeout` the worker thread is intentionally leaked:
/// the syscall will unwind whenever the kernel decides, or when the
/// process exits. The calling thread is never trapped inside a kernel
/// call.
///
/// `halt` is polled at [`POLL_INTERVAL`] granularity. Pass `None` for
/// callers that don't (yet) have a halt token plumbed through —
/// behaviour degrades to deadline-only, matching the 0.20.5
/// `wait_after_with_timeout` shape this helper generalises.
///
/// `op` returns `R: Send + 'static`. The closure must own everything
/// it touches because it may outlive this call (timeout / halt cases).
pub(crate) fn bounded_syscall<F, R>(
halt: Option<&Halt>,
timeout: Duration,
op: F,
) -> Result<R, BoundedError>
where
F: FnOnce() -> R + Send + 'static,
R: Send + 'static,
{
// Rendezvous channel: the worker sends exactly one value (the
// op's return) and then exits. Capacity-0 means the send blocks
// until we receive — fine on the happy path; on the timeout /
// halt path the receiver is dropped and the worker's send
// returns Err, which the worker ignores.
let (tx, rx) = sync_channel::<R>(0);
let _ = thread::Builder::new()
.name("freemkv-bounded-syscall".into())
.spawn(move || {
// Ignore the send error: if we time out (or get halted)
// before the worker finishes, the receiver is dropped
// and `tx.send` returns Err. Either way, the worker has
// nothing more to do.
let _ = tx.send(op());
});
let deadline = Instant::now() + timeout;
loop {
let now = Instant::now();
let remaining = deadline.saturating_duration_since(now);
let slice = remaining.min(POLL_INTERVAL);
match rx.recv_timeout(slice) {
Ok(v) => return Ok(v),
Err(RecvTimeoutError::Timeout) => {
if let Some(h) = halt {
if h.is_cancelled() {
return Err(BoundedError::Halted);
}
}
if Instant::now() >= deadline {
return Err(BoundedError::Timeout);
}
// Otherwise: another slice.
}
Err(RecvTimeoutError::Disconnected) => {
// Worker thread spawn failed, or it panicked before
// sending. Caller treats this as "no syscall ran" —
// typically a no-op + log.
return Err(BoundedError::WorkerLost);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
#[test]
fn op_completes_quickly() {
let r = bounded_syscall(None, Duration::from_secs(2), || 42u32);
assert!(matches!(r, Ok(42)));
}
#[test]
fn op_exceeds_timeout() {
// Op sleeps longer than the deadline → Timeout.
let r = bounded_syscall(None, Duration::from_millis(300), || {
thread::sleep(Duration::from_secs(2));
0u32
});
assert!(matches!(r, Err(BoundedError::Timeout)));
}
#[test]
fn halt_fires_during_wait() {
let halt = Halt::new();
let halt2 = halt.clone();
// Flip the halt from a side thread after ~300 ms — long
// enough that the receive loop has rolled at least one
// 250 ms slice and is sitting in `recv_timeout` again when
// the bit flips.
thread::spawn(move || {
thread::sleep(Duration::from_millis(300));
halt2.cancel();
});
let r = bounded_syscall(Some(&halt), Duration::from_secs(5), || {
thread::sleep(Duration::from_secs(5));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
}
#[test]
fn worker_panics() {
// Worker panics → sender drops without sending → recv sees
// Disconnected → WorkerLost. We use an explicit panic in the
// op closure rather than `panic!()` from inside the channel
// machinery; the spawned thread's panic is contained (no
// process abort) because we don't `.join()` it.
let r = bounded_syscall(None, Duration::from_secs(2), || -> u32 {
panic!("intentional test panic");
});
assert!(matches!(r, Err(BoundedError::WorkerLost)));
}
#[test]
fn halt_already_set_before_call_still_returns_halted() {
// Halt observed on the very first poll slice. The op blocks
// forever; we must not wait the full timeout to notice the
// halt is already set.
let halt = Halt::new();
halt.cancel();
let started = Instant::now();
let r = bounded_syscall(Some(&halt), Duration::from_secs(10), || {
thread::sleep(Duration::from_secs(10));
0u32
});
assert!(matches!(r, Err(BoundedError::Halted)));
// Should bail out within ~1 s; allow 2 s of slack for slow
// CI hosts.
assert!(
started.elapsed() < Duration::from_secs(2),
"halt-already-set took {:?}",
started.elapsed()
);
}
#[test]
fn ok_path_takes_no_halt_token() {
// Sanity: the `None` halt path is the documented zero-config
// form (matches the 0.20.5 `wait_after_with_timeout`
// behaviour). Op returns immediately; we must observe Ok.
let flag = Arc::new(AtomicBool::new(false));
let f2 = flag.clone();
let r = bounded_syscall(None, Duration::from_secs(2), move || {
f2.store(true, Ordering::Relaxed);
"ok"
});
assert!(matches!(r, Ok("ok")));
assert!(flag.load(Ordering::Relaxed));
}
}
+5 -2
View File
@@ -14,6 +14,7 @@
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
pub(crate) mod bounded;
mod writeback;
mod writeback_file;
@@ -27,5 +28,7 @@ pub(crate) use writeback_file::WritebackFile;
// caller — the targeted `#[allow]` keeps the re-export visible without
// dragging the rest of the module under `dead_code`.
#[allow(unused_imports)]
pub use pipeline::WRITE_THROUGH_DEPTH;
pub use pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink};
pub use pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
+418 -21
View File
@@ -27,36 +27,66 @@
//! - Consumer panic is converted into
//! `Error::IoError { source: io::Error::other(...) }`.
//!
//! ## Dead-code suppression
//! ## Debug logging
//!
//! Both `Disc::sweep` (`DEFAULT_PIPELINE_DEPTH`, `spawn_named`,
//! `try_send`) and `Disc::patch` (`WRITE_THROUGH_DEPTH`, `Flow::Stop`)
//! are now in-tree callers (0.18 round 2). The targeted `#[allow]`
//! markers below cover the corners of the API not exercised by either
//! call site (e.g. `Pipeline::spawn` without a name; `Flow::Stop`
//! return path from sweep's sink — sweep always consumes the producer's
//! full work-list). Drop the allows once mux migrates.
//! Set `FREEMKV_DEBUG=1` environment variable to enable verbose debug
//! logging throughout the pipeline (channel sends/receives, backpressure,
//! consumer lag detection). This is critical for diagnosing stalls.
use std::io;
use std::sync::mpsc::{SyncSender, sync_channel};
use std::sync::mpsc::{SyncSender, TrySendError, sync_channel};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
use crate::error::Error;
use crate::halt::Halt;
/// Deadline for [`Pipeline::finish_with_halt`]'s polling join. Chosen
/// to be comfortably longer than the autorip hard watchdog
/// (`HARD_WATCHDOG_STALL_SECS = 300s`) so the watchdog's `exit(1)`
/// fires first when both are racing on the same wedged consumer.
///
/// 10 minutes is a backstop, not a normal timeout — the consumer is
/// expected to drain in seconds. If we hit this, something is wedged
/// inside a kernel call the consumer thread can't unwind from, and the
/// caller has already lost the rip.
pub const JOIN_TIMEOUT_SECS: u64 = 600;
/// Polling slice for the halt-aware send/finish loops. Mirrors the
/// `bounded_syscall` cadence (250 ms) so halt observation feels equally
/// responsive across both primitives.
const POLL_INTERVAL: Duration = Duration::from_millis(250);
/// Polling slice for the halt-aware send loop. Smaller than
/// [`POLL_INTERVAL`] because send latency tolerance is much lower —
/// frames must move through the channel at hundreds-of-Hz on the
/// happy path; 50 ms keeps backpressure-driven wakeups fine-grained
/// without busy-looping.
const SEND_POLL_INTERVAL: Duration = Duration::from_millis(50);
/// Check if verbose debug logging is enabled via FREEMKV_DEBUG env var.
pub fn debug_enabled() -> bool {
std::env::var("FREEMKV_DEBUG")
.ok()
.map(|v| v == "1" || v == "true" || v == "yes")
.unwrap_or(false)
}
/// Default channel depth for callers without a specific reason to
/// pick another value.
///
/// Empirically tuned for sweep and mux — both want enough slack that
/// short consumer stalls don't immediately back up onto the producer,
/// but not so much that a producer outpacing the consumer accumulates
/// arbitrary buffered work. `4` matches the depth sweep has used
/// since 0.17.11 (originally in `disc/sweep_pipeline.rs`, now in
/// `disc/sweep.rs::SweepSink`). Patch should usually use
/// [`WRITE_THROUGH_DEPTH`] (`1`) instead — write-through gives clean
/// back-pressure between every read attempt and the matching write,
/// which matters when the consumer is updating the mapfile in lockstep.
/// pick another value. Kept conservative (4) — most callers should
/// use READ_PIPELINE_DEPTH or WRITE_PIPELINE_DEPTH instead.
pub const DEFAULT_PIPELINE_DEPTH: usize = 4;
/// Read pipeline depth. Larger buffer compensates for drive variability
/// and NFS sync_file_range stalls; keeps ISO reader thread fed even when
/// consumer blocks on write.
pub const READ_PIPELINE_DEPTH: usize = 32;
/// Write pipeline depth. Smaller buffer reduces backpressure risk when
/// sync_file_range blocks; prevents producer from accumulating too much
/// work while consumer waits for NFS to drain.
pub const WRITE_PIPELINE_DEPTH: usize = 16;
/// Channel depth for write-through pipelines. Each `send` fully
/// drains before the next can enqueue. Use this when the producer
/// must observe consumer side-effects (e.g. mapfile state) before
@@ -149,6 +179,12 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
let mut stopped = false;
while let Ok(item) = rx.recv() {
if debug_enabled() {
tracing::debug!("Pipeline receive: item={}", std::any::type_name::<I>());
}
let apply_start = std::time::Instant::now();
if first_err.is_some() || stopped {
// Drain remaining items so the producer never
// blocks on a dead receiver. `apply` is not
@@ -159,11 +195,32 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
Ok(Flow::Continue) => {}
Ok(Flow::Stop) => {
stopped = true;
if debug_enabled() {
tracing::debug!("Pipeline: consumer returned Flow::Stop");
}
}
Err(e) => {
if debug_enabled() {
tracing::debug!("Pipeline: apply error, stopping, err={:?}", e);
}
first_err = Some(e);
}
}
let apply_elapsed = apply_start.elapsed();
if debug_enabled() && apply_elapsed > std::time::Duration::from_millis(100) {
tracing::debug!(
"Pipeline apply: took {:.2}s, item={}",
apply_elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!(
"Pipeline apply: OK in {:.3}ms, item={}",
apply_elapsed.as_micros(),
std::any::type_name::<I>()
);
}
}
match first_err {
@@ -188,7 +245,35 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// independent signal (e.g. `Halt`) — `send` alone is not the
/// notification edge.
pub fn send(&self, item: I) -> Result<(), I> {
self.tx.send(item).map_err(|e| e.0)
let start = std::time::Instant::now();
match self.tx.send(item) {
Ok(()) => {
let elapsed = start.elapsed();
if debug_enabled() && elapsed > std::time::Duration::from_millis(10) {
tracing::debug!(
"Pipeline send: blocked {:.2}s, item={}",
elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!("Pipeline send: OK in {:.3}ms", elapsed.as_micros());
}
Ok(())
}
Err(e) => {
let elapsed = start.elapsed();
if debug_enabled() && elapsed > std::time::Duration::from_millis(10) {
tracing::debug!(
"Pipeline send: blocked {:.2}s before channel closed, item={}",
elapsed.as_secs_f64(),
std::any::type_name::<I>()
);
} else if debug_enabled() {
tracing::debug!("Pipeline send: failed after {:.3}ms", elapsed.as_micros());
}
Err(e.0)
}
}
}
/// Non-blocking variant of [`Pipeline::send`]. If the channel is
@@ -200,6 +285,65 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
self.tx.try_send(item)
}
/// Halt-aware bounded variant of [`Pipeline::send`].
///
/// Polls `try_send` on a 50 ms slice. Between slices, checks
/// (1) the [`Halt`] token and (2) the per-call `deadline`. Returns:
///
/// - `Ok(())` once the item lands in the channel.
/// - `Err(item)` if the consumer disconnected, the halt fired, or
/// the deadline elapsed — the caller gets the item back so it
/// can decide whether to drop it, route it elsewhere, or unwind.
///
/// Use this in producer threads that have a `Halt` token threaded
/// through (mux, sweep, patch). Plain [`Pipeline::send`] is
/// preserved for callers that don't (yet) plumb halt through.
///
/// Unlike [`Pipeline::send`], this never blocks the producer
/// thread inside an unkillable `mpsc::send` — if the consumer is
/// wedged inside an unkillable syscall, the producer can still
/// observe `/api/stop` and unwind.
pub fn send_with_halt(&self, item: I, halt: &Halt, deadline: Duration) -> Result<(), I> {
let end = Instant::now() + deadline;
let mut pending = item;
loop {
match self.tx.try_send(pending) {
Ok(()) => return Ok(()),
Err(TrySendError::Full(returned)) => {
pending = returned;
if halt.is_cancelled() {
if debug_enabled() {
tracing::debug!(
"Pipeline send_with_halt: halt observed, returning item={}",
std::any::type_name::<I>()
);
}
return Err(pending);
}
if Instant::now() >= end {
if debug_enabled() {
tracing::debug!(
"Pipeline send_with_halt: deadline elapsed, returning item={}",
std::any::type_name::<I>()
);
}
return Err(pending);
}
thread::sleep(SEND_POLL_INTERVAL);
}
Err(TrySendError::Disconnected(returned)) => {
if debug_enabled() {
tracing::debug!(
"Pipeline send_with_halt: consumer disconnected, item={}",
std::any::type_name::<I>()
);
}
return Err(returned);
}
}
}
}
/// Drop the producer-side channel and wait for the consumer
/// thread to finish. Returns whatever the consumer's `close()`
/// produced, or the first `apply` error, or — on consumer panic —
@@ -229,6 +373,69 @@ impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
}
}
}
/// Halt-aware, deadline-bounded variant of [`Pipeline::finish`].
///
/// Drops the producer-side channel (same as `finish`) and then
/// polls `JoinHandle::is_finished()` on a 250 ms cadence. Between
/// slices, checks (1) the optional [`Halt`] token and (2) the
/// [`JOIN_TIMEOUT_SECS`] deadline. Returns:
///
/// - `Ok(R)` on a clean consumer exit.
/// - `Err(Error::IoError)` with one of three message prefixes for
/// wedge cases:
/// - `"pipeline join halted"` — halt fired while waiting.
/// - `"pipeline join timed out"` — `JOIN_TIMEOUT_SECS` elapsed.
/// - `"pipeline consumer panicked"` — same as `finish()`.
///
/// In the `halted` and `timed out` branches the consumer thread is
/// intentionally leaked — exactly the same trade-off the
/// `bounded_syscall` primitive makes. The wedged kernel call
/// inside the consumer will unwind whenever it does, or at
/// process exit. The caller is free to fall back to a degraded
/// path (in autorip's case: `exit(1)` after the hard watchdog
/// escalation, letting Docker restart the container).
///
/// Plain [`Pipeline::finish`] is preserved for callers without a
/// halt-token plumbed through; that path still blocks indefinitely
/// on `join()`, matching pre-0.20.8 behaviour.
pub fn finish_with_halt(self, halt: Option<&Halt>) -> Result<R, Error> {
let Pipeline { tx, handle } = self;
drop(tx);
let deadline = Instant::now() + Duration::from_secs(JOIN_TIMEOUT_SECS);
loop {
if handle.is_finished() {
return match handle.join() {
Ok(result) => result,
Err(payload) => {
let msg = payload
.downcast_ref::<&'static str>()
.copied()
.or_else(|| payload.downcast_ref::<String>().map(|s| s.as_str()))
.unwrap_or("(no message)");
Err(Error::IoError {
source: io::Error::other(format!("pipeline consumer panicked: {msg}")),
})
}
};
}
if let Some(h) = halt {
if h.is_cancelled() {
// Consumer thread is intentionally leaked.
return Err(Error::IoError {
source: io::Error::other("pipeline join halted"),
});
}
}
if Instant::now() >= deadline {
// Consumer thread is intentionally leaked.
return Err(Error::IoError {
source: io::Error::other("pipeline join timed out"),
});
}
thread::sleep(POLL_INTERVAL);
}
}
}
#[cfg(test)]
@@ -493,4 +700,194 @@ mod tests {
other => panic!("expected Err(IoError), got {other:?}"),
}
}
/// Never-completing sink — `apply` blocks until cancelled. Signals
/// `started` once it has consumed its first item so the test
/// driver knows the consumer thread is wedged in `apply` (and
/// will no longer drain the channel). Used to drive the
/// halt/timeout paths of `send_with_halt` and `finish_with_halt`
/// without depending on real I/O.
struct NeverDrainsSink {
cancel: Arc<std::sync::atomic::AtomicBool>,
started: Arc<std::sync::atomic::AtomicBool>,
}
impl Sink<u64> for NeverDrainsSink {
type Output = ();
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
self.started.store(true, Ordering::SeqCst);
while !self.cancel.load(Ordering::SeqCst) {
std::thread::sleep(Duration::from_millis(20));
}
Ok(Flow::Continue)
}
fn close(self) -> Result<(), Error> {
Ok(())
}
}
/// Spin until `started` flips or `bail` elapses. Used by the
/// send_with_halt tests to synchronise with the consumer thread
/// before exercising the bounded-send timeout path.
fn wait_for_started(started: &Arc<std::sync::atomic::AtomicBool>, bail: Duration) {
let end = Instant::now() + bail;
while !started.load(Ordering::SeqCst) {
assert!(Instant::now() < end, "consumer never started apply()");
std::thread::sleep(Duration::from_millis(10));
}
}
#[test]
fn send_with_halt_returns_item_on_deadline() {
// depth=1 + consumer wedged in apply on the first item, AND
// the channel buffer already loaded with a second item, means
// any further `try_send` sees Full; with a 200 ms deadline and
// no halt fired, send_with_halt must return `Err(item)` within
// roughly the deadline. Synchronising on `started` ensures the
// consumer has actually started its wedged apply BEFORE we
// load the channel-buffer slot — without that, the consumer
// could still drain in a race window.
let cancel = Arc::new(std::sync::atomic::AtomicBool::new(false));
let started = Arc::new(std::sync::atomic::AtomicBool::new(false));
let pipe = Pipeline::spawn(
1,
NeverDrainsSink {
cancel: cancel.clone(),
started: started.clone(),
},
)
.expect("spawn should succeed");
// First send: consumer recv()s it and wedges in apply.
pipe.send(0u64).expect("first send hands off to consumer");
wait_for_started(&started, Duration::from_secs(2));
// Second send: lands in the depth=1 buffer slot, consumer
// can't pick it up because it's wedged in apply. Channel now
// full from the producer's perspective.
pipe.send(1u64).expect("second send fills the buffer");
let halt = crate::halt::Halt::new();
let start = Instant::now();
let res = pipe.send_with_halt(99u64, &halt, Duration::from_millis(200));
let elapsed = start.elapsed();
// Release the leaked consumer so the test process winds down.
cancel.store(true, Ordering::SeqCst);
let _ = pipe.finish();
assert!(matches!(res, Err(99)), "expected item returned on deadline");
assert!(
elapsed >= Duration::from_millis(150),
"deadline returned too early: {elapsed:?}"
);
assert!(
elapsed < Duration::from_secs(2),
"deadline blew past tolerance: {elapsed:?}"
);
}
#[test]
fn send_with_halt_returns_item_on_halt() {
// Same setup, but the halt fires before the deadline elapses.
// The send loop must observe the halt within ~50 ms (the
// SEND_POLL_INTERVAL) and return the item.
let cancel = Arc::new(std::sync::atomic::AtomicBool::new(false));
let started = Arc::new(std::sync::atomic::AtomicBool::new(false));
let pipe = Pipeline::spawn(
1,
NeverDrainsSink {
cancel: cancel.clone(),
started: started.clone(),
},
)
.expect("spawn should succeed");
pipe.send(0u64).expect("first send hands off to consumer");
wait_for_started(&started, Duration::from_secs(2));
pipe.send(1u64).expect("second send fills the buffer");
let halt = crate::halt::Halt::new();
let halt2 = halt.clone();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(100));
halt2.cancel();
});
let start = Instant::now();
let res = pipe.send_with_halt(7u64, &halt, Duration::from_secs(10));
let elapsed = start.elapsed();
cancel.store(true, Ordering::SeqCst);
let _ = pipe.finish();
assert!(matches!(res, Err(7)), "expected item returned on halt");
assert!(
elapsed < Duration::from_secs(2),
"halt observation took too long: {elapsed:?}"
);
}
#[test]
fn finish_with_halt_returns_halted_when_consumer_wedged() {
// Consumer wedges on the first apply; halt fires; finish
// returns the documented "pipeline join halted" error rather
// than blocking forever.
let cancel = Arc::new(std::sync::atomic::AtomicBool::new(false));
let started = Arc::new(std::sync::atomic::AtomicBool::new(false));
let pipe = Pipeline::spawn(
DEFAULT_PIPELINE_DEPTH,
NeverDrainsSink {
cancel: cancel.clone(),
started: started.clone(),
},
)
.expect("spawn should succeed");
pipe.send(0u64).expect("seed item the consumer wedges on");
wait_for_started(&started, Duration::from_secs(2));
let halt = crate::halt::Halt::new();
let halt2 = halt.clone();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(400));
halt2.cancel();
});
let start = Instant::now();
let res = pipe.finish_with_halt(Some(&halt));
let elapsed = start.elapsed();
// Release the leaked consumer so the test process exits cleanly.
cancel.store(true, Ordering::SeqCst);
match res {
Err(Error::IoError { source }) => {
assert!(
source.to_string().contains("pipeline join halted"),
"expected halt-prefix error, got: {source}"
);
}
other => panic!("expected Err(IoError) halted, got {other:?}"),
}
// Bailed out within ~1 second of the halt firing (worst case
// one POLL_INTERVAL = 250 ms of slack).
assert!(
elapsed < Duration::from_secs(2),
"halt observation took too long: {elapsed:?}"
);
}
#[test]
fn finish_with_halt_happy_path_returns_output() {
// No halt token, sink completes normally — finish_with_halt
// must return the same Output that `finish` would.
let pipe = Pipeline::spawn(DEFAULT_PIPELINE_DEPTH, SumSink { total: 0 })
.expect("spawn should succeed");
for i in 0..10u64 {
pipe.send(i).expect("send should succeed");
}
let total = pipe
.finish_with_halt(None)
.expect("happy-path finish_with_halt should succeed");
assert_eq!(total, (0..10u64).sum::<u64>());
}
}
+281 -31
View File
@@ -14,9 +14,63 @@
//! a full chunk's worth of work to flush — the wait is near-instant.
//! Dirty cache stays bounded at ~2 × `chunk_bytes` and writes drain
//! continuously instead of in bursts.
//!
//! The chunk size is adaptive: we measure the elapsed time of the
//! `WAIT_AFTER` call over a rolling window of the last 16 chunks and
//! resize the chunk based on the p95. Slow storage (NFS, network
//! shares, HDD) sees larger chunks to amortise per-chunk overhead;
//! fast storage (NVMe) sees smaller chunks to keep cache pressure
//! tight. Bounds: [4 MiB, 256 MiB].
//!
//! ## NFS escape hatch
//!
//! `sync_file_range(WAIT_AFTER)` on an NFS-mounted file can block
//! indefinitely waiting for the server's commit ack. If the server
//! never acks (network partition, server-side hang, slow commit), the
//! syscall never returns and the consumer thread is stuck inside the
//! kernel — `/api/stop` can't reach it because halt is cooperative.
//!
//! When `fstatfs` reports the file lives on an NFS mount
//! (`f_type == NFS_SUPER_MAGIC`), the pipeline skips the WAIT_AFTER +
//! `posix_fadvise(DONTNEED)` dance entirely. NFS clients have their
//! own buffering and commit semantics that handle dirty-page bounds
//! without us forcing the issue. The async `SYNC_FILE_RANGE_WRITE`
//! kickoff still runs (non-blocking by spec) so writeback still gets
//! a nudge.
//!
//! ## Defence in depth: WAIT_AFTER timeout
//!
//! Even on local storage, a degraded disk or odd filesystem driver
//! could in principle wedge inside WAIT_AFTER. Each WAIT_AFTER call
//! runs on a worker thread with a 30s recv_timeout on its result
//! channel. On timeout we log a loud error, set a `degraded` flag,
//! and from then on skip WAIT_AFTER + DONTNEED for the rest of the
//! pipeline's life (same shape as the NFS path). The worker thread
//! is intentionally leaked — it unwinds whenever the syscall
//! eventually returns or the process exits. The mux continues; the
//! original dirty-burst pathology re-emerges but the rip can still
//! finish instead of freezing.
use std::collections::VecDeque;
use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
const ADAPTIVE_WINDOW: usize = 16;
const CHUNK_BYTES_MIN: u64 = 4 * 1024 * 1024;
const CHUNK_BYTES_MAX: u64 = 256 * 1024 * 1024;
const ADAPTIVE_GROW_MS: u64 = 200;
const ADAPTIVE_SHRINK_MS: u64 = 20;
/// Every N chunks, emit a `debug!` snapshot of the current chunk
/// size so operators tailing the log can see where the autoscaler
/// settled.
const SIZE_LOG_INTERVAL: u64 = 32;
/// Hard upper bound on a single `sync_file_range(WAIT_AFTER)` call.
/// Beyond this we declare the pipeline degraded and stop calling
/// WAIT_AFTER for the rest of its life.
const WAIT_AFTER_TIMEOUT: Duration = Duration::from_secs(30);
pub(crate) struct WritebackPipeline {
/// Aliases the wrapping `WritebackFile::file`. Only valid for the
@@ -28,6 +82,25 @@ pub(crate) struct WritebackPipeline {
chunk_bytes: u64,
last_flush_pos: u64,
pending: Option<(u64, u64)>,
/// Rolling window of recent `WAIT_AFTER` elapsed_ms measurements.
wait_after_window: VecDeque<u64>,
/// Count of chunks emitted (used to space out periodic
/// `debug!` size snapshots).
chunk_count: u64,
/// True when the underlying file is on an NFS mount. NFS makes
/// WAIT_AFTER unsafe (can block forever on missing server ack), so
/// we skip it entirely and let the NFS client handle commit on
/// close.
is_nfs: bool,
/// Set the first time WAIT_AFTER exceeds [`WAIT_AFTER_TIMEOUT`].
/// Once set, behaviour matches the NFS path for the rest of the
/// pipeline's life. Wrapped in `Arc` only because both this
/// struct and the spawned worker thread (which itself doesn't
/// touch the flag) share-via-fd patterns might one day need it;
/// today it's effectively a single-owner cell — the `Arc` shape
/// keeps the door open for moving the read side into a worker
/// without re-plumbing types.
degraded: Arc<AtomicBool>,
}
impl WritebackPipeline {
@@ -36,14 +109,33 @@ impl WritebackPipeline {
/// itself, or kept inside the same struct that owns `file` — the
/// alias is unchecked.
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
let fd = file.as_raw_fd();
let is_nfs = detect_nfs(fd);
tracing::info!(
target: "mux",
"WritebackPipeline fd={fd} is_nfs={is_nfs} chunk_bytes={chunk_bytes} strategy={}",
if is_nfs { "nfs-skip-wait" } else { "wait+dontneed" }
);
Self {
fd: file.as_raw_fd(),
fd,
chunk_bytes,
last_flush_pos: start_pos,
pending: None,
wait_after_window: VecDeque::with_capacity(ADAPTIVE_WINDOW),
chunk_count: 0,
is_nfs,
degraded: Arc::new(AtomicBool::new(false)),
}
}
/// True if we should bypass the WAIT_AFTER + DONTNEED finalisation
/// step. NFS always bypasses; local storage bypasses once the
/// pipeline has flipped to degraded after a WAIT_AFTER timeout.
#[inline]
fn skip_wait(&self) -> bool {
self.is_nfs || self.degraded.load(Ordering::Relaxed)
}
/// Caller advanced the file position to `pos`. If a chunk boundary
/// was crossed, kick async writeback for the just-completed chunk
/// and finalise the previous one.
@@ -51,27 +143,115 @@ impl WritebackPipeline {
if pos < self.last_flush_pos.saturating_add(self.chunk_bytes) {
return;
}
let chunk_off = self.last_flush_pos as i64;
let chunk_len = (pos - self.last_flush_pos) as i64;
let mut wait_ms: u64 = 0;
let mut fadvise_ms: u64 = 0;
// Async kickoff for the just-completed chunk runs on every
// path (NFS, degraded, normal) — it's nominally non-blocking
// by spec and gives the kernel an early hint that this range
// is ready to flush.
unsafe {
let chunk_off = self.last_flush_pos as i64;
let chunk_len = (pos - self.last_flush_pos) as i64;
libc::sync_file_range(self.fd, chunk_off, chunk_len, libc::SYNC_FILE_RANGE_WRITE);
if let Some((prev_off, prev_len)) = self.pending.take() {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
self.pending = Some((chunk_off as u64, chunk_len as u64));
}
if let Some((prev_off, prev_len)) = self.pending.take() {
if self.skip_wait() {
// NFS branch (or degraded fallback after a prior
// timeout): the WAIT_AFTER + DONTNEED dance is what
// hangs on NFS — skip it. We still advance `pending`
// so the next call has a stable cycle.
} else {
// Normal local-storage branch with belt-and-braces
// timeout. If WAIT_AFTER hangs > WAIT_AFTER_TIMEOUT
// we mark the pipeline degraded, log a loud error,
// and fall through to the skip path on subsequent
// calls.
match wait_after_with_timeout(self.fd, prev_off, prev_len) {
Some(ms) => {
wait_ms = ms;
let t_fadv = Instant::now();
unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
fadvise_ms = t_fadv.elapsed().as_millis() as u64;
self.record_wait(wait_ms);
}
None => {
// Timeout branch: switch to NFS-style skip
// for the rest of the pipeline's life. Do
// NOT call DONTNEED — if WAIT_AFTER hasn't
// returned, the pages aren't safely flushed.
self.degraded.store(true, Ordering::Relaxed);
tracing::error!(
target: "mux",
"WritebackPipeline WAIT_AFTER timed out after {}s on chunk off={} len={}, marking writeback degraded (subsequent chunks will skip WAIT_AFTER + DONTNEED)",
WAIT_AFTER_TIMEOUT.as_secs(),
prev_off,
prev_len
);
}
}
}
}
self.pending = Some((chunk_off as u64, chunk_len as u64));
self.last_flush_pos = pos;
self.chunk_count += 1;
tracing::trace!(
target: "mux",
"WritebackPipeline chunk off={} len={} sync_file_range_ms={wait_ms} fadvise_ms={fadvise_ms} chunk_bytes={} skip_wait={}",
chunk_off,
chunk_len,
self.chunk_bytes,
self.skip_wait(),
);
if self.chunk_count % SIZE_LOG_INTERVAL == 0 {
tracing::debug!(
target: "mux",
"WritebackPipeline chunk_bytes={} after {} chunks is_nfs={} degraded={}",
self.chunk_bytes,
self.chunk_count,
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
}
}
/// Push a new `WAIT_AFTER` measurement into the rolling window
/// and, if the window is full, adapt `chunk_bytes` based on p95.
fn record_wait(&mut self, wait_ms: u64) {
if self.wait_after_window.len() == ADAPTIVE_WINDOW {
self.wait_after_window.pop_front();
}
self.wait_after_window.push_back(wait_ms);
if self.wait_after_window.len() < ADAPTIVE_WINDOW {
return;
}
// p95 of 16 samples ≈ sorted[14] (5 % of 16 = 0.8 ≈ 1 above).
let mut sorted: Vec<u64> = self.wait_after_window.iter().copied().collect();
sorted.sort_unstable();
let p95 = sorted[14];
let old = self.chunk_bytes;
let new = if p95 > ADAPTIVE_GROW_MS && self.chunk_bytes < CHUNK_BYTES_MAX {
(self.chunk_bytes * 2).min(CHUNK_BYTES_MAX)
} else if p95 < ADAPTIVE_SHRINK_MS && self.chunk_bytes > CHUNK_BYTES_MIN {
(self.chunk_bytes / 2).max(CHUNK_BYTES_MIN)
} else {
self.chunk_bytes
};
if new != old {
self.chunk_bytes = new;
tracing::info!(
target: "mux",
"WritebackPipeline adaptive chunk_bytes {} -> {} p95_ms={p95}",
old,
new
);
}
}
/// Caller is about to seek away from the current write region.
@@ -85,20 +265,90 @@ impl WritebackPipeline {
/// or when discarding the pipeline.
pub(crate) fn finalize(&mut self) {
if let Some((prev_off, prev_len)) = self.pending.take() {
unsafe {
libc::sync_file_range(
self.fd,
prev_off as i64,
prev_len as i64,
libc::SYNC_FILE_RANGE_WAIT_AFTER,
);
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
tracing::debug!(
target: "mux",
"WritebackPipeline finalize chunk off={prev_off} len={prev_len} skip_wait={} is_nfs={} degraded={}",
self.skip_wait(),
self.is_nfs,
self.degraded.load(Ordering::Relaxed),
);
if self.skip_wait() {
// NFS / degraded: skip WAIT_AFTER + DONTNEED. close()
// / sync_all() handle commit through their normal
// paths.
return;
}
match wait_after_with_timeout(self.fd, prev_off, prev_len) {
Some(_ms) => unsafe {
libc::posix_fadvise(
self.fd,
prev_off as i64,
prev_len as i64,
libc::POSIX_FADV_DONTNEED,
);
},
None => {
self.degraded.store(true, Ordering::Relaxed);
tracing::error!(
target: "mux",
"WritebackPipeline finalize WAIT_AFTER timed out after {}s on chunk off={prev_off} len={prev_len}, marking writeback degraded",
WAIT_AFTER_TIMEOUT.as_secs(),
);
}
}
}
}
}
/// Probe whether `fd` lives on an NFS mount via `fstatfs`. Returns
/// `false` on any error — we fail open, not closed: better to run the
/// normal local-storage path on a misdetected NFS mount (and surface
/// the freeze loudly via the timeout) than to needlessly disable
/// writeback bounding on every local file because of a transient
/// stat error.
fn detect_nfs(fd: RawFd) -> bool {
// `libc::statfs` is repr(C) with a fixed layout; zeroing is the
// documented init pattern for the kernel uapi struct.
let mut buf: libc::statfs = unsafe { std::mem::zeroed() };
let rc = unsafe { libc::fstatfs(fd, &mut buf) };
if rc != 0 {
let errno = std::io::Error::last_os_error();
tracing::warn!(
target: "mux",
"WritebackPipeline fstatfs(fd={fd}) failed: {errno} — defaulting is_nfs=false",
);
return false;
}
// `f_type` is signed (`__fsword_t`) on glibc and unsigned
// (`c_ulong`) on musl. Cast both sides to i64 for a portable
// comparison.
let f_type = buf.f_type as i64;
let nfs_magic = libc::NFS_SUPER_MAGIC as i64;
f_type == nfs_magic
}
/// Run `sync_file_range(WAIT_AFTER)` on a worker thread and wait up
/// to [`WAIT_AFTER_TIMEOUT`] for it to return. `Some(elapsed_ms)` on
/// success; `None` on timeout. On timeout the worker thread is
/// 0.20.6 generalizes the worker-thread + recv_timeout pattern into
/// [`crate::io::bounded::bounded_syscall`]; this helper now just adapts
/// the generic primitive to the WAIT_AFTER call shape (returns elapsed_ms
/// instead of the syscall's `()` return, treats `WorkerLost` as a benign
/// no-op to match the original semantics).
fn wait_after_with_timeout(fd: RawFd, off: u64, len: u64) -> Option<u64> {
let started = Instant::now();
match crate::io::bounded::bounded_syscall(None, WAIT_AFTER_TIMEOUT, move || unsafe {
libc::sync_file_range(fd, off as i64, len as i64, libc::SYNC_FILE_RANGE_WAIT_AFTER);
}) {
Ok(()) => Some(started.elapsed().as_millis() as u64),
Err(crate::io::bounded::BoundedError::Timeout)
| Err(crate::io::bounded::BoundedError::Halted) => None,
Err(crate::io::bounded::BoundedError::WorkerLost) => {
// Worker thread spawn failed or panicked before sending.
// Treat as a benign success (no syscall ran) rather than
// a degrade trigger — falling through with elapsed_ms=0
// matches the no-op behaviour.
Some(0)
}
}
}
+103 -3
View File
@@ -27,7 +27,7 @@ use std::path::Path;
use super::writeback::WritebackPipeline;
const CHUNK_BYTES: u64 = 32 * 1024 * 1024;
const WRITEBACK_CHUNK_BYTES: u64 = 32 * 1024 * 1024;
pub(crate) struct WritebackFile {
file: File,
@@ -42,7 +42,7 @@ impl WritebackFile {
/// or appended files).
pub(crate) fn new(mut file: File) -> io::Result<Self> {
let pos = file.stream_position()?;
let pipeline = WritebackPipeline::new(&file, pos, CHUNK_BYTES);
let pipeline = WritebackPipeline::new(&file, pos, WRITEBACK_CHUNK_BYTES);
Ok(Self {
file,
pipeline,
@@ -54,11 +54,57 @@ impl WritebackFile {
/// and wrap it. Convenience for the common
/// `File::create(path)` + `WritebackFile::new(file)` pair so callers
/// don't have to assemble a `File` first.
///
/// Callers that know the target output size should prefer
/// [`Self::create_with_size_hint`] so the kernel can pre-reserve
/// extents.
#[allow(dead_code)]
pub(crate) fn create(path: &Path) -> io::Result<Self> {
let file = File::create(path)?;
Self::new(file)
}
/// Like [`Self::create`] but pre-reserves `size_bytes` of disk
/// space via `fallocate(FALLOC_FL_KEEP_SIZE)` on Linux. The
/// reported file size is unchanged (writes still grow the file
/// naturally) — only the on-disk extent allocation is preallocated,
/// which reduces extent fragmentation on large sequential writes
/// (mux output, especially on slow storage / NFS).
///
/// On macOS / Windows the size hint is ignored and this is
/// equivalent to `create`.
pub(crate) fn create_with_size_hint(path: &Path, size_bytes: u64) -> io::Result<Self> {
let file = File::create(path)?;
#[cfg(target_os = "linux")]
{
use std::os::unix::io::AsRawFd;
// FALLOC_FL_KEEP_SIZE = 0x01 — keep the reported file
// size at 0 (writes grow it normally) while still
// pre-reserving the extents.
let rc = unsafe {
libc::fallocate(
file.as_raw_fd(),
libc::FALLOC_FL_KEEP_SIZE,
0,
size_bytes as i64,
)
};
tracing::debug!(
target: "mux",
"WritebackFile fallocate size_hint={size_bytes} rc={rc} ok={}",
rc == 0
);
}
#[cfg(not(target_os = "linux"))]
{
tracing::debug!(
target: "mux",
"WritebackFile fallocate size_hint={size_bytes} skipped (non-linux)"
);
}
Self::new(file)
}
/// Open an existing file at `path` for writing (no truncation) and
/// wrap it. Mirrors `File::open` semantics for the writable case
/// — used by patch / resume paths that mutate an existing ISO in
@@ -70,9 +116,49 @@ impl WritebackFile {
/// Drain in-flight writeback then issue a full fsync. Use this in
/// place of `File::sync_all`.
///
/// The final fsync is wrapped in
/// [`crate::io::bounded::bounded_syscall`] with a 60 s deadline.
/// fsync on a wedged NFS server (or a degraded local disk) can
/// hang the calling thread; the wrapper ensures the worst case is
/// 60 s + log-and-continue rather than indefinite. On timeout the
/// page cache is left to the kernel's normal flush-on-close path —
/// best effort, but bounded.
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
self.pipeline.finalize();
self.file.sync_all()
#[cfg(unix)]
{
use std::os::unix::io::AsRawFd;
use std::time::Duration;
let fd = self.file.as_raw_fd();
match crate::io::bounded::bounded_syscall(
None,
Duration::from_secs(60),
move || -> io::Result<()> {
let rc = unsafe { libc::fsync(fd) };
if rc == 0 {
Ok(())
} else {
Err(io::Error::last_os_error())
}
},
) {
Ok(inner) => inner,
Err(crate::io::bounded::BoundedError::Timeout) => {
tracing::error!(
target: "mux",
"WritebackFile::sync_all fsync timed out after 60s; kernel will flush on close (best-effort)"
);
Ok(())
}
Err(crate::io::bounded::BoundedError::Halted) => Ok(()),
Err(crate::io::bounded::BoundedError::WorkerLost) => Ok(()),
}
}
#[cfg(not(unix))]
{
self.file.sync_all()
}
}
}
@@ -104,6 +190,20 @@ impl Seek for WritebackFile {
// before every write, and we don't want that to drain the
// pipeline on every iteration.
if p != self.pos {
// Diagnostic for the NFS 73 % mux hang: the MKV format
// requires the muxer to seek back occasionally (cluster
// size patching, Cues index write, Segment header
// backpatch). Each such seek invalidates the writeback
// chunk tracking and forces a finalize → WAIT_AFTER on
// the in-flight chunk. Logging the seek delta lets us
// correlate hang offsets with specific muxer operations.
let from_pos = self.pos;
let to_pos = p;
let delta: i64 = (to_pos as i64).wrapping_sub(from_pos as i64);
tracing::debug!(
target: "mux",
"WritebackFile seek from={from_pos} to={to_pos} delta={delta}"
);
self.pipeline.handle_seek(p);
self.pos = p;
}
+2 -2
View File
@@ -26,7 +26,7 @@
// The module wiring (registry hook + public re-export) is added
// separately. Until then the parse/detect entry points have no
use super::xml;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
@@ -60,7 +60,7 @@ pub fn detect(udf: &UdfFs) -> bool {
/// Read every `bdmt_<lang>.xml` under `/BDMV/META/DL/` and return the
/// aggregated [`DiscMetadata`]. Returns `None` if no titles could be
/// extracted from any file.
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<DiscMetadata> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<DiscMetadata> {
let dir = udf.find_dir("/BDMV/META/DL")?;
let mut out = DiscMetadata::default();
+2 -2
View File
@@ -29,7 +29,7 @@
//! report. Surfaced via the labels-analyze tool — not part of the
//! `analyze()` pipeline (no impact on the label output).
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
@@ -116,7 +116,7 @@ impl ClpiVsMplsAudit {
/// dedup-by-PID table of (CLPI fields, MPLS fields), return the
/// merged view. Missing files (read errors, parse failures) are
/// silently skipped — this is diagnostic, not correctness-critical.
pub fn audit(reader: &mut dyn SectorReader, udf: &UdfFs) -> ClpiVsMplsAudit {
pub fn audit(reader: &mut dyn SectorSource, udf: &UdfFs) -> ClpiVsMplsAudit {
// Aggregate by PID across all CLPI files. If a PID appears in
// multiple clips (typical — main movie clip + trailers reference
// the same audio stream PIDs), first encountered wins (they should
+2 -2
View File
@@ -4,7 +4,7 @@
//! stream number mapping via playbackconfig.
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, xml};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::HashMap;
@@ -12,7 +12,7 @@ pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "streamproperties.xml")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let sp_data = super::read_jar_file(reader, udf, "streamproperties.xml")?;
let sp_text = std::str::from_utf8(&sp_data).ok()?;
+5 -5
View File
@@ -5,7 +5,7 @@
//! menu_base.prop provides stream number → button name mapping.
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, vocab};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::HashMap;
@@ -14,7 +14,7 @@ pub fn detect(udf: &UdfFs) -> bool {
|| super::jar_file_exists(udf, "language_streams.txt")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
// Try language_streams.txt first (richer structured data)
let ls_labels = parse_language_streams(reader, udf);
@@ -55,7 +55,7 @@ fn merge(ls: Vec<StreamLabel>, mb: Vec<StreamLabel>) -> Vec<StreamLabel> {
// ── language_streams.txt parser ────────────────────────────────────────────
fn parse_language_streams(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
fn parse_language_streams(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let data = super::read_jar_file(reader, udf, "language_streams.txt")?;
let text = std::str::from_utf8(&data).ok()?;
@@ -184,7 +184,7 @@ mod tests {
/// Build a minimal menu_base.prop text and run `parse_menu_base`'s
/// inner logic via a temporary closure. This isolates the prop
/// parsing without needing a SectorReader.
/// parsing without needing a SectorSource.
fn parse_props(text: &str) -> Vec<StreamLabel> {
// Mirror the inner loop of parse_menu_base exactly. Kept
// separate so the test doesn't need disc fixtures.
@@ -338,7 +338,7 @@ mod tests {
// ── menu_base.prop parser ──────────────────────────────────────────────────
fn parse_menu_base(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
fn parse_menu_base(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<Vec<StreamLabel>> {
let data = super::read_jar_file(reader, udf, "menu_base.prop")?;
let text = std::str::from_utf8(&data).ok()?;
+3 -3
View File
@@ -35,11 +35,11 @@
use super::class_reader::CpInfo;
use super::{ParseResult, StreamLabel, StreamLabelType, jar, vocab};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::BTreeMap;
/// dbp detect can't peek inside a jar without a SectorReader (the
/// dbp detect can't peek inside a jar without a SectorSource (the
/// trait function only takes `&UdfFs`), so we trigger on the cheap
/// signal "any top-level .jar in /BDMV/JAR/." That fires on every
/// BD-J disc, but parse() does the real `com/dbp/` check and
@@ -49,7 +49,7 @@ pub fn detect(udf: &UdfFs) -> bool {
jar::has_any_top_level_jar(udf)
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
jar::for_each_jar(reader, udf, |_entry_name, archive| {
if !jar::has_path_prefix(archive, "com/dbp/") {
return None;
+3 -3
View File
@@ -84,18 +84,18 @@ use super::class_reader::{
ICONST_2, ICONST_3, ICONST_4, ICONST_5, ICONST_M1, INVOKESPECIAL, LDC, LDC_W, NEW, SIPUSH,
};
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, jar, vocab};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::collections::{HashMap, HashSet};
pub fn detect(udf: &UdfFs) -> bool {
// Cheap pre-check at the dir level; the real signal is
// `com/bydeluxe/` inside any top-level jar's central directory,
// which `parse()` confirms when given a `SectorReader`.
// which `parse()` confirms when given a `SectorSource`.
jar::has_any_top_level_jar(udf)
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
jar::for_each_jar(reader, udf, |entry_name, archive| {
if !jar::has_path_prefix(archive, "com/bydeluxe/") {
return None;
+3 -3
View File
@@ -13,7 +13,7 @@
#![allow(dead_code)]
use super::class_reader::ClassFile;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::io::Cursor;
use zip::ZipArchive;
@@ -26,7 +26,7 @@ pub type Jar = ZipArchive<Cursor<Vec<u8>>>;
/// True if `/BDMV/JAR/` contains at least one top-level `.jar` file
/// (not under a subdir). Used by `detect()` in parsers whose real
/// signal lives inside a jar — they can't open the jar without a
/// `SectorReader`, so they use this cheap pre-check and do the real
/// `SectorSource`, so they use this cheap pre-check and do the real
/// `com/<vendor>/` discriminator in `parse()`.
pub fn has_any_top_level_jar(udf: &UdfFs) -> bool {
let Some(jar_dir) = udf.find_dir("/BDMV/JAR") else {
@@ -49,7 +49,7 @@ pub fn has_any_top_level_jar(udf: &UdfFs) -> bool {
///
/// Entries that fail to read from UDF or that aren't valid zips are
/// silently skipped — same defensive shape as the existing dbp parser.
pub fn for_each_jar<R, F>(reader: &mut dyn SectorReader, udf: &UdfFs, mut f: F) -> Option<R>
pub fn for_each_jar<R, F>(reader: &mut dyn SectorSource, udf: &UdfFs, mut f: F) -> Option<R>
where
F: FnMut(&str, &mut Jar) -> Option<R>,
{
+11 -12
View File
@@ -4,7 +4,7 @@
//! To add a new format:
//! 1. Create `src/labels/myformat.rs`
//! 2. Implement `pub fn detect(udf: &UdfFs) -> bool`
//! 3. Implement `pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<Vec<StreamLabel>>`
//! 3. Implement `pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<Vec<StreamLabel>>`
//! 4. Add `mod myformat;` below and one line to `PARSERS` array
mod bdmt;
@@ -18,13 +18,12 @@ pub(crate) mod jar;
mod mpls_universal;
mod paramount;
mod pixelogic;
mod png_filenames;
pub(crate) mod text;
pub mod vocab;
pub(crate) mod xml;
use crate::disc::{DiscTitle, Stream};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
// Re-export bdmt's public type so callers can construct/inspect
@@ -88,7 +87,7 @@ pub enum LabelQualifier {
// to array order on confidence ties.
type DetectFn = fn(&UdfFs) -> bool;
type ParseFn = fn(&mut dyn SectorReader, &UdfFs) -> Option<ParseResult>;
type ParseFn = fn(&mut dyn SectorSource, &UdfFs) -> Option<ParseResult>;
/// Per-parser claim of how reliable its output is. Used by the
/// registry to pick between parsers when more than one matches (e.g.
@@ -176,7 +175,7 @@ const PARSERS: &[(&str, DetectFn, ParseFn)] = &[
/// Search disc for config files, extract labels, apply to streams.
/// This is 100% optional — if anything fails, streams are untouched.
pub fn apply(reader: &mut dyn SectorReader, udf: &UdfFs, titles: &mut [DiscTitle]) {
pub fn apply(reader: &mut dyn SectorSource, udf: &UdfFs, titles: &mut [DiscTitle]) {
let labels = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| extract(reader, udf)))
.unwrap_or_default();
if labels.is_empty() {
@@ -192,7 +191,7 @@ pub fn apply(reader: &mut dyn SectorReader, udf: &UdfFs, titles: &mut [DiscTitle
/// `forced` flag.
///
/// Extracted from `apply()` so the matching logic is unit-testable
/// without needing a SectorReader / UdfFs.
/// without needing a SectorSource / UdfFs.
pub(crate) fn apply_labels(labels: &[StreamLabel], titles: &mut [DiscTitle]) {
for title in titles.iter_mut() {
let mut audio_idx: u16 = 0;
@@ -370,7 +369,7 @@ fn generate_audio_label(
}
}
fn extract(reader: &mut dyn SectorReader, udf: &UdfFs) -> Vec<StreamLabel> {
fn extract(reader: &mut dyn SectorSource, udf: &UdfFs) -> Vec<StreamLabel> {
let mut best: Option<(&'static str, ParseResult)> = None;
for (name, detect, parse) in PARSERS {
if !detect(udf) {
@@ -483,7 +482,7 @@ fn type_tag(t: StreamLabelType) -> u8 {
/// authoring tool left out of the published playlist.
fn append_clpi_orphans(
labels: &mut Vec<StreamLabel>,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf: &UdfFs,
) -> usize {
// Index existing labels by PID — but StreamLabel doesn't carry
@@ -608,7 +607,7 @@ fn append_clpi_orphans(
/// the return shape is richer (includes confidence, all detected
/// parsers, and any parsers that produced empty results).
#[doc(hidden)]
pub fn analyze(reader: &mut dyn SectorReader, udf: &UdfFs) -> LabelAnalysis {
pub fn analyze(reader: &mut dyn SectorSource, udf: &UdfFs) -> LabelAnalysis {
let inventory = jar_inventory(udf);
let mut parsers_detected: Vec<&'static str> = Vec::new();
let mut all_results: Vec<(&'static str, ParseResult)> = Vec::new();
@@ -699,7 +698,7 @@ pub fn analyze(reader: &mut dyn SectorReader, udf: &UdfFs) -> LabelAnalysis {
/// playlist filename. Skipped entries (read error, parse error, no
/// marks) silently dropped — this is a diagnostic field, not a
/// correctness-critical one.
fn collect_chapter_summary(reader: &mut dyn SectorReader, udf: &UdfFs) -> Vec<ChapterSummary> {
fn collect_chapter_summary(reader: &mut dyn SectorSource, udf: &UdfFs) -> Vec<ChapterSummary> {
let Some(playlist_dir) = udf.find_dir("/BDMV/PLAYLIST") else {
return Vec::new();
};
@@ -842,7 +841,7 @@ pub(crate) fn find_jar_file(udf: &UdfFs, filename: &str) -> Option<String> {
/// Read a file from any BDMV/JAR subdirectory by filename.
pub(crate) fn read_jar_file(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
udf: &UdfFs,
filename: &str,
) -> Option<Vec<u8>> {
@@ -1073,7 +1072,7 @@ mod gap_fill_tests {
// ── apply() integration tests ──────────────────────────────────────────────
//
// End-to-end coverage for the apply_labels + fill_defaults pipeline
// without needing a SectorReader / UdfFs. Synthetic DiscTitle +
// without needing a SectorSource / UdfFs. Synthetic DiscTitle +
// StreamLabel inputs, assert on the resulting Stream field values.
#[cfg(test)]
+2 -2
View File
@@ -24,7 +24,7 @@ use super::{
LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType,
vocab::{self, LangInfo},
};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
/// True iff `/BDMV/PLAYLIST/` exists and contains at least one
@@ -45,7 +45,7 @@ pub fn detect(udf: &UdfFs) -> bool {
/// Returns `None` if no labels could be produced (e.g. no .mpls files
/// parsed successfully, or every parsed stream was a type we skip
/// like IG / DV EL).
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let playlist_dir = udf.find_dir("/BDMV/PLAYLIST")?;
// Collect mpls filenames first so we don't hold a borrow on udf
+2 -2
View File
@@ -13,14 +13,14 @@
//! ```
use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, xml};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "playlists.xml")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let data = super::read_jar_file(reader, udf, "playlists.xml")?;
let text = std::str::from_utf8(&data).ok()?;
+2 -2
View File
@@ -9,7 +9,7 @@ use super::{
Confidence, LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelType, text,
vocab,
};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use crate::udf::UdfFs;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -24,7 +24,7 @@ pub fn detect(udf: &UdfFs) -> bool {
super::jar_file_exists(udf, "bluray_project.bin")
}
pub fn parse(reader: &mut dyn SectorReader, udf: &UdfFs) -> Option<ParseResult> {
pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option<ParseResult> {
let data = super::read_jar_file(reader, udf, "bluray_project.bin")?;
// min_len=4 matches the prior local extract_strings impl. The token
// grammar is `{lang3}_{codec?}_{purpose?}_{region?}_` so the
-72
View File
@@ -1,72 +0,0 @@
//! PNG-filename language token parser — stubbed (noop) pending need.
//!
//! ## What this would do
//!
//! Some discs encode per-language menu localization as pre-rendered PNG
//! menu buttons, one per language, with the language token embedded in
//! the filename. Examples observed in the 2026-05-10 corpus:
//!
//! - **disc-01 (The Amateur)** — `<region>_<lang>_<context>_<format>.png`
//! Region prefix: `USA` / `UK` / `JPN` / etc.
//! Lang tokens (3-char, uppercase): `ENG`, `FRC`, `FRP`, `DEU`, `DUT`,
//! `ITA`, `JPN`, `LAS`, `CSP`, `POL`, `CZE` (11 languages)
//!
//! - **disc-09 (Dune orig)** — `<title>_<variant>_<lang>_Composite<N>.png`
//! Lang tokens (3-char, mixed-case): `Eng`, `Ger` (2 languages)
//!
//! ## Why stubbed
//!
//! MPLS already gives per-stream `language` + `coding_type` + stream-type
//! (audio vs subtitle) on every disc. For the 2 unknown-framework discs
//! that PNG filenames would close (disc-01, disc-09), MPLS will produce
//! a strict superset of what filenames could give us, because MPLS knows
//! per-stream attribution while filenames only know "the disc offers
//! these N language buttons."
//!
//! The **only** thing PNG filenames give us that MPLS doesn't is **studio
//! variant disambiguation**:
//! - `FRC` (French Canadian) vs `FRP` (French Parisian) — MPLS just says `fra`
//! - `LAS` (Latin American Spanish) vs `CSP` (Castilian Spanish) — MPLS just says `spa`
//!
//! That's niche enough that it doesn't justify implementing right now.
//! Reactivate this parser only when:
//! 1. We hit a disc where MPLS is malformed/empty AND PNG filenames are
//! the only language hint, OR
//! 2. A downstream consumer needs the studio variant suffix for output
//! naming (e.g. `Title (French Canadian).mkv` vs `Title (French).mkv`).
//!
//! ## When reactivating
//!
//! Implement `parse` to:
//! 1. Iterate top-level PNG paths in `/BDMV/JAR/` (and `<id>/` subdirs).
//! 2. Tokenize each filename on `_` / `-` / `.`
//! 3. Match each token against an alias table:
//! - ISO 639-1 / 639-2 standard codes
//! - Studio variants: `FRC`/`FRP` → `fra-CA`/`fra-FR`,
//! `LAS`/`CSP` → `spa-419`/`spa-ES`,
//! mixed-case shortforms `Eng`/`Ger`/`Fra`/`Spa`/`Jpn` → ISO 639-2
//! - Country prefix filter: drop `USA`/`UK`/`JPN`/`AUS`/`GER`/`FR` when
//! they appear in position 0 (those are region markers, not langs).
//! 4. Deduplicate. Confidence stays `Low` because we still don't know
//! per-stream codec or audio/subtitle attribution.
//!
//! Wire as ENRICHMENT after MPLS in `mod.rs::analyze`, not as a primary
//! parser: PNG filenames upgrade `lang=fra` to `lang=fra-CA` when both
//! sources agree on the disc; they should never overwrite MPLS data.
use super::ParseResult;
use crate::sector::SectorReader;
use crate::udf::UdfFs;
/// Stub: returns false so the dispatcher never calls `parse`. Reactivate
/// by checking for the patterns described in the module docs.
#[allow(dead_code)] // module-level noop, not wired into PARSERS until needed
pub fn detect(_udf: &UdfFs) -> bool {
false
}
/// Stub: returns None. See module docs for the implementation sketch.
#[allow(dead_code)] // module-level noop, not wired into PARSERS until needed
pub fn parse(_reader: &mut dyn SectorReader, _udf: &UdfFs) -> Option<ParseResult> {
None
}
+20 -24
View File
@@ -82,7 +82,7 @@ pub mod event;
pub mod halt;
pub(crate) mod identity;
pub(crate) mod ifo;
pub(crate) mod io;
pub mod io;
pub mod keydb;
pub mod labels;
pub(crate) mod mpls;
@@ -128,16 +128,19 @@ pub use halt::Halt;
// Generic bounded producer/consumer primitive used by sweep, patch, and
// mux to overlap reads with writes via a dedicated consumer thread.
// `Pipeline::spawn(depth, sink)` spawns the consumer; `pipe.send(item)`
// `Pipeline::spawn(name, depth, sink)` spawns a named consumer; `pipe.send(item)`
// pushes one item with back-pressure; `pipe.finish()` joins the
// consumer and surfaces its `close()` output. Callers implement `Sink`
// to define per-item behaviour and end-of-stream finalisation.
//
// `DEFAULT_PIPELINE_DEPTH` (=4) is the depth sweep + mux use; patch
// uses `WRITE_THROUGH_DEPTH` (=1) so each read fully drains before the
// next can enqueue. Returning `Flow::Stop` from `apply` ends the
// consumer cleanly (still calls `close()`).
pub use io::pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink, WRITE_THROUGH_DEPTH};
// `DEFAULT_PIPELINE_DEPTH` (=4) is for callers without specific needs;
// most should use READ_PIPELINE_DEPTH or WRITE_PIPELINE_DEPTH instead.
// Patch uses `WRITE_THROUGH_DEPTH` (=1). Returning `Flow::Stop` from
// `apply` ends the consumer cleanly (still calls `close()`).
pub use io::pipeline::{
DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, READ_PIPELINE_DEPTH, Sink, WRITE_PIPELINE_DEPTH,
WRITE_THROUGH_DEPTH,
};
// ─── Drive events (low-level callbacks) ─────────────────────────────────────
pub use event::{Event, EventKind};
@@ -162,8 +165,8 @@ pub use decrypt::{DecryptKeys, decrypt_sectors};
// structured representation; never compare against display strings.
// Note: `disc::Stream` here is the codec enum (audio / video / sub kind)
// — not the `pes::Stream` trait re-exported below as `PesStream`. Two
// different concepts, the same short name; both stay because both are
// load-bearing in their respective domains.
// different concepts, the same short name; the trait gets the `Pes`
// prefix at the crate root to keep both addressable.
pub use disc::{
AacsState, AudioChannels, AudioStream, Clip, Codec, ColorSpace, ContentFormat, DamageSeverity,
Disc, DiscFormat, DiscId, DiscTitle, Extent, FrameRate, HdrFormat, KeySource, LabelPurpose,
@@ -176,7 +179,7 @@ pub use disc::{
// All stream types implement `pes::Stream` — read PES frames from a source,
// write PES frames to a sink. Pick the right type at construction:
//
// - `DiscStream` — physical drive or ISO (any `SectorReader`). Always read.
// - `DiscStream` — physical drive or ISO (any `SectorSource`). Read-only.
// - `MkvStream` — Matroska container. Read on `open()`, write on `create()`.
// - `M2tsStream` — Blu-ray Transport Stream. Read on `open()`, write on `create()`.
// - `NetworkStream` — TCP. Read on `listen()`, write on `connect()`.
@@ -186,14 +189,11 @@ pub use disc::{
// Most consumers use the URL resolvers (`input()` / `output()`) which pick
// the right type from a scheme:// URL. Direct construction is for callers
// that need to wire custom readers (e.g. autorip's drive-session reuse).
// 0.18 trait split: `FrameSource` (read-only) and `FrameSink` (write-only)
// supersede the unified `pes::Stream`. The old `Stream` re-export below
// stays available for the deprecation window — re-exported as
// `PesStream` to disambiguate from `disc::Stream` (the codec-kind enum
// re-exported above), which would otherwise collide at the crate root.
#[allow(deprecated)]
// The trait is re-exported as `PesStream` here to disambiguate from
// `disc::Stream` (the codec-kind enum re-exported above), which would
// otherwise collide at the crate root.
pub use pes::PesFrame;
pub use pes::Stream as PesStream;
pub use pes::{FrameSink, FrameSource, PesFrame};
pub use mux::DiscStream;
pub use mux::M2tsStream;
@@ -206,18 +206,14 @@ pub use mux::{InputOptions, StreamUrl, input, output, parse_url};
// ─── Lower-level surfaces ───────────────────────────────────────────────────
//
// `ScsiTransport` is the platform-abstraction trait Drive uses; expose for
// out-of-tree platform backends. `SectorSource` / `SectorSink` are the 0.18
// out-of-tree platform backends. `SectorSource` / `SectorSink` are the
// direction-typed read/write traits; `FileSectorSource` and `FileSectorSink`
// are the ISO-on-disk implementations. [`DecryptingSectorSource`] is the
// single decrypt-on-read decorator (AACS / CSS / none) — wrap any
// `SectorSource` to get plaintext sectors out. The legacy `SectorReader` /
// `FileSectorReader` names stay re-exported through the 0.18 migration
// window so existing call sites compile unchanged; a blanket impl makes
// every `SectorReader` automatically usable as a `SectorSource`.
// `SectorSource` to get plaintext sectors out.
pub use scsi::{DriveInfo, ScsiSense, ScsiTransport, drive_has_disc, list_drives};
pub use sector::{
DecryptingSectorSource, FileSectorReader, FileSectorSink, FileSectorSource, SectorReader,
SectorSink, SectorSource,
DecryptingSectorSource, FileSectorSink, FileSectorSource, SectorSink, SectorSource,
};
pub use speed::DriveSpeed;
pub use udf::{UdfFs, read_filesystem};
+40 -32
View File
@@ -1,6 +1,6 @@
//! DiscStream — read any disc (physical drive or ISO file) → PES frames.
//!
//! One stream type for all disc sources. The source is a SectorReader —
//! One stream type for all disc sources. The source is a SectorSource —
//! Drive (hardware) or IsoSectorReader (file). DiscStream doesn't care.
//!
//! Read-only. For disc→ISO (raw sector copy), use `Disc::copy()`.
@@ -9,7 +9,7 @@ use crate::disc::{Disc, DiscTitle, Extent};
use crate::drive::extract_scsi_context;
use crate::event::{BatchSizeReason, Event, EventKind};
use crate::halt::Halt;
use crate::sector::{DecryptingSectorSource, SectorReader, SectorSource};
use crate::sector::{DecryptingSectorSource, SectorSource};
use std::io;
use std::sync::Arc;
use std::sync::atomic::AtomicBool;
@@ -97,7 +97,7 @@ impl AdaptiveBatch {
/// Disc stream. Reads sectors from any source → PES frames.
///
/// Sources: physical drive, ISO file, or any SectorReader.
/// Sources: physical drive, ISO file, or any SectorSource.
/// Decrypt, demux, and codec parsing happen internally.
pub struct DiscStream {
/// Underlying sector source wrapped in the 0.18
@@ -105,7 +105,7 @@ pub struct DiscStream {
/// call yields plaintext, so `fill_extents` no longer needs an
/// inline `decrypt::decrypt_sectors` step. `DecryptKeys::None`
/// (raw / unencrypted disc) makes the decorator a pass-through.
reader: DecryptingSectorSource<Box<dyn SectorReader>>,
reader: DecryptingSectorSource<Box<dyn SectorSource>>,
title: DiscTitle,
disc: Option<Disc>,
/// Mirror of the keys handed in at construction. The decorator
@@ -160,11 +160,11 @@ pub struct DiscStream {
impl DiscStream {
/// Create a disc stream from any sector reader.
///
/// Works with physical drives and ISO files — both implement SectorReader.
/// Works with physical drives and ISO files — both implement SectorSource.
/// The caller opens the source, scans for titles/keys, and passes them in.
/// The stream handles demuxing, decryption, and codec parsing internally.
pub fn new(
reader: Box<dyn SectorReader>,
reader: Box<dyn SectorSource>,
title: DiscTitle,
decrypt_keys: crate::decrypt::DecryptKeys,
batch_sectors: u16,
@@ -173,6 +173,13 @@ impl DiscStream {
let extents = title.extents.clone();
let bytes_total_extents: u64 = extents.iter().map(|e| e.sector_count as u64 * 2048).sum();
// Debug log reader type at construction — critical for diagnosing mux reading from drive instead of ISO
tracing::debug!(
target: "mux",
"DiscStream constructed with reader type: {}",
std::any::type_name::<dyn SectorSource>()
);
let mut pids = Vec::new();
let mut parsers = Vec::new();
let mut pid_to_track = Vec::new();
@@ -315,10 +322,19 @@ impl DiscStream {
// this loop can spend minutes shrinking and skipping sectors; without
// the check, Stop wouldn't take effect until the outer PES read() loop
// finally emits a frame, which may never happen.
let start_time = std::time::Instant::now();
loop {
if self.is_halted() {
return Err(crate::error::Error::Halted.into());
}
// Debug: log slow reads during mux — helps diagnose stalls
if cfg!(debug_assertions) && start_time.elapsed().as_secs() > 5 {
tracing::debug!(target: "mux", "fill_extents waiting at LBA {} ({}s elapsed, sectors={})", lba, start_time.elapsed().as_secs(), remaining);
}
let mut sectors = remaining.min(self.adaptive.current() as u32) as u16;
// Align to 3-sector AACS units when possible. Partial units at
// extent boundaries are safely handled by decrypt_sectors().
@@ -387,7 +403,6 @@ impl DiscStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for DiscStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
if let Some(frame) = self.pending_frames.pop_front() {
@@ -568,31 +583,26 @@ impl crate::pes::Stream for DiscStream {
#[cfg(test)]
mod tests {
//! `DiscStream` is the only meaningful `FrameSource` impl in tree (every
//! other concrete `pes::Stream` impl in `mux/*` is a sink). The 0.18
//! round-1 blanket `impl<T: pes::Stream + Send> pes::FrameSource for T`
//! covers `DiscStream` for free as long as it is `Send`. These tests
//! lock that down: a static `Send` assertion plus a `Box<dyn FrameSource>`
//! round trip exercising every `FrameSource` method through the trait
//! object, so future Send-breaking edits to `DiscStream`'s interior
//! types fail at compile time and the trait-bridge dispatch is verified
//! at runtime.
#![allow(deprecated)] // exercising the 0.18 deprecation-window blanket bridge.
//! `DiscStream` is the only read-only `Stream` impl in tree (every
//! other concrete impl in `mux/*` is bidirectional or write-only).
//! These tests lock down a static `Send` assertion plus a
//! `Box<dyn Stream>` round trip exercising every method through the
//! trait object, so future Send-breaking edits to `DiscStream`'s
//! interior types fail at compile time.
use super::*;
use crate::disc::{ContentFormat, DiscTitle};
use crate::pes::FrameSource;
use crate::pes::Stream;
/// Static-assert `DiscStream: Send`. The blanket
/// `impl<T: pes::Stream + Send> pes::FrameSource for T` only fires for
/// `Send` types — if a future field on `DiscStream` is non-`Send` (e.g.
/// a `Box<dyn Read>` instead of `Box<dyn SectorReader>`), this fails
/// Static-assert `DiscStream: Send`. The `Stream` trait has `Send` as a
/// supertrait — if a future field on `DiscStream` is non-`Send` (e.g.
/// a `Box<dyn Read>` instead of `Box<dyn SectorSource>`), this fails
/// at compile time, before the runtime trait-object test below.
fn _assert_disc_stream_is_send() {
fn requires_send<T: Send>() {}
requires_send::<DiscStream>();
}
/// Trivial `SectorReader` that yields zeroed sectors. Empty title means
/// Trivial `SectorSource` that yields zeroed sectors. Empty title means
/// the demuxer produces no PES frames, so `read()` walks the extents to
/// EOF and returns `Ok(None)`. That's enough to exercise the trait-object
/// dispatch — the goal here is the bridge, not the demuxer.
@@ -600,7 +610,7 @@ mod tests {
capacity: u32,
}
impl crate::sector::SectorReader for ZeroReader {
impl crate::sector::SectorSource for ZeroReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -613,7 +623,7 @@ mod tests {
Ok(bytes)
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -628,13 +638,11 @@ mod tests {
}
}
/// Smallest credible witness that `DiscStream` flows through the
/// `FrameSource` blanket impl: build a `Box<dyn FrameSource>`, drive
/// `read()` to EOF, exercise `info()` / `headers_ready()` /
/// `codec_private()` through the trait object. The trait-bridge
/// correctness is what's being verified — not demuxer behaviour.
/// Smallest credible witness that `DiscStream` flows through `dyn Stream`:
/// build a `Box<dyn Stream>`, drive `read()` to EOF, exercise `info()` /
/// `headers_ready()` / `codec_private()` through the trait object.
#[test]
fn frame_source_via_dyn_object() {
fn stream_via_dyn_object() {
let reader = ZeroReader { capacity: 8 };
let title = synthetic_title(8);
let stream = DiscStream::new(
@@ -645,7 +653,7 @@ mod tests {
ContentFormat::BdTs,
);
let mut src: Box<dyn FrameSource> = Box::new(stream);
let mut src: Box<dyn Stream> = Box::new(stream);
// Empty-title fixture has no streams configured, so headers are
// trivially ready and codec_private() yields nothing on track 0.
+6 -6
View File
@@ -1,10 +1,10 @@
//! ISO sector reader — file-backed SectorReader for Blu-ray ISO images.
//! ISO sector reader — file-backed SectorSource for Blu-ray ISO images.
//!
//! An ISO is a flat image of 2048-byte sectors. Sector N starts at byte offset N * 2048.
//! Used by DiscStream::open_iso() and Disc::scan_image().
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;
@@ -32,12 +32,12 @@ impl IsoSectorReader {
Ok(Self { file, capacity })
}
pub fn capacity(&self) -> u32 {
pub fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
impl SectorReader for IsoSectorReader {
impl SectorSource for IsoSectorReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -73,7 +73,7 @@ mod tests {
std::fs::write(&dir, &data).unwrap();
let mut reader = IsoSectorReader::open(dir.to_str().unwrap()).unwrap();
assert_eq!(reader.capacity(), 4);
assert_eq!(reader.capacity_sectors(), 4);
let mut buf = [0u8; 2048];
reader.read_sectors(0, 1, &mut buf, true).unwrap();
@@ -94,7 +94,7 @@ mod tests {
std::fs::write(&dir, &data).unwrap();
let reader = IsoSectorReader::open(dir.to_str().unwrap()).unwrap();
assert_eq!(reader.capacity(), 10);
assert_eq!(reader.capacity_sectors(), 10);
std::fs::remove_file(&dir).ok();
}
-25
View File
@@ -173,7 +173,6 @@ impl M2tsStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for M2tsStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
if let Some(frame) = self.pending_frames.pop_front() {
@@ -284,27 +283,3 @@ impl crate::pes::Stream for M2tsStream {
true
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `M2tsStream::create(writer, title)` to construct the write half;
/// calling `FrameSink::write` on an `M2tsStream::open(reader)` instance
/// returns `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for M2tsStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
-30
View File
@@ -92,7 +92,6 @@ impl MkvStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for MkvStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
let rs = match self.mode {
@@ -189,35 +188,6 @@ impl crate::pes::Stream for MkvStream {
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `MkvStream::create(writer, title)` to construct the write half;
/// calling `FrameSink::write` on a `MkvStream::open(reader)` instance returns
/// `StreamReadOnly`. `finish` is where the Cues index is written, so it must
/// be called for the resulting MKV to be seekable.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for MkvStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
// The inner `MkvMuxer` is owned via `Option`, and `Stream::finish`
// already takes it via `Option::take()` — moving `*self` out of the
// box hands it the same field by-value, so the muxer's own
// by-value `finish()` runs correctly.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
// ── MKV header parsing (read side) ────────────────────────────
/// Returns (DiscTitle, codec_privates: Vec<(track_number, codec_private_bytes)>)
-26
View File
@@ -70,7 +70,6 @@ impl NetworkStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for NetworkStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
match &mut self.mode {
@@ -109,30 +108,6 @@ impl crate::pes::Stream for NetworkStream {
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `NetworkStream::connect(addr).meta(title)` to construct the write
/// half; calling `FrameSink::write` on `NetworkStream::listen(addr)` returns
/// `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for NetworkStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
// NetworkStream is PES-only — no IOStream/Read/Write byte interface.
#[cfg(test)]
@@ -182,7 +157,6 @@ mod tests {
#[test]
#[ignore] // Requires TCP; may be flaky in CI environments
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
fn network_pes_roundtrip() {
use crate::pes;
+4 -34
View File
@@ -16,7 +16,6 @@ impl NullStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSink in follow-up commit.
impl crate::pes::Stream for NullStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
Ok(None)
@@ -32,43 +31,16 @@ impl crate::pes::Stream for NullStream {
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// The deprecation-window callers eventually migrate; this impl exists so
/// new callers can target `FrameSink` without waiting for the rest of the
/// migration to complete.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for NullStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::pes::FrameSink;
use crate::pes::Stream;
/// Smallest credible witness that the new FrameSink impl on a concrete
/// `mux/*` sink works through the trait object: build a boxed
/// `dyn FrameSink`, write a frame, finish it. The trait-bridge correctness
/// is what's being verified — not NullStream-specific behaviour.
/// Verify NullStream routes through the `Stream` trait object cleanly.
#[test]
fn frame_sink_via_dyn_object_writes_and_finishes() {
fn stream_via_dyn_object_writes_and_finishes() {
let title = DiscTitle::empty();
let mut sink: Box<dyn FrameSink> = Box::new(NullStream::new(&title));
let mut sink: Box<dyn Stream> = Box::new(NullStream::new(&title));
let frame = crate::pes::PesFrame {
track: 0,
@@ -77,9 +49,7 @@ mod tests {
data: vec![0x01, 0x02, 0x03],
};
sink.write(&frame).unwrap();
// info() routes through the trait object.
let _ = sink.info();
// finish() consumes the Box<Self> — must compile and run.
sink.finish().unwrap();
}
}
+8 -7
View File
@@ -167,7 +167,6 @@ pub struct InputOptions {
}
/// Open a PES input stream (produces PES frames).
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource in follow-up commit.
pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::Stream>> {
let parsed = parse_url(url);
match parsed {
@@ -187,7 +186,7 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
None => crate::disc::ScanOptions::default(),
};
let mut reader = super::iso::IsoSectorReader::open(&path.to_string_lossy())?;
let capacity = reader.capacity();
let capacity = reader.capacity_sectors();
let disc = crate::disc::Disc::scan_image(&mut reader, capacity, &scan_opts)
.map_err(|e| -> io::Error { e.into() })?;
if disc.titles.is_empty() {
@@ -207,7 +206,8 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
// ISO file: use large batch size (16 MB) — sequential read from fast storage, no bad sectors.
// Physical drives need small batches for adaptive error handling and retry logic.
const ISO_MUX_BATCH_SECTORS: u16 = 8192;
let mut stream = DiscStream::new(Box::new(reader), title, keys, ISO_MUX_BATCH_SECTORS, format);
let mut stream =
DiscStream::new(Box::new(reader), title, keys, ISO_MUX_BATCH_SECTORS, format);
if opts.raw {
stream.set_raw();
}
@@ -238,7 +238,6 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
}
/// Open a PES output stream (consumes PES frames).
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSink in follow-up commit.
pub fn output(
url: &str,
title: &crate::disc::DiscTitle,
@@ -251,11 +250,13 @@ pub fn output(
// writeback) so a UHD-scale MKV mux to slow / network-attached
// staging doesn't hit the dirty-page burst pathology that
// sweep already side-steps. BufWriter sits on top to coalesce
// mux's many small EBML element writes.
// mux's many small EBML element writes. Pre-reserve the
// target's worth of extents on Linux via fallocate(KEEP_SIZE)
// to reduce extent fragmentation during the mux.
let writer: Box<dyn super::WriteSeek + Send> =
Box::new(std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
crate::io::WritebackFile::create_with_size_hint(path, title.size_bytes)?,
));
Ok(Box::new(MkvStream::create(writer, title)?))
}
@@ -263,7 +264,7 @@ pub fn output(
validate_file_path(path, "m2ts")?;
let writer = std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
crate::io::WritebackFile::create_with_size_hint(path, title.size_bytes)?,
);
Ok(Box::new(M2tsStream::create(writer, title)?))
}
-24
View File
@@ -60,7 +60,6 @@ impl StdioStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for StdioStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
self.ensure_header_read()?;
@@ -105,26 +104,3 @@ impl crate::pes::Stream for StdioStream {
self.header_read || self.writer.is_some()
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `StdioStream::output(title)` to construct the write half; calling
/// `FrameSink::write` on a `StdioStream::input()` returns `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for StdioStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
+44 -233
View File
@@ -78,19 +78,20 @@ impl PesFrame {
}
}
/// Deprecated; use [`FrameSource`] for read-only sources or [`FrameSink`]
/// for write-only sinks. The runtime direction-error semantics
/// (`StreamReadOnly` / `StreamWriteOnly` from a wrong-direction call) are
/// removed in 0.18 — direction is type-checked.
#[deprecated(
since = "0.18.0",
note = "use FrameSource (read-only) or FrameSink (write-only) instead"
)]
pub trait Stream {
/// Read the next frame, or `Ok(None)` at end of stream.
/// A PES frame stream. One trait per format — same type opens for read
/// (`open()` / `listen()` / `input()`) or write (`create()` / `connect()` /
/// `output()`). Calling the wrong-direction method returns a typed
/// `StreamReadOnly` / `StreamWriteOnly` error.
///
/// `Send` is required so streams can move across the producer/consumer
/// threads in autorip's mux pipeline.
pub trait Stream: Send {
/// Read the next frame, or `Ok(None)` at end of stream. Returns
/// `StreamWriteOnly` on a stream opened for writing.
fn read(&mut self) -> std::io::Result<Option<PesFrame>>;
/// Write a frame to the sink.
/// Write a frame to the sink. Returns `StreamReadOnly` on a stream
/// opened for reading.
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()>;
/// Finalize the stream: flush buffered frames, write any container
@@ -116,115 +117,10 @@ pub trait Stream {
}
}
/// Read-only source of PES frames.
///
/// Replaces the read half of the deprecated [`Stream`] trait. Implementors
/// produce frames via [`read`](FrameSource::read) and never accept writes —
/// passing a `FrameSource` where a sink is expected is a compile error,
/// not a runtime `E9001`.
///
/// `info()` returns the source's `DiscTitle` metadata (track list, codec
/// info, duration). It must be stable across the lifetime of the source.
///
/// `codec_private(track)` exposes per-track codec initialization data
/// (H.264 SPS/PPS, HEVC VPS/SPS/PPS, AC-3 fscod, etc.) that downstream
/// muxers may need before any frame is written. `headers_ready()` returns
/// false until enough input frames have been seen to populate every video
/// track's codec-private blob — callers buffer frames they read until
/// `headers_ready()` returns true.
pub trait FrameSource: Send {
/// Read the next frame, or `Ok(None)` at end of stream.
fn read(&mut self) -> std::io::Result<Option<PesFrame>>;
/// Source metadata. Stable across reads — implementors must return a
/// consistent reference for the lifetime of the source.
fn info(&self) -> &crate::disc::DiscTitle;
/// Codec initialization data for a track (SPS/PPS, AC-3 fscod, etc.).
/// `None` for tracks that don't need codec_private (raw passthrough).
fn codec_private(&self, _track: usize) -> Option<Vec<u8>> {
None
}
/// True when `codec_private` is available for every video track —
/// callers buffer input frames until this flips, since some output
/// formats (MKV) can't write frames without codec init data.
fn headers_ready(&self) -> bool {
true
}
}
/// Write-only sink of PES frames.
///
/// Replaces the write half of the deprecated [`Stream`] trait. Implementors
/// accept frames via [`write`](FrameSink::write) and finalize via
/// [`finish`](FrameSink::finish) — passing a `FrameSink` where a source is
/// expected is a compile error, not a runtime `E9000`.
///
/// `finish` takes `Box<Self>` (rather than `&mut self` like `Stream::finish`)
/// so that finalization is a one-shot terminal operation: callers cannot use
/// the sink after `finish` returns. This is the standard idiom for terminal
/// methods on `dyn Trait` objects.
pub trait FrameSink: Send {
/// Write a frame to the sink.
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()>;
/// Finalize the sink: flush buffered frames, write any container index
/// (e.g. MKV `Cues`), close the underlying file/socket. Consumes the
/// sink — callers cannot use it afterwards.
fn finish(self: Box<Self>) -> std::io::Result<()>;
/// Sink metadata. Stable across writes — implementors must return a
/// consistent reference for the lifetime of the sink.
fn info(&self) -> &crate::disc::DiscTitle;
}
// Bridge: any **`Send`** type implementing the deprecated `Stream` trait
// is also a `FrameSource`. This lets existing concrete `Stream` impls in
// `mux/*` satisfy `FrameSource` bounds without per-type migration during
// the 0.18 deprecation window.
//
// **Send caveat (read me before tightening `Stream` itself).** This
// blanket carries a `T: Send` bound rather than promoting `Send` to a
// supertrait of `Stream`, because not every concrete in-tree `Stream`
// impl is `Send`: `MkvStream` and `M2tsStream` carry `Box<dyn Read>`
// and `Box<dyn Write>` fields whose trait objects don't include `Send`.
// Adding `Stream: Send` would force a wider audit (every `Box<dyn Read>`
// becomes `Box<dyn Read + Send>`) than this commit is taking on, and
// the type-level migration target is `FrameSource` / `FrameSink`
// directly anyway. Consequence: coercing a non-Send `Box<dyn Stream>`
// (the return shape of `crate::mux::input` / `output`) to
// `Box<dyn FrameSource>` will fail with a `T: Send` trait-bound error.
// The fix on the consumer side is to construct a Send-compliant
// `FrameSource` / `FrameSink` directly rather than relying on this
// bridge for non-Send streams.
//
// Note: `FrameSink` cannot be blanket-impl'd from `Stream` because
// `Stream::finish` takes `&mut self` while `FrameSink::finish` takes
// `Box<Self>`; concrete types will be migrated in a follow-up commit.
#[allow(deprecated)]
impl<T: Stream + Send + ?Sized> FrameSource for T {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
<Self as Stream>::read(self)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as Stream>::info(self)
}
fn codec_private(&self, track: usize) -> Option<Vec<u8>> {
<Self as Stream>::codec_private(self, track)
}
fn headers_ready(&self) -> bool {
<Self as Stream>::headers_ready(self)
}
}
/// Wraps any output stream and counts bytes written.
///
/// Progress tracking is a CLI concern — streams don't know their size.
/// Wrap the output with CountingStream, then query bytes_written().
/// Wrap the output with `CountingStream`, then query `bytes_written()`.
///
/// ```text
/// let mut output = CountingStream::new(libfreemkv::output(dest, &title)?);
@@ -233,13 +129,11 @@ impl<T: Stream + Send + ?Sized> FrameSource for T {
/// let pct = output.bytes_written() as f64 / total as f64;
/// }
/// ```
#[allow(deprecated)]
pub struct CountingStream {
inner: Box<dyn Stream>,
written: u64,
}
#[allow(deprecated)]
impl CountingStream {
pub fn new(inner: Box<dyn Stream>) -> Self {
Self { inner, written: 0 }
@@ -251,7 +145,6 @@ impl CountingStream {
}
}
#[allow(deprecated)]
impl Stream for CountingStream {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
self.inner.read()
@@ -293,88 +186,35 @@ mod tests {
}
}
/// Direct `FrameSource` impl (not via the deprecated `Stream` blanket).
struct MockSource {
frames: std::vec::IntoIter<PesFrame>,
/// Minimal in-memory `Stream` for trait-shape tests. `read` replays
/// pre-seeded frames; `write` collects them.
struct MockStream {
read_queue: std::vec::IntoIter<PesFrame>,
written: Vec<PesFrame>,
title: DiscTitle,
}
impl MockSource {
fn new(frames: Vec<PesFrame>) -> Self {
impl MockStream {
fn new(read_frames: Vec<PesFrame>) -> Self {
Self {
frames: frames.into_iter(),
read_queue: read_frames.into_iter(),
written: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl FrameSource for MockSource {
impl Stream for MockStream {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
Ok(self.frames.next())
Ok(self.read_queue.next())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
/// Direct `FrameSink` impl (not via the deprecated `Stream` blanket).
struct MockSink {
collected: Vec<PesFrame>,
title: DiscTitle,
}
impl MockSink {
fn new() -> Self {
Self {
collected: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl FrameSink for MockSink {
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()> {
self.collected.push(frame.clone());
self.written.push(frame.clone());
Ok(())
}
fn finish(self: Box<Self>) -> std::io::Result<()> {
// Drop self; in real sinks this is where flush/fsync/close happens.
Ok(())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
/// Variant of `MockSink` whose `finish` returns the collected frames so
/// the test can assert on them after the consuming `Box<Self>` call.
struct CollectingSink {
collected: Vec<PesFrame>,
title: DiscTitle,
}
impl CollectingSink {
fn new() -> Self {
Self {
collected: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl FrameSink for CollectingSink {
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()> {
self.collected.push(frame.clone());
Ok(())
}
fn finish(self: Box<Self>) -> std::io::Result<()> {
// Real CollectingSink consumers would expose `take()` before
// finish; this trait method just confirms the boxed signature
// compiles and runs.
fn finish(&mut self) -> std::io::Result<()> {
Ok(())
}
@@ -384,72 +224,43 @@ mod tests {
}
#[test]
fn frame_source_yields_frames_then_eof() {
fn stream_read_yields_frames_then_eof() {
let frames = vec![make_frame(0, 0), make_frame(1, 1_000), make_frame(0, 2_000)];
let mut src = MockSource::new(frames.clone());
let mut s = MockStream::new(frames.clone());
let f0 = src.read().unwrap().expect("first frame");
let f0 = s.read().unwrap().expect("first frame");
assert_eq!(f0.track, frames[0].track);
assert_eq!(f0.pts, frames[0].pts);
assert!(f0.keyframe);
let f1 = src.read().unwrap().expect("second frame");
assert_eq!(f1.track, frames[1].track);
let f1 = s.read().unwrap().expect("second frame");
assert_eq!(f1.pts, frames[1].pts);
let f2 = src.read().unwrap().expect("third frame");
assert_eq!(f2.track, frames[2].track);
let f2 = s.read().unwrap().expect("third frame");
assert_eq!(f2.pts, frames[2].pts);
assert!(src.read().unwrap().is_none());
assert!(src.read().unwrap().is_none()); // idempotent at EOF
assert!(s.read().unwrap().is_none());
assert!(s.read().unwrap().is_none()); // idempotent at EOF
}
#[test]
fn frame_sink_collects_then_finishes() {
let mut sink = MockSink::new();
fn stream_write_collects_then_finishes() {
let mut s = MockStream::new(Vec::new());
let frames = [make_frame(0, 0), make_frame(1, 100), make_frame(2, 200)];
for f in &frames {
sink.write(f).unwrap();
s.write(f).unwrap();
}
assert_eq!(sink.collected.len(), 3);
assert_eq!(sink.collected[0].pts, 0);
assert_eq!(sink.collected[1].pts, 100);
assert_eq!(sink.collected[2].pts, 200);
// Box-and-finish — the `self: Box<Self>` shape must compile and run.
Box::new(sink).finish().unwrap();
assert_eq!(s.written.len(), 3);
s.finish().unwrap();
}
#[test]
fn frame_sink_via_dyn_object() {
let frames = [make_frame(0, 0), make_frame(0, 33)];
let mut sink: Box<dyn FrameSink> = Box::new(CollectingSink::new());
for f in &frames {
sink.write(f).unwrap();
}
// info() routes through the trait object.
let _ = sink.info();
sink.finish().unwrap();
}
/// The deprecated blanket impl: any concrete `Stream` should also act as
/// a `FrameSource`. `NullStream` has the simplest constructor of every
/// concrete `Stream` impl in `mux/*`, so it's the smallest credible
/// witness that the bridge compiles and dispatches correctly.
#[test]
#[allow(deprecated)]
fn deprecated_stream_satisfies_frame_source() {
let title = DiscTitle::empty();
let mut null = crate::mux::NullStream::new(&title);
let src: &mut dyn FrameSource = &mut null;
// NullStream::read returns Ok(None) — it's a write-only sink.
assert!(src.read().unwrap().is_none());
// info() forwards through the blanket impl.
let _ = src.info();
assert!(src.headers_ready());
fn stream_via_dyn_object() {
let mut s: Box<dyn Stream> = Box::new(MockStream::new(vec![make_frame(0, 0)]));
let frame = s.read().unwrap().expect("first frame");
s.write(&frame).unwrap();
let _ = s.info();
s.finish().unwrap();
}
}
+73 -8
View File
@@ -13,7 +13,13 @@ use std::path::Path;
use crate::error::{Error, Result};
use super::{SectorReader, SectorSink};
use super::{SectorSink, SectorSource};
/// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop on the
/// read side. Mirrors the writeback chunk size so the read-side
/// page cache stays bounded the same way the write side does.
#[cfg(target_os = "linux")]
const READ_DROP_CHUNK_BYTES: u64 = 32 * 1024 * 1024;
/// SectorSource backed by a file (ISO image).
///
@@ -27,6 +33,14 @@ use super::{SectorReader, SectorSink};
pub struct FileSectorSource {
file: File,
capacity: u32,
/// Bytes read since the last `posix_fadvise(DONTNEED)` drop.
/// Only updated on Linux; on other targets it stays at 0.
#[cfg(target_os = "linux")]
bytes_read_since_drop: u64,
/// Byte offset at which the current drop window starts (the
/// next `posix_fadvise(DONTNEED)` call drops from here).
#[cfg(target_os = "linux")]
drop_window_start: u64,
}
impl FileSectorSource {
@@ -45,18 +59,37 @@ impl FileSectorSource {
.into());
}
let capacity = sectors as u32;
Ok(Self { file, capacity })
// Hint sequential access on Linux so the kernel's readahead
// window widens for the ISO sweep. Best-effort: return value
// is ignored. On macOS / Windows this is a no-op.
#[cfg(target_os = "linux")]
{
use std::os::unix::io::AsRawFd;
unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, 0, libc::POSIX_FADV_SEQUENTIAL);
}
}
Ok(Self {
file,
capacity,
#[cfg(target_os = "linux")]
bytes_read_since_drop: 0,
#[cfg(target_os = "linux")]
drop_window_start: 0,
})
}
}
// Implement the legacy `SectorReader` trait. The blanket impl in
// Implement the legacy `SectorSource` trait. The blanket impl in
// `super` produces the `SectorSource` impl automatically — no need
// to write both, and writing both would conflict. This keeps the
// 0.17 method-resolution path intact (callers with `SectorReader`
// 0.17 method-resolution path intact (callers with `SectorSource`
// in scope can still write `fsr.read_sectors(..)` against a
// `FileSectorSource`).
impl SectorReader for FileSectorSource {
fn capacity(&self) -> u32 {
impl SectorSource for FileSectorSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
@@ -75,6 +108,38 @@ impl SectorReader for FileSectorSource {
self.file
.read_exact(&mut buf[..bytes])
.map_err(|e| Error::IoError { source: e })?;
// On Linux, periodically drop the just-read region from the
// page cache to keep cache pressure bounded during multi-GB
// sequential ISO reads. Mirrors the write-side pipeline.
#[cfg(target_os = "linux")]
{
use std::os::unix::io::AsRawFd;
self.bytes_read_since_drop += bytes as u64;
if self.bytes_read_since_drop >= READ_DROP_CHUNK_BYTES {
let drop_start = self.drop_window_start;
let drop_len = self.bytes_read_since_drop;
let t0 = std::time::Instant::now();
unsafe {
libc::posix_fadvise(
self.file.as_raw_fd(),
drop_start as i64,
drop_len as i64,
libc::POSIX_FADV_DONTNEED,
);
}
let elapsed_ms = t0.elapsed().as_millis();
let start_lba = drop_start / 2048;
let end_lba = (drop_start + drop_len) / 2048;
tracing::trace!(
target: "mux",
"FileSectorSource fadvise DONTNEED lba=[{start_lba}..{end_lba}) bytes={drop_len} elapsed_ms={elapsed_ms}"
);
self.drop_window_start = drop_start + drop_len;
self.bytes_read_since_drop = 0;
}
}
Ok(bytes)
}
}
@@ -147,8 +212,8 @@ impl SectorSink for FileSectorSink {
#[cfg(test)]
mod tests {
// Bring the 0.18 trait into scope (not super::*: the super
// module also re-exports the legacy `SectorReader`, and
// having both `SectorReader::read_sectors` and
// module also re-exports the legacy `SectorSource`, and
// having both `SectorSource::read_sectors` and
// `SectorSource::read_sectors` visible would force every
// call site to disambiguate). External consumers see the
// same surface this test exercises.
+60 -149
View File
@@ -1,29 +1,17 @@
//! Sector-level I/O traits.
//!
//! 0.18 splits the unidirectional read trait from a write trait at
//! the sector layer, so the type system catches "wrong direction"
//! mistakes at compile time instead of runtime. See
//! `(internal)/memory/0_18_redesign.md`.
//! The sector layer is direction-typed: [`SectorSource`] reads
//! 2048-byte sectors, [`SectorSink`] writes them. Concrete impls
//! never do both — physical drives are read-only, file-backed
//! ISO images are opened for read OR write at construction time.
//!
//! - [`SectorSource`] reads 2048-byte sectors. Implemented by
//! `Drive` (via the legacy [`SectorReader`] alias) and
//! [`FileSectorSource`] (ISO-backed).
//! - [`SectorSink`] writes 2048-byte sectors. Implemented by
//! [`FileSectorSink`] (ISO-backed) and, in later commits, by
//! sweep/patch consumer adapters.
//! - [`SectorSource`] is implemented by `Drive` (hardware) and
//! [`FileSectorSource`] / `IsoSectorReader` (file-backed).
//! - [`SectorSink`] is implemented by [`FileSectorSink`]
//! (ISO-backed) and sweep/patch consumer adapters.
//! - [`DecryptingSectorSource`] is a decorator that wraps any
//! `SectorSource` and applies the existing AACS / CSS in-place
//! decrypt to plaintext-out.
//!
//! [`SectorReader`] is the 0.17 read trait. It stays on through
//! the 0.18 migration window so existing call sites
//! (`Drive`, `IsoSectorReader`, `BufferedSectorReader`,
//! `DiscStream`, `verify`) compile unchanged. A blanket impl
//! forwards every `SectorReader` impl to `SectorSource`, so new
//! code should target `SectorSource` / `SectorSink` directly. The
//! formal `#[deprecated]` attribute lands once the internal
//! callers have migrated; see the comment on `SectorReader` for
//! why this commit holds it back.
//! `SectorSource` and applies AACS / CSS in-place decrypt to
//! yield plaintext sectors.
pub mod decrypting;
pub mod file;
@@ -32,13 +20,14 @@ use crate::error::Result;
/// Read 2048-byte sectors from a disc, image, or composed source.
///
/// Direction-typed: a `SectorSource` cannot be written to. Wrap the
/// inner source in [`DecryptingSectorSource`] to get plaintext
/// sectors out of an encrypted disc.
/// Wrap the inner source in [`DecryptingSectorSource`] to get
/// plaintext sectors out of an encrypted disc.
pub trait SectorSource: Send {
/// Total capacity in sectors, if known. Returns 0 when unknown
/// Total capacity in sectors, if known. Default `0` = unknown
/// (e.g. live drives that haven't completed `READ CAPACITY` yet).
fn capacity_sectors(&self) -> u32;
fn capacity_sectors(&self) -> u32 {
0
}
/// Read `count` sectors starting at `lba` into `buf`.
/// `buf` must be at least `count * 2048` bytes.
@@ -60,10 +49,52 @@ pub trait SectorSource: Send {
fn set_speed(&mut self, _kbs: u16) {}
}
// Forwarding impls so `Box<dyn SectorSource>` and `&mut dyn SectorSource`
// satisfy the `SectorSource` trait bound when wrapped by generic
// decorators like `DecryptingSectorSource<S: SectorSource>`.
impl SectorSource for Box<dyn SectorSource> {
fn capacity_sectors(&self) -> u32 {
(**self).capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
(**self).read_sectors(lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
(**self).set_speed(kbs)
}
}
impl SectorSource for &mut (dyn SectorSource + '_) {
fn capacity_sectors(&self) -> u32 {
(**self).capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
(**self).read_sectors(lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
(**self).set_speed(kbs)
}
}
/// Write 2048-byte sectors to a disc image or composed sink.
///
/// Direction-typed: a `SectorSink` cannot be read from. The
/// terminal [`finish`] takes `Box<Self>` so it can run on `dyn
/// The terminal [`finish`] takes `Box<Self>` so it can run on `dyn
/// SectorSink` and consume the sink (`fsync` + close).
///
/// [`finish`]: SectorSink::finish
@@ -78,125 +109,5 @@ pub trait SectorSink: Send {
fn finish(self: Box<Self>) -> Result<()>;
}
/// 0.17 read trait. Slated for removal once internal call sites
/// migrate to [`SectorSource`] in follow-up commits; until then
/// it remains the trait that `Drive`, `IsoSectorReader`,
/// `BufferedSectorReader`, and existing `&mut dyn SectorReader`
/// signatures use unchanged.
///
/// New code should implement [`SectorSource`] directly. The
/// blanket impl below makes any `SectorReader` automatically
/// usable wherever a `SectorSource` is expected, so a one-way
/// migration off `SectorReader` is possible per-callsite without
/// touching the impls.
//
// NOTE: not marked `#[deprecated]` in this commit — `cargo clippy
// -- -D warnings` (the CI gauntlet) treats deprecation as an
// error, and the existing `Drive` / `udf::BufferedSectorReader` /
// `mux::DiscStream` / `verify` call sites all go through this
// trait. The deprecation attribute lands together with the
// migration commits that move those call sites to
// `SectorSource`. The behavioural contract — "this trait is
// going away in 0.18" — is documented above and tracked in
// `(internal)/memory/0_18_redesign.md`.
pub trait SectorReader: Send {
/// Read `count` sectors starting at `lba` into `buf`.
/// See [`SectorSource::read_sectors`] for semantics.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Total capacity in sectors, if known.
fn capacity(&self) -> u32 {
0
}
fn set_speed(&mut self, _kbs: u16) {}
}
// Blanket impl: anything implementing the legacy `SectorReader`
// trait automatically satisfies `SectorSource`. This is what keeps
// existing impls (`Drive`, `IsoSectorReader`, `BufferedSectorReader`,
// etc.) compiling without source changes during the migration. The
// reverse direction (impl SectorReader for SectorSource) is
// intentionally NOT provided — new code targets the new trait.
impl<T: SectorReader + ?Sized> SectorSource for T {
fn capacity_sectors(&self) -> u32 {
<T as SectorReader>::capacity(self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<T as SectorReader>::read_sectors(self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<T as SectorReader>::set_speed(self, kbs)
}
}
// Forwarding impls so callers can wrap `&mut dyn SectorReader` /
// `Box<dyn SectorReader>` in [`DecryptingSectorSource`] without
// having to unbox or re-borrow inside the lib's hot paths. The
// generic `&mut T` / `Box<T>` blankets would conflict with the
// `SectorReader → SectorSource` blanket above (a downstream crate
// could `impl SectorReader for &mut U`); the specific
// `dyn SectorReader` instantiations are unambiguous because
// `SectorReader` is the very trait whose `dyn` we're targeting.
impl SectorSource for &mut (dyn SectorReader + '_) {
fn capacity_sectors(&self) -> u32 {
<dyn SectorReader as SectorReader>::capacity(*self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<dyn SectorReader as SectorReader>::read_sectors(*self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<dyn SectorReader as SectorReader>::set_speed(*self, kbs)
}
}
impl SectorSource for Box<dyn SectorReader> {
fn capacity_sectors(&self) -> u32 {
<dyn SectorReader as SectorReader>::capacity(&**self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<dyn SectorReader as SectorReader>::read_sectors(&mut **self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<dyn SectorReader as SectorReader>::set_speed(&mut **self, kbs)
}
}
pub use decrypting::DecryptingSectorSource;
pub use file::{FileSectorSink, FileSectorSource};
// Backwards-compat alias for the public API. `FileSectorReader` is
// the 0.17 name; new code uses `FileSectorSource`. Both point at
// the same type. The `#[deprecated]` attribute lands together with
// the migration commits that retire the alias from internal uses.
pub type FileSectorReader = FileSectorSource;
+17 -17
View File
@@ -19,7 +19,7 @@
//! BD-ROM Part 3 — Blu-ray filesystem profile
use crate::error::{Error, Result};
use crate::sector::SectorReader;
use crate::sector::SectorSource;
/// A UDF filesystem parsed from disc.
#[derive(Debug)]
@@ -84,7 +84,7 @@ impl UdfFs {
/// Reads sector by sector from disc — no buffering.
/// Get the absolute starting LBA of a file on disc.
/// Used by the rip pipeline to locate m2ts content sectors.
pub fn file_start_lba(&self, reader: &mut dyn SectorReader, path: &str) -> Result<u32> {
pub fn file_start_lba(&self, reader: &mut dyn SectorSource, path: &str) -> Result<u32> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
let mut current = &self.root;
for part in &parts[..parts.len() - 1] {
@@ -115,7 +115,7 @@ impl UdfFs {
Ok(self.partition_start + data_lba)
}
pub fn read_file(&self, reader: &mut dyn SectorReader, path: &str) -> Result<Vec<u8>> {
pub fn read_file(&self, reader: &mut dyn SectorSource, path: &str) -> Result<Vec<u8>> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
let mut current = &self.root;
@@ -175,7 +175,7 @@ impl UdfFs {
///
/// Skips: STREAM/ (video), BACKUP/, DUPLICATE/,
/// MKB_RO.inf, ContentHash*, ContentRevocation*
pub fn metadata_sector_ranges(&self, reader: &mut dyn SectorReader) -> Result<Vec<(u32, u32)>> {
pub fn metadata_sector_ranges(&self, reader: &mut dyn SectorSource) -> Result<Vec<(u32, u32)>> {
let mut ranges = Vec::new();
// UDF structure: sector 0 through end of metadata partition
@@ -194,7 +194,7 @@ impl UdfFs {
/// All sector ranges that contain data (metadata + all files including STREAM).
/// For full disc-to-ISO dumps — reads only allocated sectors, skips gaps.
pub fn all_sector_ranges(&self, reader: &mut dyn SectorReader) -> Result<Vec<(u32, u32)>> {
pub fn all_sector_ranges(&self, reader: &mut dyn SectorSource) -> Result<Vec<(u32, u32)>> {
let mut ranges = Vec::new();
// UDF structure sectors
@@ -212,7 +212,7 @@ impl UdfFs {
fn collect_all_file_ranges(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
entry: &DirEntry,
ranges: &mut Vec<(u32, u32)>,
) -> Result<()> {
@@ -238,7 +238,7 @@ impl UdfFs {
fn collect_file_ranges(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
entry: &DirEntry,
ranges: &mut Vec<(u32, u32)>,
) -> Result<()> {
@@ -276,7 +276,7 @@ impl UdfFs {
/// Read an Extended File Entry (tag 266) or File Entry (tag 261)
/// and return its first allocation extent: (data_lba, data_length).
/// The data_lba is partition-relative.
fn read_icb_extent(&self, reader: &mut dyn SectorReader, meta_lba: u32) -> Result<(u32, u32)> {
fn read_icb_extent(&self, reader: &mut dyn SectorSource, meta_lba: u32) -> Result<(u32, u32)> {
let extents = self.read_icb_extents(reader, meta_lba)?;
extents.first().copied().ok_or(Error::DiscRead {
sector: 0,
@@ -290,7 +290,7 @@ impl UdfFs {
/// Handles files with many extents (e.g. 88 GB m2ts files have ~90 extents).
fn read_icb_extents(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
meta_lba: u32,
) -> Result<Vec<(u32, u32)>> {
let mut icb = [0u8; 2048];
@@ -367,7 +367,7 @@ impl UdfFs {
/// Returns Vec of (absolute_lba, sector_count) covering the entire file.
pub fn file_extents(
&self,
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
path: &str,
) -> Result<Vec<(u32, u32)>> {
let parts: Vec<&str> = path.trim_matches('/').split('/').collect();
@@ -417,7 +417,7 @@ impl UdfFs {
/// 3. Metadata partition file → metadata content location
/// 4. FSD → root directory ICB
/// 5. Root directory → file tree
pub fn read_filesystem(reader: &mut dyn SectorReader) -> Result<UdfFs> {
pub fn read_filesystem(reader: &mut dyn SectorSource) -> Result<UdfFs> {
// Step 1: Anchor Volume Descriptor Pointer at sector 256
// ECMA-167 §10.2 — always at sector 256
let mut avdp = [0u8; 2048];
@@ -593,7 +593,7 @@ pub fn read_filesystem(reader: &mut dyn SectorReader) -> Result<UdfFs> {
/// and points to its ICB.
#[allow(clippy::only_used_in_recursion)]
fn read_directory(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
part_start: u32,
meta_start: u32,
meta_lba: u32,
@@ -758,7 +758,7 @@ fn read_directory(
}
/// Read file size (info_length) from an Extended File Entry ICB.
fn read_file_size(reader: &mut dyn SectorReader, meta_start: u32, meta_lba: u32) -> Result<u64> {
fn read_file_size(reader: &mut dyn SectorSource, meta_start: u32, meta_lba: u32) -> Result<u64> {
let mut icb = [0u8; 2048];
read_sector(reader, meta_start + meta_lba, &mut icb)?;
@@ -874,7 +874,7 @@ fn parse_dstring(data: &[u8]) -> String {
/// Each SCSI command has ~500ms overhead on USB drives, so reading 32 sectors
/// at once (one command) is 32x faster than 32 individual reads.
pub(crate) struct BufferedSectorReader<'a> {
inner: &'a mut dyn SectorReader,
inner: &'a mut dyn SectorSource,
cache_start: u32,
cache: Vec<u8>,
cache_sectors: u32,
@@ -884,7 +884,7 @@ pub(crate) struct BufferedSectorReader<'a> {
}
impl<'a> BufferedSectorReader<'a> {
pub(crate) fn new(inner: &'a mut dyn SectorReader, batch: u16) -> Self {
pub(crate) fn new(inner: &'a mut dyn SectorSource, batch: u16) -> Self {
Self {
inner,
cache_start: u32::MAX,
@@ -952,7 +952,7 @@ impl BufferedSectorReader<'_> {
}
}
impl SectorReader for BufferedSectorReader<'_> {
impl SectorSource for BufferedSectorReader<'_> {
fn read_sectors(
&mut self,
lba: u32,
@@ -996,7 +996,7 @@ impl SectorReader for BufferedSectorReader<'_> {
}
}
fn read_sector(reader: &mut dyn SectorReader, lba: u32, buf: &mut [u8]) -> Result<()> {
fn read_sector(reader: &mut dyn SectorSource, lba: u32, buf: &mut [u8]) -> Result<()> {
reader.read_sectors(lba, 1, buf, true)?;
Ok(())
}
+2 -2
View File
@@ -2,7 +2,7 @@
use crate::disc::{Chapter, DiscTitle};
use crate::progress::Progress;
use crate::sector::SectorReader;
use crate::sector::SectorSource;
use std::time::Instant;
/// Health status of a single sector read.
@@ -82,7 +82,7 @@ impl VerifyResult {
/// Reads in batches for speed, falls back to single-sector on failure.
/// The progress callback returns false to request early stop.
pub fn verify_title(
reader: &mut dyn SectorReader,
reader: &mut dyn SectorSource,
title: &DiscTitle,
batch_sectors: u16,
on_progress: Option<&dyn Progress>,
+2 -2
View File
@@ -1,6 +1,6 @@
//! Disc scanning pipeline tests.
use libfreemkv::SectorReader;
use libfreemkv::SectorSource;
use libfreemkv::error::Result;
use libfreemkv::{Disc, DiscTitle, ScanOptions};
use std::collections::HashMap;
@@ -20,7 +20,7 @@ impl MockSectorReader {
}
}
impl SectorReader for MockSectorReader {
impl SectorSource for MockSectorReader {
fn read_sectors(
&mut self,
lba: u32,
+17 -18
View File
@@ -1,17 +1,12 @@
//! Integration tests for progress reporting, halt behavior, drop safety,
//! and the file-backed sector reader round trip.
// 0.18 trait split: this suite still drives the deprecated `pes::Stream`
// trait directly. It will be migrated to `FrameSource`/`FrameSink` in the
// follow-up that ports concrete impls.
#![allow(deprecated)]
use libfreemkv::disc::{CopyOptions, DiscRegion};
use libfreemkv::error::Result;
use libfreemkv::pes::Stream as PesStream;
use libfreemkv::{
ContentFormat, Disc, DiscFormat, DiscStream, DiscTitle, EventKind, Extent, FileSectorReader,
SectorReader,
ContentFormat, Disc, DiscFormat, DiscStream, DiscTitle, EventKind, Extent, FileSectorSource,
SectorSource,
};
use std::io::Write;
use std::sync::Arc;
@@ -37,7 +32,7 @@ impl ZeroSectorReader {
}
}
impl SectorReader for ZeroSectorReader {
impl SectorSource for ZeroSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -51,7 +46,7 @@ impl SectorReader for ZeroSectorReader {
Ok(bytes)
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -72,7 +67,7 @@ impl SlowZeroSectorReader {
}
}
impl SectorReader for SlowZeroSectorReader {
impl SectorSource for SlowZeroSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -86,7 +81,7 @@ impl SectorReader for SlowZeroSectorReader {
Ok(bytes)
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -309,7 +304,7 @@ fn test_drop_impls_do_not_panic_or_block() {
panic!("DiscStream drop did not complete within 100ms");
}
// ── 5. FileSectorReader round trip ────────────────────────────────────────
// ── 5. FileSectorSource round trip ────────────────────────────────────────
#[test]
fn test_file_sector_reader_round_trip() {
@@ -326,9 +321,13 @@ fn test_file_sector_reader_round_trip() {
tmp.flush().expect("flush");
let path = tmp.path().to_path_buf();
let mut fsr = FileSectorReader::open(&path).expect("open FileSectorReader");
let mut fsr = FileSectorSource::open(&path).expect("open FileSectorSource");
assert_eq!(fsr.capacity(), N_SECTORS as u32, "capacity mismatch");
assert_eq!(
fsr.capacity_sectors(),
N_SECTORS as u32,
"capacity mismatch"
);
// Read each sector individually and compare.
let mut buf = vec![0u8; SECTOR_SIZE];
@@ -396,7 +395,7 @@ impl FailingSectorReader {
}
}
impl SectorReader for FailingSectorReader {
impl SectorSource for FailingSectorReader {
fn read_sectors(
&mut self,
_lba: u32,
@@ -423,7 +422,7 @@ impl SectorReader for FailingSectorReader {
})
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
@@ -572,7 +571,7 @@ struct BlockSizeFailingReader {
capacity: u32,
}
impl SectorReader for BlockSizeFailingReader {
impl SectorSource for BlockSizeFailingReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -598,7 +597,7 @@ impl SectorReader for BlockSizeFailingReader {
}
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
+3 -3
View File
@@ -15,7 +15,7 @@ use libfreemkv::disc::CopyOptions;
use libfreemkv::disc::DiscRegion;
use libfreemkv::disc::mapfile::{Mapfile, SectorStatus};
use libfreemkv::error::Result;
use libfreemkv::{ContentFormat, Disc, DiscFormat, SectorReader};
use libfreemkv::{ContentFormat, Disc, DiscFormat, SectorSource};
use std::collections::HashSet;
use std::sync::{Arc, Mutex};
@@ -46,7 +46,7 @@ impl PatternedSectorReader {
}
}
impl SectorReader for PatternedSectorReader {
impl SectorSource for PatternedSectorReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -82,7 +82,7 @@ impl SectorReader for PatternedSectorReader {
Ok(buf.len())
}
fn capacity(&self) -> u32 {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
+807
View File
@@ -0,0 +1,807 @@
//! Pass-N (`Disc::patch`) read-error handler — A/B golden fixture.
//!
//! Background (2026-05-13, v0.20.8 release bundle planning):
//!
//! `libfreemkv::disc::read_error::handle_read_error` is supposed to be
//! the single source of truth for sector-read error → recovery action
//! decisions. Pass 1 sweep routes through it. Pass N patch's
//! `handle_read_failure` (in `disc/patch.rs`) does NOT — historically
//! MEDIUM_ERROR / NOT_READY get inline handling with their own thresholds
//! (`PASSN_DAMAGE_THRESHOLD_PCT=6` vs the sweep's `12`), their own
//! damage_window (state.damage_window, separate from ReadCtx.damage_window),
//! and their own skip logic (`compute_damage_skip`, which runs AFTER
//! the failure handler and has a size-aware `range_remaining/4` cap
//! that `handle_read_error::JumpAhead` does not know about).
//!
//! This file is the A/B fixture for that unification. It pins the
//! CURRENT (pre-unification) end-to-end behavior of `Disc::patch` for
//! eight canonical damage profiles against a synthetic
//! `ScriptedSectorReader`. Each profile asserts the exact observable
//! outcome — final mapfile byte counts and outer-loop counters — so any
//! attempt to refactor the failure path either preserves the goldens or
//! the test fails loudly.
//!
//! The prompt called for "exact sequence of `ReadAction` enums per
//! LBA"; that framing doesn't fit the current architecture because
//! `handle_read_failure` produces `FailureAction`, not `ReadAction`,
//! and interleaves with `compute_damage_skip` + cursor management in
//! the outer loop. The observable contract — what `Disc::patch` does
//! to the mapfile and how many reads it performs — is the equivalent
//! invariant, captured end-to-end.
//!
//! Why we expect divergence under naïve unification (see final report
//! of the 0.20.8 unification attempt): the patch loop's skip semantics
//! live in `compute_damage_skip` POST-failure-handler, with a size-aware
//! cap that `handle_read_error` knows nothing about; routing through
//! `handle_read_error` would invert that cursor flow. The fixture stays
//! checked in regardless — it documents the contract for the next
//! refactor attempt.
use libfreemkv::ContentFormat;
use libfreemkv::Disc;
use libfreemkv::DiscFormat;
use libfreemkv::disc::CopyOptions;
use libfreemkv::disc::DiscRegion;
use libfreemkv::disc::mapfile::{Mapfile, SectorStatus};
use libfreemkv::error::{Error, Result};
use libfreemkv::scsi;
use libfreemkv::{ScsiSense, SectorSource};
use std::sync::{Arc, Mutex};
const SECTOR_SIZE: usize = 2048;
/// Per-attempt result the script can emit. `Ok` returns a deterministic
/// per-sector byte pattern (LBA mod 256 in each sector). `Err` returns
/// the SCSI sense triple supplied — the patch failure path inspects
/// `scsi_sense().sense_key` to classify (MEDIUM, NOT_READY,
/// HARDWARE, ILLEGAL_REQUEST, ABORTED_COMMAND).
#[derive(Debug, Clone, Copy)]
enum ScriptStep {
Ok,
Err { sense_key: u8, asc: u8, ascq: u8 },
}
/// A scripted reader. For each (lba, count) read attempt, picks the
/// step at `attempt_idx[lba]`, advances the index. If no script entry
/// exists for an LBA, defaults to `Ok` so we don't need to script
/// every sector of large ranges.
///
/// "Batch fails if ANY sector in the batch is bad" — matches real
/// drive behavior (`pass_n_size_aware_skip.rs` uses the same model).
/// For batched reads we synthesize an Err with the FIRST scripted
/// failure in the batch.
struct ScriptedSectorReader {
capacity: u32,
/// Per-LBA script of (step, then next step on retry, …). When
/// retries exhaust the script, the LAST step repeats forever.
script: std::collections::HashMap<u32, Vec<ScriptStep>>,
/// Per-LBA index into its script vec. Bumps on each read attempt
/// at that LBA.
attempt_idx: Mutex<std::collections::HashMap<u32, usize>>,
/// Full read trace: every (lba, count, result_was_ok) tuple in
/// call order. Lets the test assert that adaptive-batch dropped
/// to count=1, bisection happened, etc.
trace: Arc<Mutex<Vec<(u32, u16, bool)>>>,
}
impl ScriptedSectorReader {
fn new(capacity: u32) -> (Self, Arc<Mutex<Vec<(u32, u16, bool)>>>) {
let trace = Arc::new(Mutex::new(Vec::new()));
(
Self {
capacity,
script: std::collections::HashMap::new(),
attempt_idx: Mutex::new(std::collections::HashMap::new()),
trace: trace.clone(),
},
trace,
)
}
/// Set a single-step script for `lba`: every attempt yields `step`.
fn always(&mut self, lba: u32, step: ScriptStep) {
self.script.insert(lba, vec![step]);
}
/// Set a multi-step script for `lba`: first attempt yields
/// `steps[0]`, second `steps[1]`, … on retry the last step repeats.
#[allow(dead_code)]
fn sequence(&mut self, lba: u32, steps: Vec<ScriptStep>) {
self.script.insert(lba, steps);
}
fn step_for(&self, lba: u32) -> ScriptStep {
let v = match self.script.get(&lba) {
Some(v) => v,
None => return ScriptStep::Ok,
};
let mut idx = self.attempt_idx.lock().unwrap();
let i = idx.entry(lba).or_insert(0);
let step = v[(*i).min(v.len() - 1)];
*i += 1;
step
}
}
impl SectorSource for ScriptedSectorReader {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
// Look at every sector in the batch — first failure determines
// the outcome.
let mut failure: Option<(u8, u8, u8)> = None;
for offset in 0..count as u32 {
match self.step_for(lba + offset) {
ScriptStep::Ok => {}
ScriptStep::Err {
sense_key,
asc,
ascq,
} => {
failure = Some((sense_key, asc, ascq));
break;
}
}
}
let ok = failure.is_none();
self.trace.lock().unwrap().push((lba, count, ok));
if let Some((sense_key, asc, ascq)) = failure {
return Err(Error::ScsiError {
opcode: scsi::SCSI_READ_10,
status: scsi::SCSI_STATUS_CHECK_CONDITION,
sense: Some(ScsiSense {
sense_key,
asc,
ascq,
}),
});
}
// Per-sector LBA byte pattern.
for (i, chunk) in buf.chunks_mut(SECTOR_SIZE).enumerate() {
chunk.fill(((lba + i as u32) & 0xff) as u8);
}
Ok(buf.len())
}
fn capacity_sectors(&self) -> u32 {
self.capacity
}
}
fn synthetic_disc(capacity_sectors: u32) -> Disc {
Disc {
volume_id: String::new(),
meta_title: None,
format: DiscFormat::BluRay,
capacity_sectors,
capacity_bytes: capacity_sectors as u64 * SECTOR_SIZE as u64,
layers: 1,
titles: Vec::new(),
region: DiscRegion::Free,
aacs: None,
css: None,
encrypted: false,
aacs_error: None,
content_format: ContentFormat::BdTs,
}
}
fn prep_iso_and_mapfile(
iso_path: &std::path::Path,
total_bytes: u64,
finished_ranges: &[(u64, u64)],
nontrimmed_ranges: &[(u64, u64)],
) {
use std::fs::OpenOptions;
use std::io::{Seek, SeekFrom, Write};
let mut f = OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(iso_path)
.unwrap();
f.set_len(total_bytes).unwrap();
f.seek(SeekFrom::Start(0)).unwrap();
f.write_all(&[]).unwrap();
let map_path = libfreemkv::disc::mapfile_path_for(iso_path);
let mut mf = Mapfile::create(&map_path, total_bytes, "test").unwrap();
for &(pos, size) in finished_ranges {
mf.record(pos, size, SectorStatus::Finished).unwrap();
}
for &(pos, size) in nontrimmed_ranges {
mf.record(pos, size, SectorStatus::NonTrimmed).unwrap();
}
}
/// Observable outcome of a patch run. Goldens for each profile pin
/// these exact values.
#[derive(Debug, PartialEq, Eq)]
struct Golden {
/// `bytes_good` at end of patch.
bytes_good: u64,
/// `bytes_unreadable` at end.
bytes_unreadable: u64,
/// `bytes_pending` (NonTrimmed) at end.
bytes_pending: u64,
/// Did the pass exit via wedge-detection?
wedged_exit: bool,
/// Sanity bound on trace length — patch makes a finite number of
/// reads bounded by `MAX_SKIPS_PER_RANGE * range_sectors` plus
/// retries. Asserted as an UPPER bound only (so any reduction in
/// retries via future tuning doesn't fail the test spuriously).
max_reads: usize,
}
/// Common helper: prep ISO + mapfile, run `disc.copy(multipass)`,
/// return (PatchOutcome ↔ CopyResult, final-map stats, trace length).
fn run_profile(
profile_name: &str,
capacity_sectors: u32,
nontrimmed: &[(u64, u64)],
finished: &[(u64, u64)],
scripted: ScriptedSectorReader,
trace: Arc<Mutex<Vec<(u32, u16, bool)>>>,
) -> (
libfreemkv::disc::CopyResult,
libfreemkv::disc::mapfile::MapStats,
usize,
) {
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().unwrap();
let iso_path = tmp.path().to_path_buf();
drop(tmp);
prep_iso_and_mapfile(&iso_path, total_bytes, finished, nontrimmed);
let opts = CopyOptions {
decrypt: false,
multipass: true,
..Default::default()
};
let mut reader = scripted;
let pr = disc
.copy(&mut reader, &iso_path, &opts)
.unwrap_or_else(|e| panic!("[{profile_name}] disc.copy returned Err: {e:?}"));
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
let map = Mapfile::load(&map_path).unwrap();
let stats = map.stats();
let trace_len = trace.lock().unwrap().len();
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(&map_path);
(pr, stats, trace_len)
}
// ─────────────────────────── Profile 1: CLEAN ────────────────────────────
//
// The NonTrimmed range has zero scripted failures — every read succeeds.
// Patch should march through the range and mark it Finished. Validates
// the happy-path side of the failure-handler dispatch (it shouldn't
// fire at all).
#[test]
fn profile_01_clean_all_recoverable() {
let capacity_sectors: u32 = 256;
let (reader, trace) = ScriptedSectorReader::new(capacity_sectors);
// No scripted errors → all reads succeed.
let nontrimmed = [(100 * 2048, 16 * 2048)]; // 16-sector NonTrimmed range
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"01_clean",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
let expected = Golden {
bytes_good: capacity_sectors as u64 * 2048,
bytes_unreadable: 0,
bytes_pending: 0,
wedged_exit: false,
max_reads: 8, // adaptive batch=32 reads finishes 16 sectors in 1 read; allow up to 8.
};
assert_eq!(stats.bytes_good, expected.bytes_good, "01_clean bytes_good");
assert_eq!(
stats.bytes_unreadable, expected.bytes_unreadable,
"01_clean bytes_unreadable"
);
assert_eq!(
stats.bytes_pending, expected.bytes_pending,
"01_clean bytes_pending"
);
assert!(!pr.halted, "01_clean halted");
assert!(
trace_len <= expected.max_reads,
"01_clean trace_len={trace_len} exceeds bound {}",
expected.max_reads
);
}
// ─────────────────────────── Profile 2: ALL MEDIUM ───────────────────────
//
// Every LBA in the NonTrimmed range returns MEDIUM_ERROR every attempt.
// Adaptive-batch drops to count=1 on first batch failure, then each
// single-sector read fails → consecutive_failures climbs, damage_window
// fills, compute_damage_skip fires, MAX_SKIPS_PER_RANGE caps the work,
// remaining bytes stay NonTrimmed (NEVER marked Unreadable inside a
// single pass — 2026-05-11 design call).
#[test]
fn profile_02_all_medium_error() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
for lba in 100..116 {
reader.always(
lba,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
}
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"02_all_medium",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: the 16-sector bad range stays NonTrimmed (bytes_pending).
// Pre-2026-05-11 patch would mark Unreadable here; current code
// preserves NonTrimmed so subsequent passes get another shot.
assert_eq!(
stats.bytes_good,
(capacity_sectors as u64 - 16) * 2048,
"02_all_medium bytes_good"
);
assert_eq!(
stats.bytes_unreadable, 0,
"02_all_medium bytes_unreadable (must NOT be marked terminal in one pass)"
);
assert_eq!(
stats.bytes_pending,
16 * 2048,
"02_all_medium bytes_pending (NonTrimmed retained across passes)"
);
assert!(!pr.halted, "02_all_medium halted");
// Upper bound: every sector probed individually + a few batch-drop
// and skip-escalation attempts. 16 sectors × ~3 visits ≈ 50.
assert!(
trace_len <= 80,
"02_all_medium trace_len={trace_len} exceeds 80"
);
}
// ───────────────────── Profile 3: ALTERNATING GOOD/BAD ───────────────────
//
// LBAs 100, 102, 104, ... bad; odd LBAs good. Validates that good
// sectors interleaved with bad get recovered individually after the
// adaptive split (batch-fail → count=1 → per-sector probe).
#[test]
fn profile_03_alternating_good_bad() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
for lba in (100..116).step_by(2) {
reader.always(
lba,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
}
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"03_alternating",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: 8 good sectors interleaved should mostly be Finished;
// 8 bad stay NonTrimmed. Allow 2 sectors of slop for the actual
// bisect cursor advance — converging on alternating bad/good in
// a single pass isn't always exact at boundaries with the
// size-aware skip cap.
let good_total = stats.bytes_good;
let baseline_good = (capacity_sectors as u64 - 16) * 2048;
let middle_recovered = good_total - baseline_good;
assert!(
middle_recovered >= 6 * 2048,
"03_alternating recovered only {middle_recovered} bytes of 8 good sectors"
);
assert!(
middle_recovered <= 9 * 2048,
"03_alternating recovered MORE than scripted good sectors: {middle_recovered}"
);
assert_eq!(stats.bytes_unreadable, 0, "03_alternating bytes_unreadable");
// Remaining must be NonTrimmed (pending), not lost.
assert!(
stats.bytes_pending > 0,
"03_alternating expected NonTrimmed remainder, got bytes_pending=0"
);
assert!(!pr.halted, "03_alternating halted");
assert!(
trace_len <= 120,
"03_alternating trace_len={trace_len} exceeds 120"
);
}
// ───────────────────── Profile 4: EDGE-BAD (size-aware-skip canon) ───────
//
// Bad at start (100..104), good middle (104..112), bad at end (112..116).
// This is the size-aware-skip canonical case. The middle good sectors
// MUST be recovered — pre-fix patch would skip-escalate across the
// whole range and miss them.
#[test]
fn profile_04_edge_bad_good_middle() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
for lba in 100..104 {
reader.always(
lba,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
}
for lba in 112..116 {
reader.always(
lba,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
}
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"04_edge_bad",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: the 8 good middle sectors should land Finished (allowing
// 2 sectors of bisection slop at boundaries).
let middle_recovered = stats.bytes_good - (capacity_sectors as u64 - 16) * 2048;
assert!(
middle_recovered >= 6 * 2048,
"04_edge_bad recovered only {middle_recovered} bytes of 8 good middle sectors"
);
assert_eq!(stats.bytes_unreadable, 0, "04_edge_bad bytes_unreadable");
assert!(
stats.bytes_pending > 0,
"04_edge_bad bytes_pending expected > 0"
);
assert!(!pr.halted, "04_edge_bad halted");
assert!(
trace_len <= 120,
"04_edge_bad trace_len={trace_len} exceeds 120"
);
}
// ───────────────────── Profile 5: SINGLE BAD SECTOR ──────────────────────
//
// 1 bad sector in the middle of an otherwise good 16-sector NonTrimmed
// range. Validates the common "stochastic miss in Pass 1, easily picked
// up in Pass N" scenario.
#[test]
fn profile_05_single_bad_sector() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
reader.always(
108,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"05_single_bad",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: 15 of 16 sectors recovered. 1 sector stays NonTrimmed
// (NOT Unreadable — same multi-pass tolerance principle).
assert_eq!(
stats.bytes_good,
(capacity_sectors as u64 - 1) * 2048,
"05_single_bad bytes_good"
);
assert_eq!(stats.bytes_unreadable, 0, "05_single_bad bytes_unreadable");
assert_eq!(stats.bytes_pending, 2048, "05_single_bad bytes_pending");
assert!(!pr.halted, "05_single_bad halted");
assert!(
trace_len <= 80,
"05_single_bad trace_len={trace_len} exceeds 80"
);
}
// ───────────────────── Profile 6: DEEP PIT ───────────────────────────────
//
// A contiguous 8-sector bad pit in the middle of a wider 24-sector
// NonTrimmed range. Tests the damage-window threshold + size-aware-skip
// converging on the actual pit boundaries instead of bailing on
// MAX_SKIPS_PER_RANGE.
#[test]
fn profile_06_deep_pit() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
for lba in 108..116 {
reader.always(
lba,
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
);
}
// 24 sectors NonTrimmed: 100..108 good, 108..116 BAD, 116..124 good.
let nontrimmed = [(100 * 2048, 24 * 2048)];
let finished = [
(0, 100 * 2048),
(124 * 2048, (capacity_sectors as u64 - 124) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"06_deep_pit",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: 16 good (8 on each side of the pit) recovered, 8 bad
// stay NonTrimmed.
let recovered_in_range = stats.bytes_good - (capacity_sectors as u64 - 24) * 2048;
assert!(
recovered_in_range >= 14 * 2048,
"06_deep_pit recovered only {recovered_in_range} bytes of 16 good sectors"
);
assert_eq!(stats.bytes_unreadable, 0, "06_deep_pit bytes_unreadable");
assert!(
stats.bytes_pending > 0,
"06_deep_pit bytes_pending expected > 0"
);
assert!(!pr.halted, "06_deep_pit halted");
assert!(
trace_len <= 120,
"06_deep_pit trace_len={trace_len} exceeds 120"
);
}
// ───────────────────── Profile 7: MEDIUM-THEN-GOOD ───────────────────────
//
// First N attempts at each bad LBA fail with MEDIUM_ERROR, then succeed.
// Tests whether patch's retry semantics revisit failed sectors. Current
// patch dispatches NonTrimmed on first failure and ADVANCES the cursor
// — it does NOT retry the same LBA inside one pass for MEDIUM_ERROR
// (only NOT_READY retries in-place). So the goldens here are: bad
// sectors stay NonTrimmed in this pass (the recovery would happen in a
// subsequent pass, which this single-pass fixture does not run).
#[test]
fn profile_07_medium_then_good() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
// Sectors 105..110: fail twice, then succeed.
for lba in 105..110 {
reader.sequence(
lba,
vec![
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
ScriptStep::Ok,
],
);
}
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"07_medium_then_good",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: patch's cache-priming (`prime_cache`) issues 3
// throwaway single-sector reads at lba-3..lba before each count==1
// recovery read. Those throwaway reads ADVANCE the per-LBA script
// step counter even though their results are discarded. So a
// 3-step script (fail, fail, ok) gets consumed by 2 prime calls
// plus 1 real read → the real read sees `Ok` and the sector is
// recovered. Net effect: patch fully recovers the range in one
// pass thanks to priming, even though the script said "fails on
// first two attempts."
//
// This is the documented cache-prime behavior (`disc/patch.rs`
// ~line 398, "Proven 2026-05-07 with dd-as-oracle: 8/8 sectors
// recoverable when primed vs 6/8 cold"). The golden pins it.
assert_eq!(
stats.bytes_good,
capacity_sectors as u64 * 2048,
"07_medium_then_good bytes_good — cache-prime should consume \
the failing script steps so the real read sees Ok"
);
assert_eq!(
stats.bytes_unreadable, 0,
"07_medium_then_good bytes_unreadable"
);
assert_eq!(stats.bytes_pending, 0, "07_medium_then_good bytes_pending");
assert!(!pr.halted, "07_medium_then_good halted");
assert!(
trace_len <= 100,
"07_medium_then_good trace_len={trace_len} exceeds 100"
);
}
// ───────────────────── Profile 8: BATCHED-FAIL ONLY ──────────────────────
//
// LBA 108 fails on BATCH reads (any batch including it) but succeeds
// individually. Models a marginal sector that the drive can ECC-recover
// when read alone but not at multi-sector throughput. Validates that
// adaptive batch's drop-to-count=1 retries the same starting position
// and rescues the data.
//
// Implementation note: the scripted reader marks the entire batch failed
// on any failed sector. We can't easily differentiate "single vs batch"
// without bigger plumbing — so this profile uses a script that fails
// once then succeeds on retry at the same LBA, simulating "drive
// recovered after retry."
#[test]
fn profile_08_batch_fail_singles_ok() {
let capacity_sectors: u32 = 256;
let (mut reader, trace) = ScriptedSectorReader::new(capacity_sectors);
// Sector 108: fail on first call (which is the batch read), succeed
// on second call (the drop-to-count=1 retry at the same position).
reader.sequence(
108,
vec![
ScriptStep::Err {
sense_key: scsi::SENSE_KEY_MEDIUM_ERROR,
asc: 0x11,
ascq: 0x00,
},
ScriptStep::Ok,
],
);
let nontrimmed = [(100 * 2048, 16 * 2048)];
let finished = [
(0, 100 * 2048),
(116 * 2048, (capacity_sectors as u64 - 116) * 2048),
];
let (pr, stats, trace_len) = run_profile(
"08_batch_fail",
capacity_sectors,
&nontrimmed,
&finished,
reader,
trace,
);
// GOLDEN: the second attempt succeeds → all 16 sectors recovered.
assert_eq!(
stats.bytes_good,
capacity_sectors as u64 * 2048,
"08_batch_fail bytes_good — second attempt should recover"
);
assert_eq!(stats.bytes_unreadable, 0, "08_batch_fail bytes_unreadable");
assert_eq!(stats.bytes_pending, 0, "08_batch_fail bytes_pending");
assert!(!pr.halted, "08_batch_fail halted");
assert!(
trace_len <= 80,
"08_batch_fail trace_len={trace_len} exceeds 80"
);
}
// ─────────────────────────────────────────────────────────────────────────
//
// Suppressed for now: NOT_READY-then-recover, HARDWARE_ERROR (wedge),
// ILLEGAL_REQUEST (wedge), and ABORTED_COMMAND profiles. Each would
// trigger long real-time sleeps inside `handle_read_failure`:
//
// - NOT_READY (sense_key=0x02, asc=0x02/0x03/0x04): 15 s pause per
// occurrence (`patch_not_ready_pause`), and retries the same LBA
// in-place. Even one NOT_READY costs the test 15 s wall-time.
//
// - HARDWARE_ERROR / ILLEGAL_REQUEST: 30 s per occurrence
// (`WEDGE_FAMILY_COOLDOWN_SECS`), bounded by
// `WEDGE_ABORT_THRESHOLD=16` before wedged-exit. Worst case ~8
// minutes per profile.
//
// The sleeps are not injectable. Adding them would require either a
// `now()` / `sleep()` trait injection (out of scope for the unification
// task) or a "test mode" compile-time flag (architectural smell). The
// behavioural contracts for those paths are captured in
// `read_error.rs`'s in-module tests instead — they exercise the
// classifier without invoking the patch loop's sleep side-effects.
//
// If the unification ever proceeds, the next step is to add a clock
// injection point in `handle_read_failure` and extend this fixture
// with the wedge/NOT_READY profiles too.
-5
View File
@@ -1,10 +1,5 @@
//! Integration tests for the PES stream pipeline.
// 0.18 trait split: this suite still drives the deprecated `pes::Stream`
// trait directly. It will be migrated to `FrameSource`/`FrameSink` in the
// follow-up that ports concrete impls.
#![allow(deprecated)]
use libfreemkv::mux::meta::M2tsMeta;
use libfreemkv::pes::Stream as PesStream;
use libfreemkv::*;
+4 -4
View File
@@ -1,7 +1,7 @@
//! UDF parser tests using a MockSectorReader.
use libfreemkv::error::Result;
use libfreemkv::{SectorReader, read_filesystem};
use libfreemkv::{SectorSource, read_filesystem};
use std::collections::HashMap;
const SECTOR_SIZE: usize = 2048;
@@ -38,7 +38,7 @@ impl MockSectorReader {
}
}
impl SectorReader for MockSectorReader {
impl SectorSource for MockSectorReader {
fn read_sectors(
&mut self,
lba: u32,
@@ -463,11 +463,11 @@ fn find_dir_case_insensitive() {
#[test]
fn sector_reader_is_object_safe() {
// Verify SectorReader can be used as a trait object
// Verify SectorSource can be used as a trait object
let mut reader = MockSectorReader::new();
reader.set_sector(0, vec![42u8; SECTOR_SIZE]);
let dyn_reader: &mut dyn SectorReader = &mut reader;
let dyn_reader: &mut dyn SectorSource = &mut reader;
let mut buf = vec![0u8; SECTOR_SIZE];
let n = dyn_reader.read_sectors(0, 1, &mut buf, true).unwrap();
assert_eq!(n, SECTOR_SIZE);