Compare commits

...
33 Commits
Author SHA1 Message Date
MattJackson d9ce69bc9d v0.18.2: fix AACS nav-file scramble + sweep progress non-regression
decrypt::decrypt_sectors now restores chunks when decrypt_unit_full's
TS-sync verification fails, preventing 0.18.1's silent corruption of
MPLS/CLPI navigation files when DecryptingSectorSource decorates the
sweep reader. Fixes E6009 NoStreams on info iso:// for AACS-encrypted
UHDs ripped without --raw.

Disc::sweep progress takes max(snapshot.bytes_good, bytes_done) so
the user-visible counter never regresses below what the producer has
already sent.
2026-05-09 17:19:47 -07:00
MattJackson 7cd2c937ed 0.18.1 docs: refresh README, CHANGELOG, and docs/ for the trait split
The library's public-facing docs were sitting on the 0.17 trait
surface — Disc::copy, pes::Stream, SectorReader, etc. — even though
all in-tree callers migrated in 0.18 rounds 1-3. With 0.18.1 about
to ship, a user copy-pasting the README sample from crates.io would
have hit a compile error.

This commit is purely doc-side:

- README.md: Quick Start rewritten onto Disc::sweep + Disc::patch
  with caller-orchestrated multipass; Streams table footnote and
  Architecture row reference FrameSource / FrameSink.
- CHANGELOG.md: 0.18.1 entry describing the redesign — primitives,
  trait splits, deprecations (kept alive through 0.18.x, deletion
  target 0.18.2), throughput numbers.
- docs/{rip-recovery,api-design,architecture,disc-to-rip,
  drive-access,udf}.md: every Disc::copy / pes::Stream /
  SectorReader reference updated to the 0.18 trait surface.
- FEATURES.md: deleted (8+ versions stale; capabilities live in
  README.md and CHANGELOG.md now, matching the workspace-top
  FEATURES.md removal in 84acd65).
- examples/iso_dump.rs: verified compiles against 0.18.1.

No code changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 12:13:51 -07:00
MattJackson 107524da5e 0.18.1: bump version 2026-05-09 11:38:24 -07:00
MattJackson 055a3c5276 0.18 round 3: make Disc::sweep + Disc::patch pub (was pub(crate))
Round 3 step 1: lift the visibility on the two flat rip-phase verbs
so consumers (autorip + freemkv CLI) can call them directly instead
of going through Disc::copy's multipass dispatcher. Also lift their
option/outcome types and re-export at crate root.

- fn sweep -> pub fn sweep (with rustdoc explaining its role)
- fn patch -> pub fn patch (ditto)
- pub(crate) struct SweepOptions -> pub struct SweepOptions
- pub(crate) struct PatchOpts -> pub struct PatchOptions (renamed
  for consistency — both are 'Options')
- pub(crate) struct PatchOutcome -> pub struct PatchOutcome
- libfreemkv::{SweepOptions, PatchOptions, PatchOutcome} re-exports
  at crate root.

Disc::copy still exists and still calls Disc::sweep / Disc::patch
through the now-private sweep_internal / patch_internal wrappers.
Migration of the two autorip callers + the freemkv CLI's
disc_to_iso to direct sweep/patch is a follow-up; once those land
Disc::copy + CopyOptions + CopyResult delete in the same commit.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 11:19:46 -07:00
MattJackson 50b04790ce 0.18 round 2: thread Halt through DiscStream construction
# Conflicts:
#	src/mux/disc.rs
2026-05-09 11:04:07 -07:00
MattJackson a00b2c884b 0.18 round 2: adopt DecryptingSectorSource decorator at sweep + patch + DiscStream 2026-05-09 11:03:43 -07:00
MattJackson 44e2c64d7e 0.18 round 2 (Halt threading): DiscStream accepts Halt at construction
Adds a Halt field to DiscStream, populated via the new
`with_halt(halt)` builder. The internal recovery / fill_extents
loops check `halt.is_cancelled()` directly. The existing
`set_halt(Arc<AtomicBool>)` method stays through the deprecation
window for callers (autorip mux) that haven't migrated; marked
#[deprecated] with a pointer to the constructor-time path.

Both signals are unified inside DiscStream: either Halt or the
legacy Arc<AtomicBool> triggers cancellation, so callers can mix
during the deprecation window without breaking stop behaviour.

See (internal)/memory/0_18_redesign.md and
0_18_round3_migration_audit.md.

Single contributor: MattJackson.
2026-05-09 10:58:54 -07:00
MattJackson d6535b8f57 0.18 round 2 (decrypt dedup): adopt DecryptingSectorSource at the two
existing call sites — sweep producer and DiscStream demux

Round 1 shipped the DecryptingSectorSource decorator
(libfreemkv/src/sector/decrypting.rs) but the existing decrypt
sites kept calling crate::decrypt::decrypt_sectors inline. This
commit migrates both:

- Disc::sweep (disc/mod.rs): producer wraps the input reader
  in DecryptingSectorSource::new(reader, keys) before the read loop.
  The inline decrypt_sectors call goes away — read_sectors yields
  plaintext directly.

- DiscStream (mux/disc.rs): constructor wraps the underlying
  Box<dyn SectorReader> in DecryptingSectorSource so the internal
  fill_extents / read path sees plaintext bytes. The DecryptKeys
  field stays on DiscStream for metadata-side use; it just no
  longer drives decryption.

Disc::patch carried the same inline decrypt step at three call
sites (main read, backtrack read, non-NOT_READY retry read). All
three migrated onto the same wrapping for a single audit surface.

Two small support changes carry the migration without touching
the round-1 decorator shape:
- sector/mod.rs gains specific SectorSource impls for
  &mut dyn SectorReader and Box<dyn SectorReader>, mirroring
  std's Read forwarding pattern. Generic blankets would conflict
  with the existing SectorReader → SectorSource blanket under the
  orphan rule (downstream could impl SectorReader for &mut U), so
  the impls are scoped to the dyn-trait shape we actually consume.
- sector/decrypting.rs gains DecryptingSectorSource::set_keys so
  DiscStream::set_raw() can flip the wrapped reader to a
  DecryptKeys::None pass-through without rebuilding the decorator
  (which would require moving the inner Box out from behind &mut self).

After this commit, grep `decrypt_sectors` in src/ shows the
function definition, its single use inside DecryptingSectorSource,
plus comments only. One audit surface for AACS / CSS / passthrough
correctness.

Behaviour-preserving: same plaintext bytes flow through; the only
difference is which type owns the decrypt step.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:49:53 -07:00
MattJackson 90ab00ed45 0.18 round 2: cargo fmt after sweep+patch+DiscStream merge
Auto-fmt nit on the multi-line map.flush() expression that landed
when the sweep + patch + DiscStream-FrameSource branches were merged
together. No semantic change.
2026-05-09 10:36:40 -07:00
MattJackson 6bfd7dfd13 0.18 round 2: confirm DiscStream as FrameSource via blanket impl 2026-05-09 10:32:56 -07:00
MattJackson b9a7f601d4 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
# Conflicts:
#	src/io/mod.rs
#	src/io/pipeline.rs
2026-05-09 10:32:51 -07:00
MattJackson cf0a61f8b4 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink (delete sweep_pipeline.rs) 2026-05-09 10:32:05 -07:00
MattJackson f98f07b2d3 0.18 round 2: refactor Disc::sweep onto Pipeline + SweepSink
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.

- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
  WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
  carries the file-write + mapfile.record per WorkItem; close()
  drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
  (so the consumer thread keeps showing up as
  freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
  The producer-side ReadCtx state machine, decrypt, set_speed,
  halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
  preserve identifiable thread names without the primitive baking
  one in. Also adds Pipeline::try_send for the throttled
  StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
  ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
  module-private types. WorkItem::Finish dropped — dropping the
  channel is the end-of-stream signal Pipeline already uses.

Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.

See (internal)/memory/0_18_redesign.md.
2026-05-09 10:31:05 -07:00
MattJackson b53454fa09 0.18 round 2: refactor Disc::patch onto Pipeline + PatchSink
Patch was strictly serial (per-sector recovery: read → seek+write
→ mapfile.record → next). Lifting the write+record onto a consumer
thread lets the drive issue the next per-sector retry while the
previous block's recovered bytes are being committed — small but
real win on damaged discs with many bad sectors, and uniform with
sweep's threading model.

- New PatchSink: Sink<PatchItem> impl in src/disc/patch.rs. Owns
  WritebackFile + Mapfile. apply() seeks+writes recovered bytes
  and records mapfile state per item; close() runs sync_all and
  mapfile.flush.
- Channel depth: WRITE_THROUGH_DEPTH (1). Patch wants minimum
  buffering — back-pressure should kick in immediately so the
  drive's per-sector retry budget isn't ahead of the writer.
- Disc::patch: keeps every existing recovery decision on the
  producer (reverse walk, damage-window skip, NOT_READY pauses,
  bridge-degradation handling, wedge exit, range watchdog).
  WritebackFile ownership moves to the sink.

Behaviour-preserving: per-sector single-shot read budget unchanged
(BU40N+Initio bridge wedge concern still respected); recovery
algorithm bit-identical.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:28:14 -07:00
MattJackson f28b6ee6d9 0.18 round 2: re-export Pipeline + Sink + Flow at the crate root
Round 2 #1 landed `Pipeline` / `Sink` / `Flow` / `DEFAULT_PIPELINE_DEPTH`
in `crate::io::pipeline` but only re-exported them through
`crate::io` (which is `pub(crate)`), so no out-of-tree consumer could
reach them. autorip's round 2 #2 (lifting the mux loop onto Pipeline +
MuxSink) is the first such consumer; surface the primitives at the
crate root for ergonomic access.

No behaviour change — the items themselves are unchanged from round
2 #1; this is just `pub use` plumbing.

Single contributor: MattJackson.
2026-05-09 10:22:44 -07:00
MattJackson d9d778fa64 0.18 round 2: confirm DiscStream as FrameSource
Mirror of the FrameSink concrete migrations slice (f52e4c5) on the
read side. DiscStream is the only meaningful source impl in tree;
all the mux/* impls are sinks.

The round-1 blanket impl<T: Stream + Send> FrameSource for T
already covers DiscStream if it's Send. This slice:

- Audits DiscStream's interior types for Send (its Box<dyn
  SectorReader> already requires Send via the trait's super-bound;
  verify nothing else interior breaks Send).
- Adds a synthetic-input test that constructs Box<dyn FrameSource>
  over a DiscStream, reads frames through the trait object, and
  exercises info() / headers_ready() / codec_private().
- (Conditional) Adds a direct FrameSource impl on DiscStream only
  if call-site ergonomics demand it; otherwise relies on the
  blanket.

No caller migrated. mux::resolve::input still returns
Box<dyn Stream>; autorip / CLI consumers still call Stream::read.
Per-caller migration is a later slice.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 10:13:14 -07:00
MattJackson bbfb887a35 0.18 round 1+2 integration fixes
Two clippy issues surfaced when round 1 polish + round 2 FrameSink
migrations both landed on libfreemkv main:

- src/halt.rs: clippy::new_without_default fires when a public new()
  exists without Default. The polish pass dropped the derive thinking
  it was redundant — clippy disagrees, so add a manual impl that
  forwards to new(). Doc-comment notes why both exist.

- src/disc/read_error.rs:372: pre-existing
  assert_eq!(.., true) trips clippy::bool_assert_comparison. Pre-0.18
  precommits passed because that lint sat outside the gate; the
  round-2 commits brought enough new clippy surface that it now
  shows up. Trivial cleanup: assert!(...) instead of assert_eq!.

Single contributor: MattJackson.
2026-05-09 10:00:06 -07:00
MattJackson 9884346c24 0.18 round 2: FrameSink concrete migrations for mux/* sinks 2026-05-09 09:53:12 -07:00
MattJackson 766b7c6636 0.18 round 1 polish: address libfreemkv code-review findings 2026-05-09 09:53:06 -07:00
MattJackson 925c30686b 0.18 round 1 polish: address libfreemkv code-review findings
Applies must-fix + in-scope should-fix items from the round-1 code
review:

- M1: FileSectorSource::open takes &Path (was &str — non-UTF-8 panic)
- M2: drop FileSectorSource's BufReader (defeated by absolute seeks)
- M3: WritebackFile Drop impl finalises the writeback pipeline
- M4: Pipeline::finish preserves panic payload in error message
- M5: pes::Stream is left without a : Send supertrait — concrete
  in-tree impls (MkvStream, M2tsStream) hold Box<dyn Read> /
  Box<dyn Write> trait objects that aren't Send, so the simple
  trait tightening would cascade into a wider Send audit. Per the
  review's escape clause the FrameSource blanket impl keeps its
  T: Send bound and the constraint is documented loudly there.
- S6: document Pipeline::send post-Flow::Stop semantics
- S9: truncate stale Stream docs (E9001/E9000 was runtime-only)
- S10: document WritebackPipeline.fd lifetime invariant
- S11: pub use pes::Stream as PesStream to disambiguate from
  disc::Stream codec enum at crate root
- S12: rename DEFAULT_DEPTH → DEFAULT_PIPELINE_DEPTH; add
  WRITE_THROUGH_DEPTH constant
- N14: drop Halt's Default derive (redundant with Halt::new)
- N17: Pipeline::spawn propagates thread-spawn error instead of expect
- N19: deprecation since = "0.18.0" (was "0.18.0-dev", non-conventional)
- N21: rename Apply enum to Flow

Deferred to follow-up commits: SectorReader/SectorSource competition
(migration commit), WritebackFile::create/open orphans (migration
commit), AACS round-trip test (design doc defers), various nits.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:52:25 -07:00
MattJackson f52e4c5d22 0.18 round 2: add FrameSink impls to concrete mux sinks
Per-impl migration of MkvStream / M2tsStream / NetworkStream /
NullStream / StdioStream from the deprecated pes::Stream trait
to the typed pes::FrameSink trait. Both impls coexist during
the 0.18 deprecation window — the existing Stream impls are
unchanged.

The FrameSink::finish signature differs (Box<Self> vs &mut self),
which is why this couldn't be a blanket impl. Each migration
re-borrows the box and delegates to the underlying Stream::finish
body.

FrameSink: Send forced two struct fields (M2tsStream's boxed
Write/Read, MkvStream's boxed WriteSeek/Read) to gain `+ Send`
bounds — minimum surface needed to make the Send-bounded trait
impl-able. mux::resolve::output's local Box<dyn WriteSeek>
construction picks up the same `+ Send`. tests/streams.rs's
shared `stream.write/.finish/.info/.read` calls were
disambiguated to `PesStream::*` to resolve the now-multiple
candidates from coexisting trait impls.

Caller migration (mux::resolve::output return type, autorip,
CLI) is a later slice. This commit only adds new impls; nothing
removed.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:49:18 -07:00
MattJackson 8d67790d2e 0.18: thread WritebackFile rename through FileSectorSink
The SectorSource/Sink agent and the WritebackFile-rename agent both
branched from main concurrently; the sector branch wrote against the
0.17 Writer name and only the rename branch knew about WritebackFile.
This integration commit reconciles the two: FileSectorSink::create /
::open / the inner-field type all use WritebackFile directly, and the
module-level + struct-level docs are corrected.
2026-05-09 09:17:39 -07:00
MattJackson 246a439990 0.18: FrameSource/FrameSink trait split (deprecate Stream) 2026-05-09 09:13:39 -07:00
MattJackson bd19041648 0.18: SectorSource/SectorSink trait split + DecryptingSectorSource decorator 2026-05-09 09:13:34 -07:00
MattJackson 198268b725 0.18: add crate::io::Pipeline + Sink trait
# Conflicts:
#	src/io/mod.rs
2026-05-09 09:13:30 -07:00
MattJackson 31424c8203 0.18: add crate::halt::Halt cancellation token 2026-05-09 09:13:07 -07:00
MattJackson c747ecc589 0.18: rename crate::io::Writer → WritebackFile 2026-05-09 09:12:59 -07:00
MattJackson 283a561c12 0.18 primitive: SectorSource/SectorSink trait split + DecryptingSectorSource
Splits the unidirectional read trait from a (planned) write trait at
the sector level, eliminating runtime "wrong direction" potential.
Keeps SectorReader alive as a pre-deprecation alias via blanket impl
so existing callers compile unchanged through the migration window.

Adds DecryptingSectorSource decorator: wrap any SectorSource in this
to get plaintext sectors out. Replaces the duplicate decrypt code
paths in sweep_pipeline and DiscStream (those migrations are
follow-up commits).

The formal #[deprecated] attribute on SectorReader is held back to a
follow-up commit because internal call sites in disc/, udf/, mux/,
and verify/ still go through the legacy trait, and the CI gauntlet
treats deprecation lints as errors. Behavioural intent — "this trait
is going away" — is documented on the trait itself.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 09:04:16 -07:00
MattJackson 2667d68675 0.18 primitive: FrameSource/FrameSink trait split (deprecate Stream)
Splits the bidirectional pes::Stream into one-direction traits so
calling read() on a write-only sink is a compile error instead of
runtime E9001. Keeps Stream alive as deprecated through 0.18 with a
blanket FrameSource impl so existing concrete types compile unchanged.

FrameSink can't be blanket-impl'd from Stream (different finish
signature), so concrete impls migrate per-type in a follow-up.

Concrete `impl pes::Stream for X` blocks in mux/* and the existing
tests gain a one-line `#[allow(deprecated)]` to keep `-D warnings`
clean during the deprecation window — no behavior changes.

See (internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:57:48 -07:00
MattJackson 57f2c22e30 0.18 primitive: crate::io::Pipeline + Sink trait
Generic bounded producer/consumer pipeline. The same shape applies to
sweep, patch, and mux today via three near-duplicate implementations
(or, in patch's and mux's case, no implementation at all). 0.18
collapses them onto one primitive. See
(internal)/memory/0_18_redesign.md for full context.

Single contributor: MattJackson.
2026-05-09 08:56:31 -07:00
MattJackson 5f3545d244 0.18 primitive: rename crate::io::Writer → WritebackFile
The type's job is the bounded-cache writeback pipeline (sync_file_range
+ posix_fadvise(DONTNEED)) — not generic writing. The 0.17 name was
ambiguous; reading `Writer::new(file)` gave no hint about what was
special. New name makes the role obvious at every call site.

Adds `WritebackFile::create(path)` and `WritebackFile::open(path)`
constructors so callers don't have to assemble a `File` first.

No alias kept; this is a clean 0.18 rename. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson.
2026-05-09 08:53:17 -07:00
MattJackson 6e17ef0859 0.18 primitive: crate::halt::Halt cancellation token
One-bit cooperative cancellation flag. Replaces ad-hoc Arc<AtomicBool>
patterns scattered across libfreemkv (DiscStream::set_halt) and the
HALT_FLAGS global registry in autorip. See
(internal)/memory/0_18_redesign.md.

Single contributor: MattJackson (no attribution trailers).
2026-05-09 08:50:13 -07:00
MattJackson 6ec97af104 v0.17.13: thread Writer through patch + mux for big-write consistency
The bounded-cache writeback wrapper (crate::io::Writer) was added in
0.17.10 and wired into Disc::sweep in 0.17.11, but the other two
paths in the crate that write large amounts of data sequentially —
Disc::patch and the MKV/M2TS mux — were still operating on raw
std::fs::File. That meant the dirty-page burst pathology the wrapper
exists to prevent could still bite on slow / network-attached staging
during recovery and mux phases.

This release plugs those gaps:

- Disc::patch (disc/mod.rs:1981) now wraps the reopened ISO in
  Writer before any seek / write. sync_all on Writer cleanly drains
  the in-flight chunk before the existing fsync.
- mux/resolve.rs MKV and M2TS branches wrap the output File in
  Writer underneath BufWriter. UHD MKV mux routinely produces 70+ GB
  of sequential output; the page cache no longer absorbs that as a
  single hot blast on slow targets.

Mapfile, log, settings, history, and stream-pipeline byte buffers
remain unchanged: those are either small one-shot writes (where
the wrapper has zero benefit and adds a stream_position syscall) or
already use bounded persistence (mapfile time-batched in 0.17.12).
The principle: any path that writes substantial sequential data to
a single file uses Writer; trivial writes don't.
2026-05-09 06:32:08 -07:00
38 changed files with 3522 additions and 944 deletions
+142
View File
@@ -1,5 +1,147 @@
# Changelog
## 0.18.2 (2026-05-09)
### Bug fixes
- **Nav-file scramble during AACS rip** (`decrypt::decrypt_sectors`). 0.18's
`DecryptingSectorSource` decorator broadened the call surface of
`decrypt_sectors` to every sector flowing through sweep, including UDF
navigation files (MPLS playlists, CLPI clip-info). The byte-0 heuristic
in `aacs::is_unit_encrypted` correctly fires on m2ts source-packet copy
markers but false-positives on any binary file whose first byte happens
to have the top 2 bits set — most notably MPLS files (start with 'M' =
0x4D) and CLPI files (start with 'H' = 0x48). `decrypt_unit_full`
already self-checks the result via TS-sync verification and returns
`false` on a misfire, but the chunk had been mutated by then;
`decrypt_sectors` discarded the return value, leaving scrambled bytes
in the ISO. Fix: snapshot the chunk before decryption and restore on
verification failure (same pattern `decrypt_unit_try_keys` already used
for multi-key discs). Symptom: `freemkv info iso://UHD.iso` and
`iso:// → mkv://` returned E6009 NoStreams on freshly-ripped UHD ISOs;
affected only the iso-source path, not the disc-source path or the
m2ts video payload itself. Regression test:
`decrypt::tests::nav_file_unit_survives_decrypt_attempt`.
- **Sweep progress display can regress to zero** (`Disc::sweep`). The
consumer-side `bytes_good` snapshot lags producer-side `bytes_done`
whenever the consumer is behind on draining the work channel. Until
the first snapshot arrived, the placeholder branch reported
`bytes_done` correctly; once a stale snapshot landed, the report
switched to `snap.bytes_good` and could regress below `bytes_done`.
Fix: `bytes_good_total = max(snap.bytes_good, bytes_done)`, so the
user-visible counter never moves backward.
## 0.18.1 (2026-05-09)
### I/O stack redesign — primitives over orchestration
0.18 reshapes the read/write surface from "library does the multipass dance"
to "library hands the caller flat verbs and a few composable primitives." All
in-tree consumers (autorip, the `freemkv` CLI) drove their own multipass
loops in 0.18 rounds 1-3; 0.18.1 lands the cleanup with the deprecated names
still alive for one minor-version window.
The 0.18 design notes are in
`(internal)/memory/0_18_redesign.md` (private) — this entry sticks to
what changed at the public surface.
#### Flat verbs
`Disc::sweep` is the forward Pass 1. `Disc::patch` is one retry pass over the
mapfile. Neither knows about pass index, retry budget, or accept-loss policy
— the caller invokes them in whatever sequence its use case dictates. The
old multipass-aware `Disc::copy` dispatcher is deprecated and slated for
deletion in 0.18.2; no in-tree caller still uses it.
#### Trait splits — direction-typed at compile time
- `pes::Stream` (combined read+write) is split into `FrameSource` and
`FrameSink`. Calling `read()` on a write-only sink is now a compile error,
not the runtime `E9001` (`StreamWriteOnly`) it used to be.
- `SectorReader` is split into `SectorSource` (read) and `SectorSink`
(write). `Drive` impls `SectorSource` only; `FileSectorSource` /
`FileSectorSink` replace `FileSectorReader` for ISO-backed I/O.
- A blanket impl bridges legacy `SectorReader` callers onto the new
`SectorSource` so existing code keeps compiling through the deprecation
window.
#### New primitives
- `Halt` — one cancellation token (cloneable, `Arc<AtomicBool>` under the
hood) replaces the three near-duplicate halt flags scattered through the
workspace. Threaded through every long-running loop.
- `Pipeline<I, R>` + `Sink<I>` — generic producer/consumer primitive in
`crate::io`. Replaces the bespoke `disc/sweep_pipeline.rs` and now also
drives `Disc::patch` and the autorip mux loop. `DEFAULT_PIPELINE_DEPTH`
for streaming reads (`4`); `WRITE_THROUGH_DEPTH` for write-through patch
semantics (`1`).
- `WritebackFile` — was `crate::io::Writer`. The renamed type makes its job
explicit: a `File` wrapper that runs continuous `sync_file_range` +
`posix_fadvise(DONTNEED)` to keep the kernel dirty-page cache bounded on
long sequential writes.
- `DecryptingSectorSource<S>` — a single decorator wrapping any
`SectorSource` to yield plaintext sectors. One audit surface for AACS /
CSS / passthrough; the previous two-site decrypt (sweep producer +
`DiscStream` demux) is gone.
#### Throughput
The round-2 producer/consumer split is now applied uniformly to sweep,
patch, and mux. Mux on NFS-staged UHD measured ~16 MB/s sustained on the
test bed (was ~12 MB/s pre-round-2).
#### Module reorg
- `sector/` and `io/` are now module directories.
- `disc/sweep.rs`, `disc/patch.rs`, and `disc/mapfile.rs` split out of the
monolithic `disc/mod.rs`.
#### Renames (no behavior change)
- `crate::io::Writer` → `crate::io::WritebackFile`.
- `Apply` → `Flow` (`Sink::apply` return value).
- `DEFAULT_DEPTH` → `DEFAULT_PIPELINE_DEPTH`.
- `PatchOpts` → `PatchOptions`.
#### Deprecated (alive in 0.18.1, deletion target 0.18.2)
`Disc::copy`, `pes::Stream`, `SectorReader`, `FileSectorReader`,
`CopyOptions`, `CopyResult`, `DiscStream::set_halt`. Each compiles with a
deprecation warning; all in-tree call sites have migrated.
## 0.17.13 (2026-05-09)
### Use `crate::io::Writer` uniformly for all binary file output
0.17.10 introduced the bounded-cache writeback wrapper (`sync_file_range`
+ `posix_fadvise(DONTNEED)` per chunk) and 0.17.11 wired it into
`Disc::sweep`. The other two big-write paths in the crate were still
opening raw `std::fs::File` and would have hit the same dirty-page
burst pathology against slow / network-attached staging.
This release threads `crate::io::Writer` through the remaining sites:
- **`Disc::patch`** (`disc/mod.rs:1981`): the ISO file reopened for
Pass-N recovery now wraps in `Writer` before any seek / write.
Recovery writes are sparse, but the wrapper costs nothing when
there's no chunk crossing — and on heavily-damaged discs it
matters as patch accumulates GB of recovered data.
- **MKV mux output** (`mux/resolve.rs:243`): `BufWriter::with_capacity`
now wraps `Writer::new(file)` instead of a raw `File`. UHD MKVs
routinely exceed 70 GB of sequential writes; pre-0.17.13 those
bursts went straight to the kernel writeback queue.
- **M2TS mux output** (`mux/resolve.rs:251`): same change for
parity with the MKV path — anyone using `m2ts://` URLs gets it
too.
No new public surface. `Writer::sync_all()` is called from `Disc::patch`
and `sweep_pipeline`'s consumer at end of pass; mux exits via Drop on
the wrapping `BufWriter`, which propagates flush down to `Writer`'s
final `note_progress` (kernel finishes the in-flight chunk on file
close — there's no explicit `sync_all` for mux today, same behaviour
as before).
## 0.17.12 (2026-05-09)
### Mapfile time-batched persistence — unblock NFS staging
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "libfreemkv"
version = "0.17.12"
version = "0.18.2"
edition = "2024"
rust-version = "1.86"
license = "AGPL-3.0-only"
-34
View File
@@ -1,34 +0,0 @@
# libfreemkv — Feature List
## v0.10.10 (current)
### Done
- [x] Drive access: open, identify, unlock, firmware upload, speed calibration, eject
- [x] 206 bundled drive profiles (MediaTek MT1959 A + B variants)
- [x] SCSI transport: Linux SG_IO, macOS IOKit, Windows SPTI
- [x] UDF 2.50 filesystem parser (metadata partitions, Blu-ray profile)
- [x] MPLS playlist parser (play items, STN table, secondary streams)
- [x] CLPI clip info parser (EP map, sector extents)
- [x] AACS 1.0 decryption (4 VUK paths: KEYDB, media key, processing key, device key)
- [x] AACS 2.0 SCSI handshake (P-256/SHA-256 ECDH, bus decryption, read data key)
- [x] DVD IFO parser (VMG, VTS, PGC chains, cell addresses)
- [x] DVD CSS decryption (bus auth, disc key via player keys, title key, sector descramble)
- [x] CSS Stevenson plaintext attack for ISO key recovery
- [x] MPEG-2 PS demuxer (DVD Program Stream with PES extraction)
- [x] MPEG-2 video codec parser (sequence header, quantizer matrices, keyframe detection)
- [x] KEYDB.cfg download, verify, save (raw TCP, zero HTTP deps)
- [x] Content reading with adaptive batch sizing, error recovery, 12+ MB/s
- [x] Stream labels: 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
- [x] MKV muxer: 14 codec parsers (H.264, HEVC, MPEG-2, AC-3, DTS, TrueHD, PGS, DVD Sub, VC-1, LPCM, +4 more)
- [x] SectorReader trait: decouples disc scanning from SCSI
- [x] 7 stream types: Disc, ISO, MKV, M2TS, Network, Stdio, Null
- [x] PES pipeline with unified Stream trait (any source → any dest)
- [x] FMKV metadata header for M2TS and network streams
- [x] Numeric error codes only (no English text in library)
- [x] Event system for progress callbacks
### Planned
- [ ] Windows testing on real hardware
- [ ] Pioneer Renesas platform support (48 drives, need GET_CONFIG 010C)
- [ ] TranscodeStream (ffmpeg integration)
- [ ] ISO write with BD-compliant UDF structure
+41 -19
View File
@@ -18,7 +18,7 @@ Part of the [freemkv](https://github.com/freemkv) project.
```toml
[dependencies]
libfreemkv = "0.17"
libfreemkv = "0.18"
```
## Quick Start
@@ -53,27 +53,48 @@ output.finish()?;
### Multi-pass recovery rip
For damaged discs, the library offers a two-stage rip model: fast sweep with zero-fill and a ddrescue-format mapfile, then targeted retry of bad ranges. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for the full architecture.
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
loops; the multipass policy is the caller's job. See
[`docs/rip-recovery.md`](docs/rip-recovery.md) and the design notes in
`(internal)/memory/0_18_redesign.md`.
```rust
use libfreemkv::disc::{CopyOptions, PatchOptions};
use libfreemkv::{SweepOptions, PatchOptions};
use libfreemkv::disc::{mapfile, mapfile_path_for};
use std::path::Path;
// Pass 1: disc → ISO. Fast 64 KB reads, skip-forward on failure,
// zero-fill bad blocks, write a sidecar .mapfile.
let mut result = disc.copy(
&mut drive,
Path::new("disc.iso"),
&CopyOptions { skip_on_error: true, skip_forward: true, ..Default::default() },
)?;
let iso = Path::new("disc.iso");
// Pass 2..N: retry bad ranges with full drive recovery.
// Idempotent — call as many times as you want.
while result.bytes_unreadable + result.bytes_pending > 0 {
let pr = disc.patch(&mut drive, Path::new("disc.iso"), &PatchOptions::default())?;
if pr.bytes_recovered_this_pass == 0 { break; }
// Pass 1: disc → ISO. Skip-on-error, zero-fill, write the sidecar mapfile.
disc.sweep(&mut drive, iso, &SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true,
progress: None,
halt: None,
})?;
// Pass 2..N: retry every non-finished range. Idempotent.
loop {
let map = mapfile::Mapfile::load(&mapfile_path_for(iso))?;
let stats = map.stats();
if stats.bytes_pending + stats.bytes_unreadable == 0 { break; }
let outcome = disc.patch(&mut drive, iso, &PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true,
wedged_threshold: 50,
progress: None,
halt: None,
})?;
if outcome.bytes_recovered_this_pass == 0 { break; }
}
// Then mux from the ISO via the normal stream pipeline (no drive involvement).
// Mux from the ISO via the normal stream pipeline (no drive involvement).
```
## What It Does
@@ -92,14 +113,14 @@ while result.bytes_unreadable + result.bytes_pending > 0 {
| Stream | Input | Output | Transport |
|--------|-------|--------|-----------|
| DiscStream | Yes | -- | Optical drive via SCSI |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::copy()`) |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::sweep()`) |
| MkvStream | Yes | Yes | Matroska container |
| M2tsStream | Yes | Yes | BD transport stream with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement `pes::Stream` (frame-level). `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
AACS decryption requires a KEYDB.cfg file. If available at `~/.config/aacs/KEYDB.cfg` or passed via `ScanOptions`, the library handles everything — handshake, key derivation, and per-sector decryption — without the application needing to know anything about encryption.
@@ -122,7 +143,8 @@ Disc — scan titles, streams, AACS/CSS state
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── pes::Stream — read()/write() PES frames
├── FrameSource — read() PES frames (direction-typed)
├── FrameSink — write() PES frames (direction-typed)
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
+39 -37
View File
@@ -44,35 +44,30 @@ while let Ok(Some(frame)) = input.read() {
output.finish()?;
```
The `pes::Stream` trait:
The `FrameSource` and `FrameSink` traits — direction is type-checked, so
calling `read()` on a write-only sink (or `write()` on a read-only source)
is a compile error rather than a runtime fault:
```rust
pub trait Stream {
fn read(&mut self) -> io::Result<Option<PesFrame>>;
fn write(&mut self, frame: &PesFrame) -> io::Result<()>;
fn finish(&mut self) -> io::Result<()>;
pub trait FrameSource: Send {
fn read(&mut self) -> Result<Option<PesFrame>, Error>;
fn info(&self) -> &DiscTitle;
fn codec_private(&self, track: usize) -> Option<Vec<u8>>;
fn headers_ready(&self) -> bool;
fn codec_private(&self, track: usize) -> Option<Vec<u8>> { None }
fn headers_ready(&self) -> bool { true }
}
```
## IOStream (byte-level API)
For raw byte copies (disc→ISO, resume, benchmarks). Lower level than PES.
```rust
let opts = InputOptions::default();
let mut input = open_input("iso://Disc.iso", &opts)?;
let mut output = open_output("mkv://Movie.mkv", input.info())?;
io::copy(&mut *input, &mut *output)?;
output.finish()?;
pub trait FrameSink: Send {
fn write(&mut self, frame: &PesFrame) -> Result<(), Error>;
fn finish(self: Box<Self>) -> Result<(), Error>;
fn info(&self) -> &DiscTitle;
}
```
## Streams
All streams implement `IOStream` (byte-level) and/or `pes::Stream` (frame-level).
URL-based resolvers open any stream by string.
All streams implement `FrameSource` (read) and/or `FrameSink` (write); the
directional split prevents runtime "wrong-direction" errors. URL-based
resolvers open any stream by string.
| Stream | Input | Output | URL | Transport |
|--------|-------|--------|-----|-----------|
@@ -87,21 +82,14 @@ URL-based resolvers open any stream by string.
All URLs require a `scheme://path` format. Bare paths are rejected.
```rust
// PES pipeline (frame-level)
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://10.0.0.1:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
// IOStream (byte-level)
let input = open_input("disc://", &opts)?; // DiscStream
let input = open_input("iso://Dune.iso", &opts)?; // IsoStream
let output = open_output("iso://Copy.iso", &meta)?; // IsoStream (write)
let output = open_output("mkv://Dune.mkv", &meta)?; // MkvStream
let output = open_output("m2ts://Dune.m2ts", &meta)?; // M2tsStream
let output = open_output("null://", &meta)?; // NullStream
```
### FMKV Metadata Header
@@ -175,20 +163,34 @@ libfreemkv/src/
├── lib.rs Public exports
├── error.rs Error codes (no English)
├── event.rs Event types for callbacks
├── halt.rs Halt cancellation token (Arc<AtomicBool> wrapper)
├── io/ Pipeline + WritebackFile primitives
│ ├── mod.rs Re-exports WritebackFile, Pipeline, Sink, Flow
│ ├── pipeline.rs Generic Pipeline<I, R> + Sink trait
│ ├── writeback_file.rs WritebackFile (was crate::io::Writer)
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Drive profile capture for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup)
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── platform/ Drive unlock (MT1959 A/B)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, Stream trait
├── sector.rs SectorReader trait
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O (was sector.rs in 0.17)
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
├── udf.rs UDF 2.50 filesystem parser
├── mpls.rs MPLS playlist parser
├── clpi.rs CLPI clip info parser
@@ -199,15 +201,15 @@ libfreemkv/src/
├── profile.rs Bundled drive profiles
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs IOStream trait, public exports
│ ├── resolve.rs URL parser + open_input/open_output + input/output
│ ├── mod.rs Public mux exports
│ ├── resolve.rs URL parser + input/output (Box<dyn FrameSource/Sink>)
│ ├── meta.rs FMKV header format
│ ├── disc.rs DiscStream (optical drive → PES)
│ ├── iso.rs IsoStream (ISO image read/write)
│ ├── iso.rs IsoStream (ISO image read)
│ ├── isowriter.rs ISO image writer (UDF, AVDP, multi-extent)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska, IOStream)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska)
│ ├── mkvout.rs MkvOutputStream (PES → MKV)
│ ├── m2ts.rs M2tsStream (BD-TS, IOStream)
│ ├── m2ts.rs M2tsStream (BD-TS)
│ ├── pesout.rs PES output streams (M2ts, Network, Stdio, Null)
│ ├── network.rs NetworkStream (TCP + FMKV header)
│ ├── stdio.rs StdioStream (stdin/stdout pipe)
+6 -2
View File
@@ -65,8 +65,12 @@ libfreemkv (lib.rs)
│
├── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types, Stream trait (read/write frames)
│ └── sector SectorReader trait — abstracts disc vs ISO vs file
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
│
├── I/O Primitives
│ ├── halt Halt cancellation token (one Arc<AtomicBool>, cloneable)
│ └── io/ Pipeline<I, R> + Sink trait + WritebackFile (bounded-cache writer)
│
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
+8 -4
View File
@@ -97,7 +97,9 @@ drive.probe_disc()?;
// Scan disc (UDF + playlists + AACS — all automatic)
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Stream pipeline — PES frames from any source to any output
// Stream pipeline — PES frames from any source to any output.
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// direction is type-checked, so calling .write() on an input is a compile error.
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
@@ -121,11 +123,13 @@ output.finish()?;
| aacs/ | [aacs.md](aacs.md) | Key resolution + content decrypt + bus handshake |
| css/ | -- | DVD CSS cipher |
| decrypt.rs | -- | Unified decrypt dispatcher (AACS/CSS/None) |
| disc/ | -- | High-level scan + read API |
| disc/ | [rip-recovery.md](rip-recovery.md) | Disc::scan + Disc::sweep + Disc::patch + mapfile |
| labels/ | -- | BD-J stream labels (5 format parsers) |
| mux/ | -- | Stream implementations (7 stream types) |
| pes.rs | -- | PES frame types + Stream trait |
| sector.rs | -- | SectorReader trait |
| pes.rs | -- | PES frame types + FrameSource / FrameSink traits |
| sector/ | -- | SectorSource / SectorSink + DecryptingSectorSource decorator |
| io/ | -- | Pipeline<I, R> + Sink trait + WritebackFile |
| halt.rs | -- | Halt cancellation token |
| keydb.rs | -- | KEYDB download, parse, save |
| error.rs | -- | Error codes (E1xxx-E8xxx) |
| event.rs | -- | Drive event system |
+2 -2
View File
@@ -52,8 +52,8 @@ selects the per-CDB timeout:
| `recovery` | Timeout | Used by |
|------------|----------|------------------------------------------|
| `false` | 1.5 s | `Disc::copy` fast skip-forward sweep, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` multi-pass over the mapfile |
| `false` | 1.5 s | `Disc::sweep` fast skip-forward pass, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` retry pass over the mapfile |
On any SCSI failure or timeout, `read` returns `Err(DiscRead)` immediately.
There are no inline retries, no SCSI reset, no Phase 1/2/3 escalation.
+43 -22
View File
@@ -12,21 +12,24 @@ reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (multi-pass over the mapfile) | Re-reads non-`+` ranges with the long timeout. Idempotent; call N times. |
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The caller orchestrates layer 1. Autorip's `rip_disc` loops `copy` then
N × `patch` per the `MAX_RETRIES` config, then hands the ISO off to the
existing mux pipeline. Layer 3 runs inside any consumer of `DiscStream`
(direct PES pipeline, ISO playback, etc.) without caller involvement.
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
involvement.
Three primitives compose the disc-side flow:
| Primitive | What it does |
|---------------------------|-----------------------------------------------------------------------|
| `Disc::copy` | disc → ISO. Writes a sidecar `.mapfile`. Opt-in skip-forward on failure. |
| `Disc::patch` | Re-reads bad ranges from the drive. Idempotent; call N times. |
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
## Data model
@@ -60,28 +63,46 @@ Status characters match ddrescue:
Position and size are hex byte offsets into the ISO.
### `CopyOptions` and `PatchOptions`
### `SweepOptions` and `PatchOptions`
`Disc::copy()` auto-detects the pass from mapfile state:
The library no longer dispatches between sweep and patch internally — the
caller picks the verb explicitly per pass. The two option structs are flat
and have no overlap:
```rust
CopyOptions {
decrypt: true, // decrypt AACS/CSS sectors
multipass: true, // enable mapfile + skip-on-error + damage-jump
progress: Some(&reporter), // progress callback
halt: Some(flag), // halt flag for graceful stop
SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true, // damage-jump + zero-fill on read failure
progress: Some(&reporter),
halt: Some(flag),
}
PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true, // walk bad ranges high → low LBA
wedged_threshold: 50,
progress: Some(&reporter),
halt: Some(flag),
}
```
Dispatch logic:
- No mapfile → sweep (fresh Pass 1)
- Mapfile with NonTried (`?`) → sweep with resume
- Mapfile covering full disc, only NonTrimmed/NonScraped/Unreadable → patch
- Mapfile clean → no-op
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
- No mapfile → `sweep` (fresh Pass 1).
- Mapfile with `?` ranges → `sweep` with `resume: true`.
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
- Mapfile clean → done; no further pass needed.
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
lines of `Mapfile::stats()` checks.
## Algorithm
### Pass 1 — fast sweep (`Disc::copy` → `sweep_internal`)
### Pass 1 — fast sweep (`Disc::sweep`)
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
2. On success: write data to ISO, mark `+`, advance.
@@ -95,7 +116,7 @@ Dispatch logic:
Pass 1 completes when every byte has been visited (either `+` or `*`).
### Pass 2+ — patch (`Disc::copy` → `patch_internal`)
### Pass 2+ — patch (`Disc::patch`)
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
(walks ranges from highest LBA to lowest, within each range from end to start).
@@ -177,4 +198,4 @@ scrape vs. retry with direction reversal) if there's measured benefit.
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/mod.rs`](../src/disc/mod.rs) (`Disc::copy`, `Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).
+1 -1
View File
@@ -126,7 +126,7 @@ USB optical drives have ~500ms round-trip latency per SCSI command. Since `read_
`Disc::scan()` wraps the drive in a `BufferedSectorReader` before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's `max_hw_sectors_kb` for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorReader` implementation.
The buffer is transparent -- `read_filesystem()`, `read_file()`, and all downstream code still call `read_sectors(lba, 1, buf)` as before. The batching happens inside the `SectorSource` implementation.
### UDF Filename Encoding
+50 -1
View File
@@ -62,7 +62,23 @@ pub fn decrypt_sectors(
for chunk in buf.chunks_mut(unit_len) {
if chunk.len() == unit_len && aacs::is_unit_encrypted(chunk) {
aacs::decrypt_unit_full(chunk, &uk, rdk);
// `is_unit_encrypted` is a byte-0 heuristic: it fires on any
// unit whose first byte has the top 2 bits set, which is
// correct for m2ts source packets (where those bits are the
// copy-control marker) but false-positives on any other binary
// data with similarly-shaped first bytes — notably MPLS/CLPI
// navigation files that begin with ASCII magic ('M', 'H'…)
// and survive sweep mixed in with encrypted m2ts payloads.
// `decrypt_unit_full` self-checks via TS-sync verification and
// returns false on a misfire, but it has already mutated the
// chunk by then. Snapshot and restore on verification failure
// — same pattern `decrypt_unit_try_keys` uses for multi-key
// discs. Real m2ts units verify and stay decrypted; nav-file
// sectors get scrambled briefly and then put back as-was.
let original: Vec<u8> = chunk.to_vec();
if !aacs::decrypt_unit_full(chunk, &uk, rdk) {
chunk.copy_from_slice(&original);
}
}
}
}
@@ -74,3 +90,36 @@ pub fn decrypt_sectors(
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// Regression for the 0.18.1 nav-file scramble bug. A non-m2ts unit whose
/// first byte has the top 2 bits set (here: the ASCII letter 'M' that
/// MPLS files start with, 0x4D = 0b01001101) trips `is_unit_encrypted`,
/// gets AES-decrypted with the unit key, fails the TS-sync verification,
/// and must be restored to its original bytes — not left scrambled.
#[test]
fn nav_file_unit_survives_decrypt_attempt() {
let mut unit = vec![0u8; aacs::ALIGNED_UNIT_LEN];
unit[0] = b'M';
unit[1] = b'P';
unit[2] = b'L';
unit[3] = b'S';
for (i, b) in unit.iter_mut().enumerate().skip(4) {
*b = (i as u8).wrapping_mul(31);
}
let snapshot = unit.clone();
let keys = DecryptKeys::Aacs {
unit_keys: vec![(0, [0xAB; 16])],
read_data_key: None,
};
decrypt_sectors(&mut unit, &keys, 0).unwrap();
assert_eq!(
unit, snapshot,
"non-m2ts unit must be restored after failed decrypt"
);
}
}
+356 -232
View File
@@ -12,8 +12,9 @@ mod bluray;
mod dvd;
mod encrypt;
pub mod mapfile;
mod patch;
pub mod read_error;
mod sweep_pipeline;
mod sweep;
use crate::drive::{Drive, extract_scsi_context};
use crate::error::{Error, Result};
@@ -1332,7 +1333,7 @@ impl Disc {
path: &std::path::Path,
opts: &CopyOptions,
) -> Result<CopyResult> {
let patch_opts = PatchOpts {
let patch_opts = PatchOptions {
decrypt: opts.decrypt,
block_sectors: Some(1),
full_recovery: true,
@@ -1364,16 +1365,25 @@ impl Disc {
})
}
fn sweep(
/// Pass 1 of a multipass rip: walk the disc forward, write
/// every readable sector into `path`, and record the result
/// in the sidecar mapfile. With `skip_on_error: true`, a bad
/// sector zero-fills + marks `NonTrimmed` and the sweep keeps
/// going (jumping ahead through dense damage); without it,
/// the first read failure aborts.
///
/// 0.18: this is one of the two flat verbs the library exposes
/// for rip orchestration. Multipass + retry decisions are the
/// caller's job — see [`PatchOptions`] for the retry primitive.
pub fn sweep(
&self,
reader: &mut dyn SectorReader,
path: &std::path::Path,
opts: &SweepOptions,
) -> Result<CopyResult> {
use sweep_pipeline::{
ConsumerInputs, ProgressSnapshot, WorkItem, send_or_abort, spawn_consumer,
try_recv_progress, try_request_stats,
};
use crate::io::{DEFAULT_PIPELINE_DEPTH, Pipeline};
use crate::sector::{DecryptingSectorSource, SectorSource};
use sweep::{ProgressSnapshot, SweepSink, WorkItem, try_recv_progress};
let total_bytes = self.capacity_sectors as u64 * 2048;
let keys = if opts.decrypt {
@@ -1382,6 +1392,15 @@ impl Disc {
crate::decrypt::DecryptKeys::None
};
// Wrap the producer-side reader once so every read_sectors call
// yields plaintext. `DecryptKeys::None` makes the decorator a
// pass-through, so the wrapping is cheap when --raw / unencrypted
// discs are being swept and we keep the pipeline shape uniform.
// Replaces the inline `decrypt::decrypt_sectors` calls that used
// to live in this loop and in the bisect inner loop below.
let mut reader = DecryptingSectorSource::new(reader, keys);
let reader = &mut reader;
// Mapfile: load if resuming, else wipe + recreate.
let mapfile_path = self.mapfile_for(path);
if !opts.resume {
@@ -1418,10 +1437,11 @@ impl Disc {
f
};
// Wrap the raw `File` in our bounded-cache writer (drains
// dirty pages continuously instead of bursting; see
// `crate::io`). The Writer moves into the consumer thread.
let file = crate::io::Writer::new(file).map_err(|e| Error::IoError { source: e })?;
// Wrap the raw `File` in our bounded-cache `WritebackFile`
// (drains dirty pages continuously instead of bursting; see
// `crate::io`). The `WritebackFile` moves into the consumer
// thread.
let file = crate::io::WritebackFile::new(file).map_err(|e| Error::IoError { source: e })?;
let batch: u16 = match opts.batch_sectors {
Some(b) => b,
None if opts.skip_on_error => ecc_sectors(self.format),
@@ -1435,13 +1455,22 @@ impl Disc {
// sweep finishes are the patch pass's job.
let regions: Vec<(u64, u64)> = map.ranges_with(&[mapfile::SectorStatus::NonTried]);
// Spawn the consumer. It owns Writer + Mapfile; the producer
// Spawn the consumer. It owns WritebackFile + Mapfile; the producer
// (this thread) keeps `reader`, `read_ctx`, halt + set_speed.
let (work_tx, prog_rx, consumer_handle) = spawn_consumer(ConsumerInputs {
file,
map,
is_regular,
});
// The thread name is preserved from the 0.17.x sweep_pipeline so it
// stays identifiable in stack traces / `top -H`.
let (sink, prog_rx) = SweepSink::new(file, map, is_regular);
let pipe: Pipeline<WorkItem, sweep::ConsumerSummary> =
Pipeline::spawn_named("freemkv-sweep-consumer", DEFAULT_PIPELINE_DEPTH, sink)?;
// Translate `Pipeline::send` failure (consumer gone) into the
// same `Error` shape the 0.17.x `send_or_abort` produced, so
// the producer-error semantics are unchanged.
fn consumer_gone() -> Error {
Error::IoError {
source: std::io::Error::other("sweep consumer terminated unexpectedly"),
}
}
let mut buf = vec![0u8; batch as usize * 2048];
let mut bytes_done = 0u64;
@@ -1519,23 +1548,17 @@ impl Disc {
}
read_ctx.bridge_degradation_count = 0;
// Decrypt on producer; consumer expects plaintext.
if opts.decrypt {
crate::decrypt::decrypt_sectors(
&mut buf[..block_bytes as usize],
&keys,
0,
)?;
}
// Plaintext: the wrapped reader (DecryptingSectorSource)
// applied AACS / CSS in-place during read_sectors above.
// The consumer thread sees decrypted bytes; the
// pre-0.18 inline decrypt_sectors call lived here.
// Move the batch into the channel via fresh
// owned Vec. The producer's `buf` is reused
// for the next read.
let send_buf = buf[..block_bytes as usize].to_vec();
if let Err(e) =
send_or_abort(&work_tx, WorkItem::Good { pos, buf: send_buf })
{
producer_err = Some(e);
if pipe.send(WorkItem::Good { pos, buf: send_buf }).is_err() {
producer_err = Some(consumer_gone());
break 'outer;
}
bytes_done = bytes_done.saturating_add(block_bytes);
@@ -1584,24 +1607,17 @@ impl Disc {
) {
Ok(_) => {
read_ctx.on_success();
// Decrypt single sector before send. Pre-split
// bisect path silently skipped this — encrypted
// bytes were written for bisect-recovered sectors.
if opts.decrypt {
crate::decrypt::decrypt_sectors(
&mut sector_buf,
&keys,
0,
)?;
}
if let Err(e) = send_or_abort(
&work_tx,
WorkItem::BisectGood {
// Plaintext via the wrapping
// DecryptingSectorSource — same
// decrypt path the batch read takes.
if pipe
.send(WorkItem::BisectGood {
pos: write_pos,
buf: Box::new(sector_buf),
},
) {
producer_err = Some(e);
})
.is_err()
{
producer_err = Some(consumer_gone());
bisect_aborted = true;
break;
}
@@ -1611,11 +1627,11 @@ impl Disc {
&inner_err,
&mut read_ctx,
);
if let Err(e) = send_or_abort(
&work_tx,
WorkItem::BisectBad { pos: write_pos },
) {
producer_err = Some(e);
if pipe
.send(WorkItem::BisectBad { pos: write_pos })
.is_err()
{
producer_err = Some(consumer_gone());
bisect_aborted = true;
break;
}
@@ -1631,14 +1647,14 @@ impl Disc {
pos += block_bytes;
}
read_error::ReadAction::SkipBlock { pause_secs } => {
if let Err(e) = send_or_abort(
&work_tx,
WorkItem::SkipFill {
if pipe
.send(WorkItem::SkipFill {
pos,
len: block_bytes,
},
) {
producer_err = Some(e);
})
.is_err()
{
producer_err = Some(consumer_gone());
break 'outer;
}
bytes_done = bytes_done.saturating_add(block_bytes);
@@ -1651,14 +1667,14 @@ impl Disc {
sectors,
pause_secs,
} => {
if let Err(e) = send_or_abort(
&work_tx,
WorkItem::SkipFill {
if pipe
.send(WorkItem::SkipFill {
pos,
len: block_bytes,
},
) {
producer_err = Some(e);
})
.is_err()
{
producer_err = Some(consumer_gone());
break 'outer;
}
bytes_done = bytes_done.saturating_add(block_bytes);
@@ -1678,14 +1694,14 @@ impl Disc {
let gap_start = pos + block_bytes;
let gap_bytes = jump_pos.saturating_sub(gap_start);
if gap_bytes > 0 {
if let Err(e) = send_or_abort(
&work_tx,
WorkItem::GapFill {
if pipe
.send(WorkItem::GapFill {
pos: gap_start,
len: gap_bytes,
},
) {
producer_err = Some(e);
})
.is_err()
{
producer_err = Some(consumer_gone());
break 'outer;
}
bytes_done = bytes_done.saturating_add(gap_bytes);
@@ -1740,8 +1756,11 @@ impl Disc {
"Disc::sweep inner iter"
);
}
// Throttled stats refresh request.
try_request_stats(&work_tx);
// Throttled stats refresh request — best-effort
// try_send so a busy consumer doesn't stall the
// producer; the cached snapshot stays current
// enough for one more iteration.
let _ = pipe.try_send(WorkItem::StatsRequest);
}
if let Some(reporter) = opts.progress {
@@ -1762,9 +1781,18 @@ impl Disc {
.unwrap_or(0),
None => 0,
};
// The consumer's snapshot is the source of truth for
// bytes_unreadable / bytes_pending (the producer doesn't
// see them), but its bytes_good lags producer-side
// `bytes_done` whenever the consumer is behind on draining
// the work channel. Take the max so the user-visible
// counter never regresses below what the producer has
// already sent — Anomaly B in the 0.18.1 prod test was
// this regression: a stale early snapshot pinned the
// display to 0 GB while bytes_done was already advancing.
let (bytes_good, bytes_unreadable, bytes_pending) = match &cached_snapshot {
Some(snap) => (
snap.stats.bytes_good,
snap.stats.bytes_good.max(bytes_done),
snap.stats.bytes_unreadable,
snap.stats.bytes_pending,
),
@@ -1791,24 +1819,25 @@ impl Disc {
}
}
// Tell the consumer we're done. Even on producer error, send
// Finish so the consumer drains cleanly and we get a summary.
let _ = work_tx.send(WorkItem::Finish);
drop(work_tx);
let summary = consumer_handle.join().map_err(|_| Error::IoError {
source: std::io::Error::other("sweep consumer thread panicked"),
})?;
// Producer side is done. Drop the channel and let the
// consumer drain whatever's still in flight, then run its
// close() (drain writeback, fsync, mapfile.flush) and return
// the final stats. On consumer panic `pipe.finish` returns
// the wrapped panic message via Error::IoError — same shape
// the previous `consumer_handle.join().map_err(...)` produced.
let summary = pipe.finish();
// Producer-side error wins over consumer-side (the read failure
// is what motivated quitting; the consumer's flush error, if
// any, is downstream).
if let Some(e) = producer_err {
// Drop the consumer's result if we already have a producer
// error, but propagate consumer-panic on top of nothing
// since that's strictly informative.
let _ = summary;
return Err(e);
}
if let Some(e) = summary.error {
return Err(e);
}
let summary = summary?;
let stats = summary.stats;
tracing::debug!(
@@ -1854,7 +1883,8 @@ pub struct CopyResult {
pub halted: bool,
}
pub(crate) struct SweepOptions<'a> {
/// Options for [`Disc::sweep`] (Pass 1 / forward sequential pass).
pub struct SweepOptions<'a> {
pub decrypt: bool,
pub resume: bool,
pub batch_sectors: Option<u16>,
@@ -1863,7 +1893,8 @@ pub(crate) struct SweepOptions<'a> {
pub halt: Option<std::sync::Arc<std::sync::atomic::AtomicBool>>,
}
pub(crate) struct PatchOpts<'a> {
/// Options for [`Disc::patch`] (Pass N retry pass over bad ranges).
pub struct PatchOptions<'a> {
pub decrypt: bool,
pub block_sectors: Option<u16>,
pub full_recovery: bool,
@@ -1873,8 +1904,8 @@ pub(crate) struct PatchOpts<'a> {
pub halt: Option<std::sync::Arc<std::sync::atomic::AtomicBool>>,
}
#[allow(dead_code)]
pub(crate) struct PatchOutcome {
/// Result returned by [`Disc::patch`].
pub struct PatchOutcome {
pub bytes_total: u64,
pub bytes_good: u64,
pub bytes_unreadable: u64,
@@ -1941,13 +1972,26 @@ impl Disc {
bytes_bad_in_title(title, &bad_ranges)
}
fn patch(
/// Pass 2..N of a multipass rip: re-read the bad ranges
/// recorded in the sidecar mapfile and try to recover them.
/// With `reverse: true` (the default for the recovery walker),
/// the bad-range walk runs end-to-start so escalating skips
/// converge on the actual bad sub-zones inside any
/// `NonTrimmed` block. Returns a [`PatchOutcome`] with
/// recovered byte counts and wedge-detection signals.
///
/// 0.18: paired with [`Disc::sweep`] as the library's other flat
/// rip-phase verb. Caller drives the retry loop and the
/// sweep-vs-patch dispatch.
pub fn patch(
&self,
reader: &mut dyn SectorReader,
path: &std::path::Path,
opts: &PatchOpts,
opts: &PatchOptions,
) -> Result<PatchOutcome> {
use std::io::{Seek, SeekFrom, Write};
use crate::io::pipeline::{Pipeline, WRITE_THROUGH_DEPTH};
use crate::sector::{DecryptingSectorSource, SectorSource};
use patch::{PatchItem, PatchSink};
const BRIDGE_DEGRADATION_PAUSE_SECS: u64 = 10;
const POST_FAILURE_PAUSE_SECS: u64 = 1;
@@ -1966,7 +2010,7 @@ impl Disc {
}
let mapfile_path = self.mapfile_for(path);
let mut map =
let map =
mapfile::Mapfile::load(&mapfile_path).map_err(|e| Error::IoError { source: e })?;
let total_bytes = map.total_size();
let keys = if opts.decrypt {
@@ -1975,13 +2019,81 @@ impl Disc {
crate::decrypt::DecryptKeys::None
};
// Wrap the producer-side reader once so every read_sectors
// call (the main recovery read, the backtrack read, and the
// non-NOT_READY retry read) yields plaintext. Replaces three
// inline decrypt_sectors call sites that all keyed off the
// same `keys`. `DecryptKeys::None` keeps the unencrypted /
// --raw path a pass-through.
let mut reader = DecryptingSectorSource::new(reader, keys);
let reader = &mut reader;
let is_regular = std::fs::metadata(path)
.map(|m| m.file_type().is_file())
.unwrap_or(false);
let mut file = std::fs::OpenOptions::new()
.write(true)
.open(path)
.map_err(|e| Error::IoError { source: e })?;
// Snapshot fields we need from the mapfile *before* it moves into
// the consumer thread: bytes_good baseline, total entries, the
// initial `bad_ranges` work list, and the start-of-patch
// diagnostic dump. The shared state (`shared`) republishes these
// throughout the pass; the consumer owns the live `Mapfile`.
let bytes_good_before = map.stats().bytes_good;
let bytes_good_start = bytes_good_before;
let initial_stats = map.stats();
let initial_entries: Vec<_> = map.entries().to_vec();
// Every retry pass acts on every non-Finished range. Including
// Unreadable means a sector that failed in pass N gets a fresh
// shot in pass N+1 — drive state evolves, the same read can
// succeed later. Each pass owns its own jumps/skips; if pass 5
// jumps over the same zone as pass 2, fine.
let mut bad_ranges = map.ranges_with(&[
mapfile::SectorStatus::NonTrimmed,
mapfile::SectorStatus::NonScraped,
mapfile::SectorStatus::Unreadable,
]);
if opts.reverse {
bad_ranges.reverse();
}
let work_total: u64 = bad_ranges.iter().map(|(_, sz)| *sz).sum();
// Spawn the consumer. The `WritebackFile` (same bounded-cache
// wrapper sweep uses, so patch's recovery writes — sparse but
// can be many across a damaged region — get the burst-flush
// protection on slow / NFS-backed staging) and the `Mapfile`
// both move into the sink. We hold an `Arc<Mutex<…>>` snapshot
// the sink republishes after every record so producer-side
// stall guards / progress callbacks can read consumer side-
// effects.
let (sink, shared) = PatchSink::new(path, map, is_regular)?;
// Why: WRITE_THROUGH_DEPTH (=1) — patch reads ONE sector per
// recovery decision and the producer's stall / damage-window
// logic checks consumer-published stats inline. Sweep's
// DEFAULT_PIPELINE_DEPTH (=4) would let several sectors of
// recovered bytes queue up between producer decisions and
// writes, which conflicts with the per-sector lockstep this
// loop was written against.
let pipe = Pipeline::<PatchItem, _>::spawn(WRITE_THROUGH_DEPTH, sink)?;
// Send a `PatchItem` and translate a `SendError` (consumer
// thread died / panicked) into a useful library error so the
// caller can propagate cleanly. Mirrors `sweep_pipeline.rs`'s
// `send_or_abort`.
let send_or_abort = |pipe: &Pipeline<PatchItem, _>, item: PatchItem| -> Result<()> {
pipe.send(item).map_err(|_| Error::IoError {
source: std::io::Error::other("patch consumer terminated unexpectedly"),
})
};
// Snapshot helper for producer-side stats reads. Holds the
// mutex briefly; we never read across operations so a fresh
// snapshot per call is fine.
let read_shared = |shared: &std::sync::Mutex<patch::SharedPatchState>| -> (
mapfile::MapStats,
Vec<(u64, u64)>,
) {
let g = shared.lock().expect("PatchSink shared state mutex poisoned");
(g.stats, g.bad_ranges.clone())
};
// Log ISO file size at patch start for write monitoring
if let Ok(metadata) = std::fs::metadata(path) {
@@ -1996,8 +2108,6 @@ impl Disc {
let block_sectors = opts.block_sectors.unwrap_or(1);
let recovery = opts.full_recovery;
let bytes_good_before = map.stats().bytes_good;
let bytes_good_start = bytes_good_before;
let mut halted = false;
let mut wedged_exit = false;
let mut blocks_attempted: u64 = 0;
@@ -2057,24 +2167,23 @@ impl Disc {
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_snapshot",
total_entries = map.entries().len(),
total_entries = initial_entries.len(),
bytes_good_before,
bytes_retryable = map.stats().bytes_retryable,
bytes_unreadable = map.stats().bytes_unreadable,
bytes_nontried = map.stats().bytes_nontried,
bytes_retryable = initial_stats.bytes_retryable,
bytes_unreadable = initial_stats.bytes_unreadable,
bytes_nontried = initial_stats.bytes_nontried,
"Mapfile state snapshot at patch start"
);
// Log first 10 and last 10 entries for inspection
let entries = map.entries();
if !entries.is_empty() {
if !initial_entries.is_empty() {
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_entries_start",
num_to_log = (entries.len().min(10)) as u32,
num_to_log = (initial_entries.len().min(10)) as u32,
"First 10 entries"
);
for entry in entries.iter().take(10) {
for entry in initial_entries.iter().take(10) {
tracing::debug!(
target: "freemkv::disc",
phase = "patch_mapfile_entry_start",
@@ -2085,14 +2194,14 @@ impl Disc {
);
}
}
if entries.len() > 10 {
if initial_entries.len() > 10 {
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_entries_end",
num_to_log = (entries.len().min(10)) as u32,
num_to_log = (initial_entries.len().min(10)) as u32,
"Last 10 entries"
);
for entry in entries.iter().skip(entries.len() - 10) {
for entry in initial_entries.iter().skip(initial_entries.len() - 10) {
tracing::debug!(
target: "freemkv::disc",
phase = "patch_mapfile_entry_end",
@@ -2104,21 +2213,6 @@ impl Disc {
}
}
// Every retry pass acts on every non-Finished range. Including
// Unreadable means a sector that failed in pass N gets a fresh
// shot in pass N+1 — drive state evolves, the same read can
// succeed later. Each pass owns its own jumps/skips; if pass 5
// jumps over the same zone as pass 2, fine.
let mut bad_ranges = map.ranges_with(&[
mapfile::SectorStatus::NonTrimmed,
mapfile::SectorStatus::NonScraped,
mapfile::SectorStatus::Unreadable,
]);
if opts.reverse {
bad_ranges.reverse();
}
let work_total: u64 = bad_ranges.iter().map(|(_, sz)| *sz).sum();
tracing::info!(
target: "freemkv::disc",
phase = "patch_bad_ranges",
@@ -2209,7 +2303,7 @@ impl Disc {
break;
}
let bytes_good_now = map.stats().bytes_good;
let bytes_good_now = read_shared(&shared).0.bytes_good;
if bytes_good_now > range_bytes_good {
range_bytes_good = bytes_good_now;
range_start = std::time::Instant::now();
@@ -2246,21 +2340,23 @@ impl Disc {
// patch-marked Unreadable sectors are actually readable.
let unmarked_bytes = block_end.saturating_sub(*range_pos);
if opts.reverse {
map.record(
*range_pos,
unmarked_bytes,
mapfile::SectorStatus::NonTrimmed,
)
.map_err(|e| Error::IoError { source: e })?;
send_or_abort(
&pipe,
PatchItem::NonTrimmed {
pos: *range_pos,
len: unmarked_bytes,
},
)?;
} else {
let remaining_start = *range_pos + (end - block_end);
if remaining_start < end {
map.record(
remaining_start,
end - remaining_start,
mapfile::SectorStatus::NonTrimmed,
)
.map_err(|e| Error::IoError { source: e })?;
send_or_abort(
&pipe,
PatchItem::NonTrimmed {
pos: remaining_start,
len: end - remaining_start,
},
)?;
}
}
// Continue to next range (break inner loop only)
@@ -2357,12 +2453,10 @@ impl Disc {
"Read succeeded"
);
}
if opts.decrypt {
crate::decrypt::decrypt_sectors(&mut buf[..bytes], &keys, 0)?;
}
// Plaintext: DecryptingSectorSource applied AACS / CSS
// in-place during the read_sectors call above. The
// pre-0.18 inline decrypt_sectors call lived here.
let write_start = std::time::Instant::now();
file.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
tracing::debug!(
target: "freemkv::disc",
phase = "patch_write_start",
@@ -2370,8 +2464,18 @@ impl Disc {
bytes,
"Starting ISO write"
);
file.write_all(&buf[..bytes])
.map_err(|e| Error::IoError { source: e })?;
// Hand the recovered bytes off to the consumer:
// seek + write + mapfile.record(Finished) all
// happen on the consumer thread, so the producer
// can immediately move on to the next read while
// these bytes are being committed.
send_or_abort(
&pipe,
PatchItem::Recovered {
pos,
buf: buf[..bytes].to_vec(),
},
)?;
let write_duration_ms = write_start.elapsed().as_millis();
tracing::info!(
target: "freemkv::disc",
@@ -2381,21 +2485,21 @@ impl Disc {
write_duration_ms,
"ISO write succeeded"
);
let mapfile_record_start = std::time::Instant::now();
map.record(pos, block_bytes, mapfile::SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
let mapfile_record_duration_ms = mapfile_record_start.elapsed().as_millis();
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_record_ok",
pos,
block_bytes,
mapfile_record_duration_ms,
"Mapfile record written"
"Mapfile record dispatched"
);
// Stall guard: watch bytes_good (real progress), not pos (advances on skips)
let bytes_good_now = map.stats().bytes_good;
// Stall guard: watch bytes_good (real progress),
// not pos (advances on skips). With the consumer
// running in its own thread, this read can lag
// by up to one item; the watchdog operates at
// STALL_SECS=3600 granularity so single-item lag
// is irrelevant.
let bytes_good_now = read_shared(&shared).0.bytes_good;
if bytes_good_now > bytes_good_last {
stall_start = std::time::Instant::now();
bytes_good_last = bytes_good_now;
@@ -2440,32 +2544,26 @@ impl Disc {
) {
Ok(_) => {
blocks_read_ok += 1;
if opts.decrypt {
crate::decrypt::decrypt_sectors(
&mut buf[..bt_bytes],
&keys,
0,
)?;
}
file.seek(SeekFrom::Start(bt_pos))
.map_err(|e| Error::IoError { source: e })?;
file.write_all(&buf[..bt_bytes])
.map_err(|e| Error::IoError { source: e })?;
map.record(
bt_pos,
span,
mapfile::SectorStatus::Finished,
)
.map_err(|e| Error::IoError { source: e })?;
// Plaintext via DecryptingSectorSource
// wrapping; same path the main read
// takes above.
send_or_abort(
&pipe,
PatchItem::Recovered {
pos: bt_pos,
buf: buf[..bt_bytes].to_vec(),
},
)?;
}
Err(_err) => {
blocks_read_failed += 1;
map.record(
bt_pos,
span,
mapfile::SectorStatus::Unreadable,
)
.map_err(|e| Error::IoError { source: e })?;
send_or_abort(
&pipe,
PatchItem::Unreadable {
pos: bt_pos,
len: span,
},
)?;
tracing::info!(
target: "freemkv::disc",
phase = "patch_backtrack_stop",
@@ -2591,16 +2689,9 @@ impl Disc {
"Retry succeeded after non-NOT_READY error"
);
if opts.decrypt {
crate::decrypt::decrypt_sectors(
&mut buf[..bytes],
&keys,
0,
)?;
}
// Plaintext via DecryptingSectorSource;
// same path the original read takes.
let write_start = std::time::Instant::now();
file.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
tracing::debug!(
target: "freemkv::disc",
phase = "patch_write_start",
@@ -2608,8 +2699,13 @@ impl Disc {
bytes,
"Starting ISO write"
);
file.write_all(&buf[..bytes])
.map_err(|e| Error::IoError { source: e })?;
send_or_abort(
&pipe,
PatchItem::Recovered {
pos,
buf: buf[..bytes].to_vec(),
},
)?;
let write_duration_ms = write_start.elapsed().as_millis();
tracing::info!(
target: "freemkv::disc",
@@ -2619,22 +2715,16 @@ impl Disc {
write_duration_ms,
"ISO write succeeded"
);
let mapfile_record_start = std::time::Instant::now();
map.record(pos, block_bytes, mapfile::SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
let mapfile_record_duration_ms =
mapfile_record_start.elapsed().as_millis();
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_record_ok",
pos,
block_bytes,
mapfile_record_duration_ms,
"Mapfile record written"
"Mapfile record dispatched"
);
// Stall guard after successful retry
let bytes_good_now = map.stats().bytes_good;
let bytes_good_now = read_shared(&shared).0.bytes_good;
if bytes_good_now > bytes_good_last {
stall_start = std::time::Instant::now();
bytes_good_last = bytes_good_now;
@@ -2654,8 +2744,13 @@ impl Disc {
}
// All retries exhausted - mark as Unreadable
map.record(pos, block_bytes, mapfile::SectorStatus::Unreadable)
.map_err(|e| Error::IoError { source: e })?;
send_or_abort(
&pipe,
PatchItem::Unreadable {
pos,
len: block_bytes,
},
)?;
damage_window.push(false);
if damage_window.len() > PASSN_DAMAGE_WINDOW {
@@ -2663,7 +2758,7 @@ impl Disc {
}
// Stall guard: check on failures too, not just successes
let bytes_good_now = map.stats().bytes_good;
let bytes_good_now = read_shared(&shared).0.bytes_good;
if bytes_good_now > bytes_good_last {
stall_start = std::time::Instant::now();
bytes_good_last = bytes_good_now;
@@ -2768,19 +2863,25 @@ impl Disc {
}
}
// Log mapfile record for Unreadable status
let mapfile_record_start = std::time::Instant::now();
map.record(pos, block_bytes, mapfile::SectorStatus::Unreadable)
.map_err(|e| Error::IoError { source: e })?;
let mapfile_record_duration_ms = mapfile_record_start.elapsed().as_millis();
// Redundant second Unreadable mark — preserved
// bit-for-bit from the pre-split loop (`record`
// is idempotent for same-status replacement of
// the same range). Routes through the consumer
// like every other state change.
send_or_abort(
&pipe,
PatchItem::Unreadable {
pos,
len: block_bytes,
},
)?;
tracing::info!(
target: "freemkv::disc",
phase = "patch_mapfile_record_unreadable",
pos,
block_bytes,
consecutive_failures,
mapfile_record_duration_ms,
"Mapfile record written as Unreadable"
"Mapfile record dispatched as Unreadable"
);
let pause_secs = if err.is_bridge_degradation() {
@@ -2894,7 +2995,7 @@ impl Disc {
work_done = work_done.saturating_add(block_bytes);
if let Some(reporter) = opts.progress {
let s = map.stats();
let (s, bad_ranges_now) = read_shared(&shared);
let kind = if block_sectors == 1 {
crate::progress::PassKind::Scrape {
reverse: opts.reverse,
@@ -2904,16 +3005,10 @@ impl Disc {
reverse: opts.reverse,
}
};
let bad_ranges = map.ranges_with(&[
mapfile::SectorStatus::NonTrimmed,
mapfile::SectorStatus::Unreadable,
mapfile::SectorStatus::NonScraped,
mapfile::SectorStatus::NonTried,
]);
let main_title_bad = self
.titles
.first()
.map(|t| bytes_bad_in_title(t, &bad_ranges))
.map(|t| bytes_bad_in_title(t, &bad_ranges_now))
.unwrap_or(0);
let main_title = self.titles.first();
let pp = crate::progress::PassProgress {
@@ -2937,26 +3032,13 @@ impl Disc {
}
}
if let Err(e) = file.sync_all() {
if is_regular {
tracing::warn!(
target: "freemkv::disc",
phase = "patch_sync_failed",
error = %e,
os_error = e.raw_os_error(),
error_kind = ?e.kind(),
"patch: sync_all failed"
);
return Err(Error::IoError { source: e });
}
tracing::debug!(
target: "freemkv::disc",
phase = "patch_sync_skipped",
error = %e,
"patch: sync_all failed for non-regular file; ignoring"
);
}
map.flush().map_err(|e| Error::IoError { source: e })?;
// Drain the consumer thread: drop tx, wait for `close` to run
// sync_all + mapfile.flush, then take the final stats from the
// sink's summary. `close` failing on a regular-file sync_all is
// surfaced here as `Error::IoError`, matching pre-split
// behaviour.
let summary = pipe.finish()?;
let stats = summary.stats;
// Log final ISO file size for write verification
if let Ok(metadata) = std::fs::metadata(path) {
@@ -2964,12 +3046,11 @@ impl Disc {
target: "freemkv::disc",
phase = "patch_iso_size_end",
iso_bytes = metadata.len(),
bytes_recovered = map.stats().bytes_good.saturating_sub(bytes_good_before),
bytes_recovered = stats.bytes_good.saturating_sub(bytes_good_before),
"ISO file size at patch end"
);
}
let stats = map.stats();
tracing::info!(
target: "freemkv::disc",
phase = "patch_done",
@@ -3463,4 +3544,47 @@ mod tests {
patch_result.err()
);
}
/// Synthetic regression test for the 0.18 SweepSink + Pipeline
/// migration. ~100 batches of clean reads (6000 sectors at the
/// default 60-sector single-pass batch size); verifies all bytes
/// land in the ISO and the consumer's final stats match the input.
/// The throughput regression check (vs 0.17.13) is a separate
/// manual / live-drive concern; here we only assert correctness.
#[test]
fn sweep_pipeline_full_good_100_batches() {
let tmp = tempfile::tempdir().unwrap();
let iso_path = tmp.path().join("test.iso");
// 6000 sectors / 60-sector default batch = exactly 100
// produce/consume cycles through the pipeline.
let sectors: u32 = 6000;
let mut reader = MockReader {
total_sectors: sectors,
bad_sectors: std::collections::HashSet::new(),
};
let disc = make_test_disc(sectors, "TPipeline100");
let opts = CopyOptions {
decrypt: false,
multipass: false,
progress: None,
halt: None,
};
let result = disc.copy(&mut reader, &iso_path, &opts);
let r = result.expect("100-batch clean sweep should succeed");
assert!(r.complete, "complete=true expected");
assert!(!r.halted, "halted=false expected");
assert_eq!(
r.bytes_good,
sectors as u64 * 2048,
"all sectors must be marked good after a 100% clean sweep"
);
assert_eq!(
r.bytes_pending, 0,
"no pending bytes expected after a clean sweep"
);
// The ISO file must end up the right size — the consumer
// wrote everything before fsync.
let meta = std::fs::metadata(&iso_path).unwrap();
assert_eq!(meta.len(), sectors as u64 * 2048);
}
}
+243
View File
@@ -0,0 +1,243 @@
//! Producer / consumer split for `Disc::patch`.
//!
//! Background: pre-0.18 patch ran strictly serial — single-sector
//! recovery read → seek + write recovered bytes → mapfile.record →
//! next iteration. The drive sat idle while the previous block's
//! recovered bytes were committed. On a damaged disc with many bad
//! sectors that adds up: per-sector write + mapfile.record costs a
//! handful of milliseconds each, which the drive could be using to
//! issue the next per-sector retry.
//!
//! This module decouples them. A consumer thread owns the
//! [`crate::io::WritebackFile`] (the ISO file) and the
//! [`super::mapfile::Mapfile`]. The producer thread (`Disc::patch`)
//! keeps the [`crate::sector::SectorReader`], the wedge / damage-window
//! state, the per-range watchdog, decrypt — so what enters the channel
//! is already-clean cleartext bytes (or an "Unreadable" terminal mark).
//!
//! Producer and consumer run concurrently; the channel uses
//! [`crate::io::pipeline::WRITE_THROUGH_DEPTH`] (=1) so back-pressure
//! kicks in immediately. We want the drive's per-sector retry budget
//! to stay in lockstep with the writer — sweep's `DEFAULT_PIPELINE_DEPTH`
//! (4) would let several sectors of recovered bytes queue up between
//! the producer's retry decisions and the writer, and patch's recovery
//! loop reads stats (`bytes_good`, range progress) inline to drive its
//! skip / wedge decisions. WRITE_THROUGH_DEPTH gives "read N+1 while
//! writing N", no further pipelining — exactly the model the producer
//! logic was written against.
//!
//! Correctness invariants preserved:
//! - Mapfile is single-writer (consumer-only). No locking on it.
//! - All recovery state (damage window, consecutive_failures, skip
//! escalation, range watchdog) stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorReader`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has received"
//! survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic. The threading primitive only overlaps the
//! *write* with the *next read*; the per-sector single-shot read
//! budget that the bridge wedge concern was originally about is
//! untouched.
//!
//! Per-range watchdog (`MAX_RANGE_SECS` / `RANGE_BUDGET_CAP_SECS`)
//! checks `bytes_good` for forward progress. With work in flight on
//! the consumer, the producer would otherwise see stale values; the
//! sink publishes a [`SharedPatchState`] snapshot after every record
//! so the producer's stall guards observe consumer side-effects with
//! at most one item of lag (which is fine — the watchdog uses minute-
//! scale budgets, not single-record latency).
use std::io::{Seek, SeekFrom, Write};
use std::sync::{Arc, Mutex};
use crate::error::{Error, Result};
use crate::io::pipeline::{Flow, Sink};
use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Item the producer hands to the patch consumer. One per per-sector
/// recovery decision.
pub(super) enum PatchItem {
/// Sector / small batch successfully recovered (and decrypted on the
/// producer side if `opts.decrypt` was set). Consumer seeks to
/// `pos`, writes `buf`, records the range as `Finished`.
Recovered { pos: u64, buf: Vec<u8> },
/// Producer exhausted retries on `[pos, pos+len)`. Consumer records
/// the range as `Unreadable`. No file write — the existing zero-fill
/// from sweep is preserved in place.
Unreadable { pos: u64, len: u64 },
/// Producer hit the per-range skip limit and is leaving the
/// remaining bytes as `NonTrimmed` for a future pass. CRITICAL:
/// this is not the same as `Unreadable` — sectors we never tried
/// stay hopeful. (See the comment at the skip-limit branch in
/// `Disc::patch`: ~36% of patch-marked Unreadable sectors are
/// actually readable on a later pass.) No file write.
NonTrimmed { pos: u64, len: u64 },
}
/// Mapfile snapshot the sink republishes after every record so the
/// producer can drive its stall / progress logic without holding the
/// mapfile lock for long. `bad_ranges` mirrors what
/// `Mapfile::ranges_with(&[NonTrimmed, Unreadable, NonScraped, NonTried])`
/// would return — same set the pre-split patch loop computed inline
/// for the progress callback.
pub(super) struct SharedPatchState {
pub stats: MapStats,
pub bad_ranges: Vec<(u64, u64)>,
}
impl SharedPatchState {
fn from_map(map: &Mapfile) -> Self {
Self {
stats: map.stats(),
bad_ranges: map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]),
}
}
}
/// Final summary returned by [`Sink::close`] when the consumer drains
/// cleanly. Mirrors what the pre-split patch loop computed at the end
/// of the function — final mapfile stats plus whether `sync_all`
/// failed on a regular file (the only kind of fsync error patch ever
/// surfaced; `/dev/null` and pipes always fail `sync_all`, that's not
/// a real error).
pub(super) struct PatchSummary {
pub stats: MapStats,
}
/// Consumer-side of the patch pipeline. Owns the ISO writeback file
/// and the mapfile; publishes a shared snapshot after every record so
/// the producer can read `bytes_good` for stall detection and
/// progress reporting.
pub(super) struct PatchSink {
file: crate::io::WritebackFile,
map: Mapfile,
/// Whether the output is a regular file (so a `sync_all` failure
/// is real). `/dev/null` etc. always fail `sync_all`; ignore those.
is_regular: bool,
/// Snapshot the producer reads. Updated after every successful
/// `record()` call. `Mutex` rather than separate atomics because
/// the producer wants stats + bad_ranges as a coherent pair.
shared: Arc<Mutex<SharedPatchState>>,
}
impl PatchSink {
/// Open `path` as a [`crate::io::WritebackFile`] and pair it with
/// `map` for the consumer. The producer holds onto the returned
/// `Arc<Mutex<SharedPatchState>>` so it can poll mapfile state
/// while the consumer is mutating it.
pub(super) fn new(
path: &std::path::Path,
map: Mapfile,
is_regular: bool,
) -> Result<(Self, Arc<Mutex<SharedPatchState>>)> {
let file =
crate::io::WritebackFile::open(path).map_err(|e| Error::IoError { source: e })?;
let shared = Arc::new(Mutex::new(SharedPatchState::from_map(&map)));
let shared_clone = shared.clone();
Ok((
Self {
file,
map,
is_regular,
shared,
},
shared_clone,
))
}
fn republish(&self) {
// Best-effort lock — only the producer reads, only the consumer
// writes; contention is single-acquire so the lock is never
// poisoned in practice. If it ever did get poisoned we'd want
// the underlying error surfaced rather than silently swallowed,
// so we propagate the poison panic. (Same posture as
// `sweep_pipeline.rs` — it never recovers from a poisoned
// mutex either.)
let mut guard = self
.shared
.lock()
.expect("PatchSink shared state mutex poisoned");
*guard = SharedPatchState::from_map(&self.map);
}
}
impl Sink<PatchItem> for PatchSink {
type Output = PatchSummary;
fn apply(&mut self, item: PatchItem) -> std::result::Result<Flow, Error> {
match item {
PatchItem::Recovered { pos, buf } => {
let len = buf.len() as u64;
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
PatchItem::Unreadable { pos, len } => {
self.map
.record(pos, len, SectorStatus::Unreadable)
.map_err(|e| Error::IoError { source: e })?;
}
PatchItem::NonTrimmed { pos, len } => {
self.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
}
self.republish();
Ok(Flow::Continue)
}
fn close(mut self) -> std::result::Result<Self::Output, Error> {
// Drain in-flight writeback then issue a full fsync. A failure
// here matters only on regular files — pipes / `/dev/null` etc.
// always fail `sync_all`.
if let Err(e) = self.file.sync_all() {
if self.is_regular {
tracing::warn!(
target: "freemkv::disc",
phase = "patch_sync_failed",
error = %e,
os_error = e.raw_os_error(),
error_kind = ?e.kind(),
"patch: sync_all failed"
);
return Err(Error::IoError { source: e });
}
tracing::debug!(
target: "freemkv::disc",
phase = "patch_sync_skipped",
error = %e,
"patch: sync_all failed for non-regular file; ignoring"
);
}
self.map.flush().map_err(|e| Error::IoError { source: e })?;
// Final republish so anyone reading the shared snapshot after
// `Pipeline::finish` sees the post-flush state. (The producer
// already has its own copy of the final `MapStats` in the
// returned `PatchSummary`, but the snapshot is part of the
// public-ish contract of the consumer: it stays current
// through close.)
self.republish();
Ok(PatchSummary {
stats: self.map.stats(),
})
}
}
+1 -1
View File
@@ -369,6 +369,6 @@ mod tests {
ctx.on_success();
assert_eq!(ctx.consecutive_good, 1);
assert_eq!(ctx.consecutive_failures, 0);
assert_eq!(*ctx.damage_window.last().unwrap(), true);
assert!(*ctx.damage_window.last().unwrap());
}
}
+239
View File
@@ -0,0 +1,239 @@
//! `Disc::sweep`'s consumer-side `Sink<WorkItem>`.
//!
//! Background: the original sweep loop runs strictly serialised —
//! SCSI read → decrypt → seek + write → mapfile.record → next iter.
//! On a healthy disc the SCSI read costs ~5-12 ms per 64 KB batch and
//! the post-read work (decrypt 1-3 ms + file write + mapfile fsync
//! 5-15 ms) adds another batch's worth of latency. The drive idles
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! 0.17.11 introduced a bespoke producer/consumer split (the now-
//! removed `disc/sweep_pipeline.rs`) to overlap the two stages. 0.18
//! collapses that split — together with the analogous splits patch
//! and mux need — onto the generic [`crate::io::Pipeline`] +
//! [`crate::io::Sink`] primitive. This module is the sweep-specific
//! `Sink` impl; the producer-side state machine (read_error context,
//! decrypt, set_speed, halt) stays in `Disc::sweep` in `disc/mod.rs`.
//!
//! Correctness invariants preserved (same as 0.17.11):
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorReader`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record), so the on-disk
//! invariant "mapfile only marks Finished what the file has
//! received" survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, sync_channel};
use crate::error::Error;
use crate::io::{Flow, Sink};
use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 65 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
/// preserved.
pub(super) enum WorkItem {
/// Successful batch read. Producer has already decrypted `buf` if
/// `opts.decrypt` was set. Consumer writes `buf` at `pos` and
/// records the range as `Finished`.
Good { pos: u64, buf: Vec<u8> },
/// Bisect inner-loop good single sector (already decrypted by the
/// producer). 2048 bytes.
BisectGood { pos: u64, buf: Box<[u8; 2048]> },
/// Bisect inner-loop bad single sector. Consumer writes 2048
/// zeros at `pos` and records the sector as `NonTrimmed`.
BisectBad { pos: u64 },
/// Whole-batch zero-fill (failed batch on `SkipBlock`, or the
/// failed batch portion of `JumpAhead`). Consumer streams zeros
/// across `[pos, pos+len)` and records the range as `NonTrimmed`.
SkipFill { pos: u64, len: u64 },
/// Gap fill following a `JumpAhead`. Same effect as `SkipFill`;
/// distinguished only so future logging / instrumentation can
/// tell them apart without parsing a flag.
GapFill { pos: u64, len: u64 },
/// Producer wants the latest mapfile stats for the progress
/// callback. Consumer responds on `prog_tx` with a fresh
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
/// drained the previous snapshot, the new one is silently
/// dropped — the producer's local cache stays current enough.
StatsRequest,
}
/// Snapshot the consumer sends back to the producer for the progress
/// callback.
pub(super) struct ProgressSnapshot {
pub stats: MapStats,
pub bad_ranges: Vec<(u64, u64)>,
}
/// Final summary returned by the consumer thread on shutdown — what
/// `SweepSink::close` produces, surfaced to the producer via
/// `Pipeline::finish`.
pub(super) struct ConsumerSummary {
pub stats: MapStats,
}
/// Drain any pending progress snapshots from the consumer. Returns
/// the most recent one, if any. The producer caches it and uses it
/// for subsequent progress callbacks until a fresh one arrives.
pub(super) fn try_recv_progress(rx: &Receiver<ProgressSnapshot>) -> Option<ProgressSnapshot> {
let mut latest = None;
while let Ok(snap) = rx.try_recv() {
latest = Some(snap);
}
latest
}
/// `Sink<WorkItem>` for sweep. Owns the writeback file + mapfile +
/// progress back-channel. `apply` carries the file-write +
/// mapfile.record per item; `close` drains the writeback pipeline,
/// fsyncs the ISO, and flushes the mapfile.
pub(super) struct SweepSink {
file: crate::io::WritebackFile,
map: Mapfile,
/// `sync_all`-on-failure-is-an-error iff the output is a regular
/// file. `/dev/null` and pipes always fail `sync_all`; that's not
/// a real error.
is_regular: bool,
/// Back-channel for `StatsRequest` responses. The producer caches
/// the latest snapshot and uses it for the progress callback;
/// dropped sends on a full channel are by design.
prog_tx: SyncSender<ProgressSnapshot>,
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. Held
/// in the sink so each apply call doesn't reallocate.
zero: Box<[u8; ZERO_CHUNK]>,
}
impl SweepSink {
/// Construct a new `SweepSink` plus the matching progress
/// receiver. Channel depth on the back-channel is `1` — the
/// producer's cache is the source of truth between snapshots.
pub(super) fn new(
file: crate::io::WritebackFile,
map: Mapfile,
is_regular: bool,
) -> (Self, Receiver<ProgressSnapshot>) {
let (prog_tx, prog_rx) = sync_channel::<ProgressSnapshot>(1);
let sink = SweepSink {
file,
map,
is_regular,
prog_tx,
zero: Box::new([0u8; ZERO_CHUNK]),
};
(sink, prog_rx)
}
}
impl Sink<WorkItem> for SweepSink {
type Output = ConsumerSummary;
fn apply(&mut self, item: WorkItem) -> Result<Flow, Error> {
match item {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectGood { pos, buf } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&buf[..])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectBad { pos } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
self.file
.write_all(&self.zero[..2048])
.map_err(|e| Error::IoError { source: e })?;
self.map
.record(pos, 2048, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
self.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
// Subsequent writes are sequential; `WritebackFile`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(self.zero.len() as u64) as usize;
self.file
.write_all(&self.zero[..chunk])
.map_err(|e| Error::IoError { source: e })?;
filled += chunk as u64;
}
self.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::StatsRequest => {
let stats = self.map.stats();
let bad_ranges = self.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
let _ = self
.prog_tx
.try_send(ProgressSnapshot { stats, bad_ranges });
}
}
Ok(Flow::Continue)
}
fn close(mut self) -> Result<Self::Output, Error> {
// Drain the writeback pipeline + fsync the ISO, then persist
// any pending mapfile state. Same finalisation order as the
// pre-Pipeline consumer loop.
if let Err(e) = self.file.sync_all() {
if self.is_regular {
return Err(Error::IoError { source: e });
}
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
self.map.flush().map_err(|e| Error::IoError { source: e })?;
Ok(ConsumerSummary {
stats: self.map.stats(),
})
}
}
-313
View File
@@ -1,313 +0,0 @@
//! Producer / consumer split for `Disc::sweep`.
//!
//! Background: the original sweep loop runs strictly serialised —
//! SCSI read → decrypt → seek + write → mapfile.record → next iter.
//! On a healthy disc the SCSI read costs ~5-12 ms per 64 KB batch and
//! the post-read work (decrypt 1-3 ms + file write + mapfile fsync
//! 5-15 ms) adds another batch's worth of latency. The drive idles
//! during the post-read work; throughput tops out at the *sum* of
//! both costs.
//!
//! This module decouples them. A consumer thread owns the
//! [`crate::io::Writer`] (the ISO file) and the
//! [`super::mapfile::Mapfile`]. The producer thread (the caller of
//! `Disc::sweep`) keeps the [`crate::sector::SectorReader`], the
//! [`super::read_error`] state machine, and decrypt — so what enters
//! the channel is already-clean cleartext bytes, matching the "disc
//! hands plaintext to its consumers" semantic that
//! [`crate::mux::DiscStream`] already follows. Producer and consumer
//! run concurrently; with a healthy disc the drive can read the next
//! batch while the previous one is being written and recorded.
//!
//! Correctness invariants preserved:
//! - Mapfile is single-writer (consumer-only). No locking.
//! - All `read_error::ReadCtx` state stays on the producer thread.
//! - `set_speed` calls happen on the producer thread (same thread that
//! owns the `SectorReader`). No new SCSI concurrency.
//! - Per-iteration ordering of file-write → mapfile-record is kept
//! intact in the consumer (write before record, same as today), so
//! the on-disk invariant "mapfile only marks Finished what the file
//! has received" survives a crash mid-pass.
//! - The BU40N+Initio bridge wedge concern is unchanged: only one
//! SCSI command in flight at a time, error-path timing identical,
//! no new retry logic.
use std::io::{Seek, SeekFrom, Write};
use std::sync::mpsc::{Receiver, SyncSender, TrySendError, sync_channel};
use std::thread::{self, JoinHandle};
use crate::error::{Error, Result};
use super::mapfile::{MapStats, Mapfile, SectorStatus};
/// Channel depth for in-flight work items. 4 is enough to absorb a
/// mapfile-flush burst on the consumer without growing memory
/// unboundedly. Producer back-pressure is the natural rate limiter:
/// `SyncSender::send` blocks when the channel is full.
const CHANNEL_DEPTH: usize = 4;
/// Reusable zero buffer for SkipFill / GapFill / BisectBad. 64 KB
/// matches the existing zero_gap chunk size used by the pre-split
/// sweep loop.
const ZERO_CHUNK: usize = 65 * 1024;
/// Producer → Consumer messages. The consumer applies these in FIFO
/// order; ordering of file writes and mapfile records across items is
/// preserved.
pub(super) enum WorkItem {
/// Successful batch read. Producer has already decrypted `buf` if
/// `opts.decrypt` was set. Consumer writes `buf` at `pos` and
/// records the range as `Finished`.
Good { pos: u64, buf: Vec<u8> },
/// Bisect inner-loop good single sector (already decrypted by the
/// producer). 2048 bytes.
BisectGood { pos: u64, buf: Box<[u8; 2048]> },
/// Bisect inner-loop bad single sector. Consumer writes 2048
/// zeros at `pos` and records the sector as `NonTrimmed`.
BisectBad { pos: u64 },
/// Whole-batch zero-fill (failed batch on `SkipBlock`, or the
/// failed batch portion of `JumpAhead`). Consumer streams zeros
/// across `[pos, pos+len)` and records the range as `NonTrimmed`.
SkipFill { pos: u64, len: u64 },
/// Gap fill following a `JumpAhead`. Same effect as `SkipFill`;
/// distinguished only so future logging / instrumentation can
/// tell them apart without parsing a flag.
GapFill { pos: u64, len: u64 },
/// Producer wants the latest mapfile stats for the progress
/// callback. Consumer responds on `prog_tx` with a fresh
/// [`ProgressSnapshot`]. Best-effort: if the producer hasn't
/// drained the previous snapshot, the new one is silently
/// dropped — the producer's local cache stays current enough.
StatsRequest,
/// Producer is done. Consumer drains, runs `sync_all` on the
/// file, and exits. Final stats are returned via the
/// `JoinHandle<ConsumerSummary>` from `spawn_consumer`.
Finish,
}
/// Snapshot the consumer sends back to the producer for the progress
/// callback.
pub(super) struct ProgressSnapshot {
pub stats: MapStats,
pub bad_ranges: Vec<(u64, u64)>,
}
/// Final summary returned by the consumer thread on shutdown. The
/// producer reads it via the `JoinHandle` returned from
/// [`spawn_consumer`].
pub(super) struct ConsumerSummary {
pub stats: MapStats,
/// First mapfile/write error the consumer hit, if any. The
/// producer treats this as fatal on the way back up.
pub error: Option<Error>,
}
/// Owned bundle the consumer thread takes ownership of. Decrypt
/// happens on the producer side before send, so the consumer never
/// sees keys.
pub(super) struct ConsumerInputs {
pub file: crate::io::Writer,
pub map: Mapfile,
/// `sync_all`-on-failure-is-an-error iff the output is a regular
/// file. `/dev/null` and pipes always fail `sync_all`; that's not
/// a real error.
pub is_regular: bool,
}
/// Spawn the consumer thread. The producer keeps the work-tx and
/// prog-rx; the join handle yields the final summary on `Finish` (or
/// on channel close).
pub(super) fn spawn_consumer(
inputs: ConsumerInputs,
) -> (
SyncSender<WorkItem>,
Receiver<ProgressSnapshot>,
JoinHandle<ConsumerSummary>,
) {
let (work_tx, work_rx) = sync_channel::<WorkItem>(CHANNEL_DEPTH);
let (prog_tx, prog_rx) = sync_channel::<ProgressSnapshot>(1);
let handle = thread::Builder::new()
.name("freemkv-sweep-consumer".into())
.spawn(move || consumer_loop(inputs, work_rx, prog_tx))
.expect("spawning sweep consumer thread should not fail");
(work_tx, prog_rx, handle)
}
/// Send a work item, translating a `SendError` (consumer thread died
/// / panicked) into a useful library error so the caller can
/// propagate cleanly.
pub(super) fn send_or_abort(tx: &SyncSender<WorkItem>, item: WorkItem) -> Result<()> {
tx.send(item).map_err(|_| Error::IoError {
source: std::io::Error::other("sweep consumer terminated unexpectedly"),
})
}
/// Best-effort `StatsRequest` send. If the channel is full, skip —
/// the producer's cached snapshot is fine for one more iteration.
pub(super) fn try_request_stats(tx: &SyncSender<WorkItem>) {
if let Err(TrySendError::Full(_)) = tx.try_send(WorkItem::StatsRequest) {
// expected when the consumer is busy; cached snapshot is
// still recent enough.
}
}
/// Drain any pending progress snapshots from the consumer. Returns
/// the most recent one, if any. The producer caches it and uses it
/// for subsequent progress callbacks until a fresh one arrives.
pub(super) fn try_recv_progress(rx: &Receiver<ProgressSnapshot>) -> Option<ProgressSnapshot> {
let mut latest = None;
while let Ok(snap) = rx.try_recv() {
latest = Some(snap);
}
latest
}
fn consumer_loop(
mut inputs: ConsumerInputs,
work_rx: Receiver<WorkItem>,
prog_tx: SyncSender<ProgressSnapshot>,
) -> ConsumerSummary {
let zero = [0u8; ZERO_CHUNK];
let mut first_error: Option<Error> = None;
// Channel closed without a Finish == treat as Finish (producer
// dropped tx without explicit teardown — should not happen in
// normal operation but be defensive).
while let Ok(item) = work_rx.recv() {
// Once we have an error, drain remaining items without
// applying side-effects so the producer never blocks on a
// dead consumer. Loop until Finish or channel close.
if first_error.is_some() {
if matches!(item, WorkItem::Finish) {
break;
}
continue;
}
match apply_item(&mut inputs, item, &zero, &prog_tx) {
Ok(true) => {}
Ok(false) => break, // Finish received
Err(e) => first_error = Some(e),
}
}
// Final flush — drain the writeback pipeline + fsync the ISO,
// then persist any pending mapfile state.
if first_error.is_none() {
if let Err(e) = inputs.file.sync_all() {
if inputs.is_regular {
first_error = Some(Error::IoError { source: e });
}
// Non-regular outputs (/dev/null, pipes) always fail
// sync_all; that's not a real error.
}
if let Err(e) = inputs.map.flush() {
first_error = Some(Error::IoError { source: e });
}
}
ConsumerSummary {
stats: inputs.map.stats(),
error: first_error,
}
}
/// Apply a single `WorkItem`. Returns `Ok(true)` to continue, `Ok(false)`
/// to break the consumer loop on `Finish`, `Err(_)` on first failure
/// (caller captures and continues draining).
fn apply_item(
inputs: &mut ConsumerInputs,
item: WorkItem,
zero: &[u8; ZERO_CHUNK],
prog_tx: &SyncSender<ProgressSnapshot>,
) -> Result<bool> {
match item {
WorkItem::Good { pos, buf } => {
// Decrypt is on the producer; consumer assumes plaintext.
let len = buf.len() as u64;
inputs
.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
inputs
.file
.write_all(&buf)
.map_err(|e| Error::IoError { source: e })?;
inputs
.map
.record(pos, len, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectGood { pos, buf } => {
inputs
.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
inputs
.file
.write_all(&buf[..])
.map_err(|e| Error::IoError { source: e })?;
inputs
.map
.record(pos, 2048, SectorStatus::Finished)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::BisectBad { pos } => {
inputs
.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
inputs
.file
.write_all(&zero[..2048])
.map_err(|e| Error::IoError { source: e })?;
inputs
.map
.record(pos, 2048, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::SkipFill { pos, len } | WorkItem::GapFill { pos, len } => {
inputs
.file
.seek(SeekFrom::Start(pos))
.map_err(|e| Error::IoError { source: e })?;
// Subsequent writes are sequential; `crate::io::Writer`'s
// seek-elision keeps them on the writeback pipeline path.
let mut filled = 0u64;
while filled < len {
let chunk = (len - filled).min(zero.len() as u64) as usize;
inputs
.file
.write_all(&zero[..chunk])
.map_err(|e| Error::IoError { source: e })?;
filled += chunk as u64;
}
inputs
.map
.record(pos, len, SectorStatus::NonTrimmed)
.map_err(|e| Error::IoError { source: e })?;
}
WorkItem::StatsRequest => {
let stats = inputs.map.stats();
let bad_ranges = inputs.map.ranges_with(&[
SectorStatus::NonTrimmed,
SectorStatus::Unreadable,
SectorStatus::NonScraped,
SectorStatus::NonTried,
]);
// Best-effort: drop on backpressure; producer's cache
// stays current enough.
let _ = prog_tx.try_send(ProgressSnapshot { stats, bad_ranges });
}
WorkItem::Finish => return Ok(false),
}
Ok(true)
}
+166
View File
@@ -0,0 +1,166 @@
//! One-bit cooperative cancellation flag.
//!
//! `Halt` is a clonable token wrapping `Arc<AtomicBool>`. Pass clones into
//! every long-running loop; the loop polls `is_cancelled()` and bails out
//! cleanly. Calling `cancel()` from any clone flips the shared flag, and
//! every other clone observes it on its next poll.
//!
//! Why: `Ordering::Relaxed` is sufficient on both load and store because
//! this flag is purely advisory — no other memory operations piggyback on
//! it for happens-before ordering. Callers that need to publish data
//! across threads do so via channels or other synchronization, not via
//! this bit.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
/// Clonable, infallible cooperative-cancellation token.
///
/// Clones share the same underlying flag. `cancel()` is one-way; there is
/// no `reset()` by design — construct a fresh `Halt` for a fresh
/// operation.
///
/// Construct with [`Halt::new`] (or [`Halt::default`]). The `Default`
/// impl forwards to `new()` — both produce a fresh, uncancelled token.
/// The pair exists because clippy's `new_without_default` lint requires
/// `Default` whenever a public `new()` is present, even when the two
/// would do exactly the same thing.
#[derive(Clone, Debug)]
pub struct Halt(Arc<AtomicBool>);
impl Halt {
/// Construct a fresh, uncancelled token.
pub fn new() -> Self {
Self(Arc::new(AtomicBool::new(false)))
}
/// Wrap an existing `Arc<AtomicBool>` as a `Halt`. Useful as a
/// bridge during the 0.18 deprecation window: callers that already
/// hold an `Arc<AtomicBool>` (e.g. `Drive::halt_flag()`, the
/// deprecated `DiscStream::set_halt`) can adopt the new token API
/// without changing the underlying flag.
///
/// Cancelling either side flips the same bit — the wrapping `Halt`
/// and the original `Arc` are two views over one shared flag.
pub fn from_arc(flag: Arc<AtomicBool>) -> Self {
Self(flag)
}
/// Borrow the underlying `Arc<AtomicBool>`. Used at boundaries with
/// pre-`Halt` APIs that still take an `Arc<AtomicBool>` directly
/// (`CopyOptions::halt`, the deprecated `DiscStream::set_halt`).
/// Round 3 deletes those boundaries and this accessor with them.
pub fn as_arc(&self) -> &Arc<AtomicBool> {
&self.0
}
/// Flip the shared flag to cancelled. Idempotent.
pub fn cancel(&self) {
self.0.store(true, Ordering::Relaxed);
}
/// Read the shared flag.
pub fn is_cancelled(&self) -> bool {
self.0.load(Ordering::Relaxed)
}
}
impl Default for Halt {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fresh_is_not_cancelled() {
let h = Halt::new();
assert!(!h.is_cancelled());
}
#[test]
fn cancel_flips_state() {
let h = Halt::new();
assert!(!h.is_cancelled());
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn cancel_is_idempotent() {
let h = Halt::new();
h.cancel();
h.cancel();
assert!(h.is_cancelled());
}
#[test]
fn clone_shares_state() {
let original = Halt::new();
let cloned = original.clone();
assert!(!original.is_cancelled());
assert!(!cloned.is_cancelled());
// Cancel via the clone; the original observes it.
cloned.cancel();
assert!(original.is_cancelled());
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_reverse_direction() {
let original = Halt::new();
let cloned = original.clone();
// Cancel via the original; the clone observes it.
original.cancel();
assert!(cloned.is_cancelled());
}
#[test]
fn clone_shares_state_across_threads() {
let h = Halt::new();
let h2 = h.clone();
let handle = std::thread::spawn(move || {
h2.cancel();
});
handle.join().unwrap();
assert!(h.is_cancelled());
}
#[test]
fn from_arc_shares_state() {
// The 0.18 deprecation-window bridge: a Halt built from an
// existing Arc<AtomicBool> must be a *view* over the same bit,
// not a fresh copy. Cancelling either side flips both.
let arc = Arc::new(AtomicBool::new(false));
let halt = Halt::from_arc(arc.clone());
assert!(!halt.is_cancelled());
assert!(!arc.load(Ordering::Relaxed));
// Cancel via the wrapping Halt; the original Arc observes it.
halt.cancel();
assert!(arc.load(Ordering::Relaxed));
// Conversely: flip the Arc directly; the Halt view observes it.
let arc2 = Arc::new(AtomicBool::new(false));
let halt2 = Halt::from_arc(arc2.clone());
arc2.store(true, Ordering::Relaxed);
assert!(halt2.is_cancelled());
}
#[test]
fn as_arc_returns_backing_flag() {
// `as_arc()` must hand back the *same* Arc, not a clone of a
// different bit. Verified by writing through the borrowed Arc
// and observing through the Halt.
let halt = Halt::new();
let arc = halt.as_arc().clone();
assert!(!halt.is_cancelled());
arc.store(true, Ordering::Relaxed);
assert!(halt.is_cancelled());
}
}
+26 -11
View File
@@ -1,16 +1,31 @@
//! File I/O helpers that bound kernel cache pressure on big writes.
//!
//! `Writer` is a drop-in wrapper around `std::fs::File` for any call
//! site that performs large sequential writes (sweep, mux, etc.). It
//! implements `Write` and `Seek` so existing code paths can swap
//! `File` for `Writer` with no body changes. Internally it drives a
//! `WritebackPipeline` that, on Linux, drains dirty pages continuously
//! at 32 MB granularity to avoid the kernel's accumulate-then-burst
//! flush behaviour. macOS and Windows use a no-op pipeline — their
//! default cache policies have not been shown to exhibit the same
//! pathology for this access pattern.
//! `WritebackFile` is a drop-in wrapper around `std::fs::File` for any
//! call site that performs large sequential writes (sweep, patch, mux,
//! etc.). It implements `Write` and `Seek` so existing code paths can
//! swap `File` for `WritebackFile` with no body changes. Internally it
//! drives a `WritebackPipeline` that, on Linux, drains dirty pages
//! continuously at 32 MB granularity to avoid the kernel's
//! accumulate-then-burst flush behaviour. macOS and Windows use a
//! no-op pipeline — their default cache policies have not been shown
//! to exhibit the same pathology for this access pattern.
//!
//! `Pipeline` + `Sink` (0.18) is the generic producer/consumer primitive
//! used by sweep, patch, and mux to overlap reads with writes via a
//! bounded channel + dedicated consumer thread.
mod writeback;
mod writer;
mod writeback_file;
pub(crate) use writer::Writer;
pub mod pipeline;
pub(crate) use writeback_file::WritebackFile;
// Re-exports for the 0.18 redesign. Sweep + patch are both wired up
// (disc/sweep.rs, disc/patch.rs); mux migrates separately in autorip.
// `WRITE_THROUGH_DEPTH` is patch-specific and has no other in-tree
// caller — the targeted `#[allow]` keeps the re-export visible without
// dragging the rest of the module under `dead_code`.
#[allow(unused_imports)]
pub use pipeline::WRITE_THROUGH_DEPTH;
pub use pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink};
+496
View File
@@ -0,0 +1,496 @@
//! Generic bounded producer/consumer pipeline.
//!
//! `Pipeline<I, R>` spawns a single consumer thread, hands it items
//! through a bounded `mpsc::sync_channel`, and joins it on `finish()`.
//! The consumer's behaviour is supplied by a [`Sink`] implementation:
//! `apply` is called once per item, `close` is called once at the end.
//!
//! Three call sites in libfreemkv want a producer/consumer split —
//! sweep (migrated to `disc/sweep.rs::SweepSink`), patch, and mux.
//! 0.18 collapses all three onto this primitive; sweep is in,
//! patch and mux migrate in later 0.18 slices.
//! See `(internal)/memory/0_18_redesign.md` for the full picture.
//!
//! ## Cancellation and error semantics
//!
//! - Producer dropping the channel (via `Pipeline::finish` dropping
//! `tx`) signals end-of-stream; consumer flushes via `close()` and
//! returns its `Output`.
//! - Consumer returning [`Flow::Stop`] also calls `close()` and
//! returns its `Output`. `send()` from the producer will then either
//! succeed (if the item already fit in the channel buffer) or fail
//! with `Err(item)` once the consumer has dropped its receiver.
//! - Consumer returning `Err` from `apply` skips `close()` entirely;
//! the consumer keeps draining the channel so the producer never
//! blocks on a dead receiver, and the first error is propagated as
//! the `JoinHandle` result.
//! - Consumer panic is converted into
//! `Error::IoError { source: io::Error::other(...) }`.
//!
//! ## Dead-code suppression
//!
//! Both `Disc::sweep` (`DEFAULT_PIPELINE_DEPTH`, `spawn_named`,
//! `try_send`) and `Disc::patch` (`WRITE_THROUGH_DEPTH`, `Flow::Stop`)
//! are now in-tree callers (0.18 round 2). The targeted `#[allow]`
//! markers below cover the corners of the API not exercised by either
//! call site (e.g. `Pipeline::spawn` without a name; `Flow::Stop`
//! return path from sweep's sink — sweep always consumes the producer's
//! full work-list). Drop the allows once mux migrates.
use std::io;
use std::sync::mpsc::{SyncSender, sync_channel};
use std::thread::{self, JoinHandle};
use crate::error::Error;
/// Default channel depth for callers without a specific reason to
/// pick another value.
///
/// Empirically tuned for sweep and mux — both want enough slack that
/// short consumer stalls don't immediately back up onto the producer,
/// but not so much that a producer outpacing the consumer accumulates
/// arbitrary buffered work. `4` matches the depth sweep has used
/// since 0.17.11 (originally in `disc/sweep_pipeline.rs`, now in
/// `disc/sweep.rs::SweepSink`). Patch should usually use
/// [`WRITE_THROUGH_DEPTH`] (`1`) instead — write-through gives clean
/// back-pressure between every read attempt and the matching write,
/// which matters when the consumer is updating the mapfile in lockstep.
pub const DEFAULT_PIPELINE_DEPTH: usize = 4;
/// Channel depth for write-through pipelines. Each `send` fully
/// drains before the next can enqueue. Use this when the producer
/// must observe consumer side-effects (e.g. mapfile state) before
/// emitting the next item.
#[allow(dead_code)]
pub const WRITE_THROUGH_DEPTH: usize = 1;
/// Outcome of [`Sink::apply`]: either keep feeding items
/// ([`Flow::Continue`]), or stop the pipeline early and run `close()`
/// ([`Flow::Stop`]).
///
/// `Stop` has no in-tree caller in this slice — sweep never returns
/// it (it always processes the producer's full work-list before the
/// channel is dropped). Patch and mux are the intended consumers and
/// migrate in later 0.18 slices. The variant ships now so the contract
/// is fixed; the targeted `#[allow]` is removed when patch lands.
pub enum Flow {
Continue,
#[allow(dead_code)]
Stop,
}
/// Consumer-side of a [`Pipeline`]. The pipeline owns one of these on
/// its consumer thread and calls `apply` once per received item, then
/// `close` once at end-of-stream.
pub trait Sink<I>: Send + 'static {
/// Type returned from `close()` and surfaced via
/// [`Pipeline::finish`].
type Output: Send + 'static;
/// Apply one item. Returning [`Flow::Continue`] keeps the
/// pipeline running; [`Flow::Stop`] ends it cleanly (still calls
/// `close()`). An error short-circuits: `close()` is *not* called
/// and the error is what `finish()` will return, but the consumer
/// keeps draining the channel so the producer never blocks on a
/// dead receiver.
fn apply(&mut self, item: I) -> Result<Flow, Error>;
/// Called once at end-of-stream — either because the producer
/// dropped `tx` or because `apply` returned [`Flow::Stop`]. Use
/// this to flush, fsync, finalise. Skipped if any prior `apply`
/// returned `Err`.
fn close(self) -> Result<Self::Output, Error>;
}
/// Bounded producer/consumer pipeline. Holds the producer-side
/// channel and the consumer thread's join handle.
pub struct Pipeline<I: Send + 'static, R: Send + 'static> {
tx: SyncSender<I>,
handle: JoinHandle<Result<R, Error>>,
}
impl<I: Send + 'static, R: Send + 'static> Pipeline<I, R> {
/// Spawn the consumer thread with the given channel depth and
/// [`Sink`].
///
/// The thread is named `freemkv-pipeline-consumer` so it shows up
/// distinctly in stack traces and `top -H`. Callers that want a
/// more specific name (e.g. `freemkv-sweep-consumer`) should use
/// [`Pipeline::spawn_named`] instead. Returns an `Error::IoError`
/// if the OS refuses the thread spawn (resource exhaustion);
/// callers already operate in fallible context, so this is
/// propagated rather than panicked.
///
/// Sweep uses [`Pipeline::spawn_named`] directly so the consumer
/// thread shows up as `freemkv-sweep-consumer`; this function has
/// no in-tree caller yet. Patch and mux migrate in later 0.18
/// slices. The targeted `#[allow]` is removed when one of them
/// lands on the default name.
#[allow(dead_code)]
pub fn spawn<S: Sink<I, Output = R>>(depth: usize, sink: S) -> Result<Self, Error> {
Self::spawn_named("freemkv-pipeline-consumer", depth, sink)
}
/// Like [`Pipeline::spawn`] but lets the caller supply the
/// consumer thread's name. Useful when several pipelines run in
/// the same process and stack traces / `top -H` need to tell them
/// apart (e.g. `freemkv-sweep-consumer`, `freemkv-mux-consumer`).
pub fn spawn_named<S: Sink<I, Output = R>>(
name: &str,
depth: usize,
sink: S,
) -> Result<Self, Error> {
let (tx, rx) = sync_channel::<I>(depth);
let handle = thread::Builder::new()
.name(name.into())
.spawn(move || -> Result<R, Error> {
let mut sink = sink;
let mut first_err: Option<Error> = None;
let mut stopped = false;
while let Ok(item) = rx.recv() {
if first_err.is_some() || stopped {
// Drain remaining items so the producer never
// blocks on a dead receiver. `apply` is not
// called once we've decided to stop.
continue;
}
match sink.apply(item) {
Ok(Flow::Continue) => {}
Ok(Flow::Stop) => {
stopped = true;
}
Err(e) => {
first_err = Some(e);
}
}
}
match first_err {
Some(e) => Err(e),
None => sink.close(),
}
})
.map_err(|e| Error::IoError { source: e })?;
Ok(Pipeline { tx, handle })
}
/// Push one item. Blocks if the channel is full — that's the
/// back-pressure the whole primitive exists to provide. Returns
/// the item back if the consumer thread is gone (panicked or
/// already returned).
///
/// After the consumer returns [`Flow::Stop`], `send` will silently
/// buffer items into the channel until the channel fills, then
/// return `Err(item)` once the consumer has dropped its receiver.
/// Producers that need to stop pushing on `Stop` should track an
/// independent signal (e.g. `Halt`) — `send` alone is not the
/// notification edge.
pub fn send(&self, item: I) -> Result<(), I> {
self.tx.send(item).map_err(|e| e.0)
}
/// Non-blocking variant of [`Pipeline::send`]. If the channel is
/// full or the consumer has hung up, the item is returned in
/// `Err`. Useful for best-effort signalling (e.g. sweep's
/// throttled `StatsRequest`) where dropping the message is
/// preferable to blocking the producer.
pub fn try_send(&self, item: I) -> Result<(), std::sync::mpsc::TrySendError<I>> {
self.tx.try_send(item)
}
/// Drop the producer-side channel and wait for the consumer
/// thread to finish. Returns whatever the consumer's `close()`
/// produced, or the first `apply` error, or — on consumer panic —
/// an `Error::IoError` whose source is `io::Error::other(...)`
/// with a "pipeline consumer panicked: <payload>" message
/// (callers can match on the constant prefix).
pub fn finish(self) -> Result<R, Error> {
let Pipeline { tx, handle } = self;
// Explicit drop, although the destructure already drops `tx`
// at end-of-scope. Being explicit keeps the intent obvious.
drop(tx);
match handle.join() {
Ok(result) => result,
Err(payload) => {
// Preserve the original panic message when the
// consumer's panic payload was a `&str` or `String`
// (the two stdlib formats that `panic!` produces).
// Anything else falls back to "(no message)".
let msg = payload
.downcast_ref::<&'static str>()
.copied()
.or_else(|| payload.downcast_ref::<String>().map(|s| s.as_str()))
.unwrap_or("(no message)");
Err(Error::IoError {
source: io::Error::other(format!("pipeline consumer panicked: {msg}")),
})
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
/// Sums u64s; returns the total from `close`.
struct SumSink {
total: u64,
}
impl Sink<u64> for SumSink {
type Output = u64;
fn apply(&mut self, item: u64) -> Result<Flow, Error> {
self.total += item;
Ok(Flow::Continue)
}
fn close(self) -> Result<u64, Error> {
Ok(self.total)
}
}
#[test]
fn happy_path_sums_items() {
let pipe = Pipeline::spawn(DEFAULT_PIPELINE_DEPTH, SumSink { total: 0 })
.expect("spawn should succeed");
let mut expected = 0u64;
for i in 0..100u64 {
expected += i;
pipe.send(i).expect("send should succeed");
}
let total = pipe.finish().expect("finish should succeed");
assert_eq!(total, expected);
assert_eq!(total, (0..100u64).sum::<u64>());
}
/// Sleeps `delay` per apply; counts how many it received.
struct SlowSink {
delay: Duration,
count: Arc<AtomicUsize>,
}
impl Sink<()> for SlowSink {
type Output = usize;
fn apply(&mut self, _item: ()) -> Result<Flow, Error> {
std::thread::sleep(self.delay);
self.count.fetch_add(1, Ordering::SeqCst);
Ok(Flow::Continue)
}
fn close(self) -> Result<usize, Error> {
Ok(self.count.load(Ordering::SeqCst))
}
}
#[test]
fn back_pressure_blocks_sender() {
// depth=2 + 5 sends + 50ms/apply: with the consumer pinned at
// 50 ms per item, the producer can buffer 2 (channel cap) +
// 1 (consumer in flight) = 3 items before sends 4 and 5 must
// block on consumer progress. Wall-clock floor across all 5
// sends is therefore ~2 * 50ms = 100ms (sends 4 and 5 each
// wait roughly one apply-cycle). Use 80 ms as the assertion
// floor to stay above the 50ms-per-item progress floor while
// tolerating CI jitter — it still proves blocking is real.
let count = Arc::new(AtomicUsize::new(0));
let sink = SlowSink {
delay: Duration::from_millis(50),
count: count.clone(),
};
let pipe = Pipeline::spawn(2, sink).expect("spawn should succeed");
let start = Instant::now();
for _ in 0..5 {
pipe.send(()).expect("send should succeed");
}
let elapsed_send = start.elapsed();
let total = pipe.finish().expect("finish should succeed");
assert_eq!(total, 5);
assert!(
elapsed_send >= Duration::from_millis(80),
"back-pressure not observed: 5 sends with depth=2 and 50ms/apply \
took {elapsed_send:?}, expected ≥ ~100ms (one or more sends \
should have blocked behind the consumer)"
);
}
/// Returns `Err` on the Nth apply (1-indexed). Tracks all calls.
struct FailOnNthSink {
n: usize,
seen: Arc<AtomicUsize>,
close_called: Arc<AtomicUsize>,
}
impl Sink<u64> for FailOnNthSink {
type Output = ();
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
let i = self.seen.fetch_add(1, Ordering::SeqCst) + 1;
if i == self.n {
Err(Error::DecryptFailed)
} else {
Ok(Flow::Continue)
}
}
fn close(self) -> Result<(), Error> {
self.close_called.fetch_add(1, Ordering::SeqCst);
Ok(())
}
}
#[test]
fn apply_error_drains_then_propagates() {
let seen = Arc::new(AtomicUsize::new(0));
let close_called = Arc::new(AtomicUsize::new(0));
let pipe = Pipeline::spawn(
DEFAULT_PIPELINE_DEPTH,
FailOnNthSink {
n: 3,
seen: seen.clone(),
close_called: close_called.clone(),
},
)
.expect("spawn should succeed");
// Send 10 items. Subsequent sends after the 3rd error must
// still succeed (the consumer is draining).
for i in 0..10u64 {
pipe.send(i).expect("send should succeed even after error");
}
let res = pipe.finish();
assert!(matches!(res, Err(Error::DecryptFailed)));
assert_eq!(
close_called.load(Ordering::SeqCst),
0,
"close() must not be called when apply returned Err"
);
// The consumer kept calling `recv` to drain after the error;
// it just stopped invoking `apply`. So `seen` is exactly 3
// (apply was called for items 1, 2, 3).
assert_eq!(seen.load(Ordering::SeqCst), 3);
}
/// Returns `Flow::Stop` on the Nth apply.
struct StopOnNthSink {
n: usize,
seen: Arc<AtomicUsize>,
close_called: Arc<AtomicUsize>,
}
impl Sink<u64> for StopOnNthSink {
type Output = usize;
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
let i = self.seen.fetch_add(1, Ordering::SeqCst) + 1;
if i >= self.n {
Ok(Flow::Stop)
} else {
Ok(Flow::Continue)
}
}
fn close(self) -> Result<usize, Error> {
self.close_called.fetch_add(1, Ordering::SeqCst);
Ok(self.seen.load(Ordering::SeqCst))
}
}
#[test]
fn apply_stop_calls_close_and_returns_output() {
let seen = Arc::new(AtomicUsize::new(0));
let close_called = Arc::new(AtomicUsize::new(0));
let pipe = Pipeline::spawn(
DEFAULT_PIPELINE_DEPTH,
StopOnNthSink {
n: 3,
seen: seen.clone(),
close_called: close_called.clone(),
},
)
.expect("spawn should succeed");
// Send 10 items. After Stop, subsequent sends may either
// succeed (already buffered) or fail with Err(I) (channel
// closed). Both are valid — we don't assert on the send
// results.
for i in 0..10u64 {
let _ = pipe.send(i);
}
let out = pipe.finish().expect("finish should succeed after Stop");
assert_eq!(close_called.load(Ordering::SeqCst), 1);
// At least 3 items processed (the one that returned Stop).
assert!(
out >= 3,
"expected ≥ 3 applies before Stop took effect, got {out}"
);
}
/// Panics on the first apply call.
struct PanickingSink;
impl Sink<u64> for PanickingSink {
type Output = ();
fn apply(&mut self, _item: u64) -> Result<Flow, Error> {
panic!("synthetic test panic");
}
fn close(self) -> Result<(), Error> {
Ok(())
}
}
#[test]
fn consumer_panic_becomes_io_error() {
// Silence the panic message that would otherwise pollute the
// test output — we expect this panic.
let prev = std::panic::take_hook();
std::panic::set_hook(Box::new(|_| {}));
let pipe =
Pipeline::spawn(DEFAULT_PIPELINE_DEPTH, PanickingSink).expect("spawn should succeed");
// First send may succeed (item buffered before panic) or fail
// (channel closed after panic) — either is fine.
let _ = pipe.send(1);
// Drain a few more sends; once the channel is closed they'll
// return Err(I), which we just discard.
for i in 0..5u64 {
let _ = pipe.send(i);
}
let res = pipe.finish();
std::panic::set_hook(prev);
match res {
Err(Error::IoError { source }) => {
let msg = source.to_string();
// Constant prefix lets callers match without parsing
// the variable payload tail.
assert!(
msg.contains("pipeline consumer panicked"),
"expected constant panic prefix, got: {msg}"
);
// The original `panic!` payload (a `&'static str`) must
// be preserved — without the downcast the message
// would just be the prefix.
assert!(
msg.contains("synthetic test panic"),
"expected original panic payload, got: {msg}"
);
}
other => panic!("expected Err(IoError), got {other:?}"),
}
}
}
+9
View File
@@ -19,6 +19,11 @@ use std::fs::File;
use std::os::unix::io::{AsRawFd, RawFd};
pub(crate) struct WritebackPipeline {
/// Aliases the wrapping `WritebackFile::file`. Only valid for the
/// lifetime of that struct — moving the `File` independently
/// would silently UAF this fd. The pipeline is a private field of
/// `WritebackFile` and never exposed outside that wrapper, which
/// is what keeps the alias sound.
fd: RawFd,
chunk_bytes: u64,
last_flush_pos: u64,
@@ -26,6 +31,10 @@ pub(crate) struct WritebackPipeline {
}
impl WritebackPipeline {
/// Construct a pipeline aliasing `file`'s file descriptor. The
/// returned `WritebackPipeline` MUST be dropped before `file`
/// itself, or kept inside the same struct that owns `file` — the
/// alias is unchecked.
pub(crate) fn new(file: &File, start_pos: u64, chunk_bytes: u64) -> Self {
Self {
fd: file.as_raw_fd(),
+129
View File
@@ -0,0 +1,129 @@
//! `WritebackFile` — a `File` wrapper whose reason for existing is the
//! bounded-cache writeback pipeline.
//!
//! Why: large sequential writes (sweep, patch, mux on UHD-scale output)
//! left to the kernel's default writeback policy accumulate hundreds of
//! megabytes of dirty pages and then burst-flush, stalling subsequent
//! writes for seconds at a time. `WritebackFile` drives a continuous
//! [`super::writeback::WritebackPipeline`] that on Linux issues
//! incremental `sync_file_range` + `posix_fadvise(DONTNEED)` calls at
//! 32 MB granularity so dirty pages drain at the same rate they're
//! produced. macOS and Windows fall through to a no-op pipeline — their
//! default cache policies have not been shown to exhibit the same
//! pathology for this access pattern.
//!
//! It implements `Write` and `Seek` so any call site that wrote to a
//! plain `File` through those traits (sweep, patch, mux) can swap in
//! `WritebackFile` without touching the body of the loop. The wrapper
//! also tracks the current file position to feed the pipeline with
//! progress + seek boundaries.
//!
//! See `super::writeback::linux` for the underlying pathology and the
//! strategy.
use std::fs::{File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write};
use std::path::Path;
use super::writeback::WritebackPipeline;
const CHUNK_BYTES: u64 = 32 * 1024 * 1024;
pub(crate) struct WritebackFile {
file: File,
pipeline: WritebackPipeline,
pos: u64,
}
impl WritebackFile {
/// Wrap an open `File`. The current OS file position is queried
/// once so the pipeline starts tracking from wherever the file
/// already is (typically 0 for fresh files; non-zero for resumed
/// or appended files).
pub(crate) fn new(mut file: File) -> io::Result<Self> {
let pos = file.stream_position()?;
let pipeline = WritebackPipeline::new(&file, pos, CHUNK_BYTES);
Ok(Self {
file,
pipeline,
pos,
})
}
/// Create a new file at `path` (truncating any existing contents)
/// and wrap it. Convenience for the common
/// `File::create(path)` + `WritebackFile::new(file)` pair so callers
/// don't have to assemble a `File` first.
pub(crate) fn create(path: &Path) -> io::Result<Self> {
let file = File::create(path)?;
Self::new(file)
}
/// Open an existing file at `path` for writing (no truncation) and
/// wrap it. Mirrors `File::open` semantics for the writable case
/// — used by patch / resume paths that mutate an existing ISO in
/// place.
pub(crate) fn open(path: &Path) -> io::Result<Self> {
let file = OpenOptions::new().write(true).open(path)?;
Self::new(file)
}
/// Drain in-flight writeback then issue a full fsync. Use this in
/// place of `File::sync_all`.
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
self.pipeline.finalize();
self.file.sync_all()
}
}
impl Write for WritebackFile {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.file.write(buf)?;
self.pos += n as u64;
self.pipeline.note_progress(self.pos);
Ok(n)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.file.write_all(buf)?;
self.pos += buf.len() as u64;
self.pipeline.note_progress(self.pos);
Ok(())
}
fn flush(&mut self) -> io::Result<()> {
self.file.flush()
}
}
impl Seek for WritebackFile {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
let p = self.file.seek(from)?;
// Only treat seeks that actually move the position as
// boundaries — sweep does a redundant `seek(Current(pos))`
// before every write, and we don't want that to drain the
// pipeline on every iteration.
if p != self.pos {
self.pipeline.handle_seek(p);
self.pos = p;
}
Ok(p)
}
}
impl Drop for WritebackFile {
fn drop(&mut self) {
// Run the pipeline's tail finalize so the last in-flight chunk
// gets its `WAIT_AFTER` + `posix_fadvise(DONTNEED)`. Without
// this, callers that drop a `WritebackFile` without calling
// `sync_all` (panic, early-return, idiomatic `let _ = w;`)
// leave the trailing chunk in cache; the kernel still flushes
// on close, but the bounded-cache invariant fails at the tail.
// We deliberately do *not* call `self.file.sync_all()` here —
// close already triggers a flush, and an `fsync` from `Drop`
// would silently swallow its `io::Error` anyway. `finalize` is
// idempotent so an explicit `sync_all` followed by drop is
// still safe.
self.pipeline.finalize();
}
}
-84
View File
@@ -1,84 +0,0 @@
//! `Writer` — a drop-in `File` wrapper that keeps the kernel page
//! cache bounded during large sequential output.
//!
//! Implements `Write` and `Seek`, so any call site that uses `File`
//! through those traits (sweep, mux, patch) can swap to `Writer`
//! without touching the body of the loop. The wrapper tracks the
//! current file position and forwards each write to a
//! [`super::writeback::WritebackPipeline`], which on Linux schedules
//! incremental `sync_file_range` + `posix_fadvise(DONTNEED)` calls
//! to drain dirty pages continuously instead of letting the kernel
//! burst-flush hundreds of MB at a time.
//!
//! See `super::writeback::linux` for the pathology and the strategy.
use std::fs::File;
use std::io::{self, Seek, SeekFrom, Write};
use super::writeback::WritebackPipeline;
const CHUNK_BYTES: u64 = 32 * 1024 * 1024;
pub(crate) struct Writer {
file: File,
pipeline: WritebackPipeline,
pos: u64,
}
impl Writer {
/// Wrap an open `File`. The current OS file position is queried
/// once so the pipeline starts tracking from wherever the file
/// already is (typically 0 for fresh files; non-zero for resumed
/// or appended files).
pub(crate) fn new(mut file: File) -> io::Result<Self> {
let pos = file.stream_position()?;
let pipeline = WritebackPipeline::new(&file, pos, CHUNK_BYTES);
Ok(Self {
file,
pipeline,
pos,
})
}
/// Drain in-flight writeback then issue a full fsync. Use this in
/// place of `File::sync_all`.
pub(crate) fn sync_all(&mut self) -> io::Result<()> {
self.pipeline.finalize();
self.file.sync_all()
}
}
impl Write for Writer {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let n = self.file.write(buf)?;
self.pos += n as u64;
self.pipeline.note_progress(self.pos);
Ok(n)
}
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
self.file.write_all(buf)?;
self.pos += buf.len() as u64;
self.pipeline.note_progress(self.pos);
Ok(())
}
fn flush(&mut self) -> io::Result<()> {
self.file.flush()
}
}
impl Seek for Writer {
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
let p = self.file.seek(from)?;
// Only treat seeks that actually move the position as
// boundaries — sweep does a redundant `seek(Current(pos))`
// before every write, and we don't want that to drain the
// pipeline on every iteration.
if p != self.pos {
self.pipeline.handle_seek(p);
self.pos = p;
}
Ok(p)
}
}
+50 -6
View File
@@ -79,6 +79,7 @@ pub mod disc;
pub mod drive;
pub mod error;
pub mod event;
pub mod halt;
pub(crate) mod identity;
pub(crate) mod ifo;
pub(crate) mod io;
@@ -116,6 +117,28 @@ pub use drive::{Drive, DriveStatus, find_drive};
// codes to localized messages. See `error.rs` for the full taxonomy.
pub use error::{Error, Result};
// ─── 0.18 primitives ────────────────────────────────────────────────────────
//
// One-bit cooperative cancellation token, shared by every long-running loop
// in libfreemkv (sweep, patch, mux). Replaces the ad-hoc `Arc<AtomicBool>`
// flags scattered across 0.17 (`DiscStream::set_halt`, autorip's
// `HALT_FLAGS` registry). Clone it cheaply; pass it by value into each
// component; poll `is_cancelled()` inside the loop body.
pub use halt::Halt;
// Generic bounded producer/consumer primitive used by sweep, patch, and
// mux to overlap reads with writes via a dedicated consumer thread.
// `Pipeline::spawn(depth, sink)` spawns the consumer; `pipe.send(item)`
// pushes one item with back-pressure; `pipe.finish()` joins the
// consumer and surfaces its `close()` output. Callers implement `Sink`
// to define per-item behaviour and end-of-stream finalisation.
//
// `DEFAULT_PIPELINE_DEPTH` (=4) is the depth sweep + mux use; patch
// uses `WRITE_THROUGH_DEPTH` (=1) so each read fully drains before the
// next can enqueue. Returning `Flow::Stop` from `apply` ends the
// consumer cleanly (still calls `close()`).
pub use io::pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink, WRITE_THROUGH_DEPTH};
// ─── Drive events (low-level callbacks) ─────────────────────────────────────
pub use event::{Event, EventKind};
pub use identity::DriveId;
@@ -137,11 +160,15 @@ pub use decrypt::{DecryptKeys, decrypt_sectors};
// for displaying disc name + format quickly while a full scan runs in the
// background. The codec / channel / resolution enums are the canonical
// structured representation; never compare against display strings.
// Note: `disc::Stream` here is the codec enum (audio / video / sub kind)
// — not the `pes::Stream` trait re-exported below as `PesStream`. Two
// different concepts, the same short name; both stay because both are
// load-bearing in their respective domains.
pub use disc::{
AacsState, AudioChannels, AudioStream, Clip, Codec, ColorSpace, ContentFormat, DamageSeverity,
Disc, DiscFormat, DiscId, DiscTitle, Extent, FrameRate, HdrFormat, KeySource, LabelPurpose,
LabelQualifier, Resolution, SampleRate, ScanOptions, Stream, SubtitleStream, VideoStream,
classify_damage,
LabelQualifier, PatchOptions, PatchOutcome, Resolution, SampleRate, ScanOptions, Stream,
SubtitleStream, SweepOptions, VideoStream, classify_damage,
};
// ─── Streams ────────────────────────────────────────────────────────────────
@@ -159,6 +186,15 @@ pub use disc::{
// Most consumers use the URL resolvers (`input()` / `output()`) which pick
// the right type from a scheme:// URL. Direct construction is for callers
// that need to wire custom readers (e.g. autorip's drive-session reuse).
// 0.18 trait split: `FrameSource` (read-only) and `FrameSink` (write-only)
// supersede the unified `pes::Stream`. The old `Stream` re-export below
// stays available for the deprecation window — re-exported as
// `PesStream` to disambiguate from `disc::Stream` (the codec-kind enum
// re-exported above), which would otherwise collide at the crate root.
#[allow(deprecated)]
pub use pes::Stream as PesStream;
pub use pes::{FrameSink, FrameSource, PesFrame};
pub use mux::DiscStream;
pub use mux::M2tsStream;
pub use mux::MkvStream;
@@ -170,10 +206,18 @@ pub use mux::{InputOptions, StreamUrl, input, output, parse_url};
// ─── Lower-level surfaces ───────────────────────────────────────────────────
//
// `ScsiTransport` is the platform-abstraction trait Drive uses; expose for
// out-of-tree platform backends. `SectorReader` lets callers feed any byte
// source (test harness, network image, SMB share) into the disc scan
// pipeline; `FileSectorReader` is the standard ISO-on-disk implementation.
// out-of-tree platform backends. `SectorSource` / `SectorSink` are the 0.18
// direction-typed read/write traits; `FileSectorSource` and `FileSectorSink`
// are the ISO-on-disk implementations. [`DecryptingSectorSource`] is the
// single decrypt-on-read decorator (AACS / CSS / none) — wrap any
// `SectorSource` to get plaintext sectors out. The legacy `SectorReader` /
// `FileSectorReader` names stay re-exported through the 0.18 migration
// window so existing call sites compile unchanged; a blanket impl makes
// every `SectorReader` automatically usable as a `SectorSource`.
pub use scsi::{DriveInfo, ScsiSense, ScsiTransport, drive_has_disc, list_drives};
pub use sector::{FileSectorReader, SectorReader};
pub use sector::{
DecryptingSectorSource, FileSectorReader, FileSectorSink, FileSectorSource, SectorReader,
SectorSink, SectorSource,
};
pub use speed::DriveSpeed;
pub use udf::{UdfFs, read_filesystem};
+209 -22
View File
@@ -8,10 +8,11 @@
use crate::disc::{Disc, DiscTitle, Extent};
use crate::drive::extract_scsi_context;
use crate::event::{BatchSizeReason, Event, EventKind};
use crate::sector::SectorReader;
use crate::halt::Halt;
use crate::sector::{DecryptingSectorSource, SectorReader, SectorSource};
use std::io;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::AtomicBool;
/// Ramp back up to the preferred batch size after this many sectors
/// of clean reading at the current (reduced) size. 100 MiB = 51,200 sectors.
@@ -99,9 +100,19 @@ impl AdaptiveBatch {
/// Sources: physical drive, ISO file, or any SectorReader.
/// Decrypt, demux, and codec parsing happen internally.
pub struct DiscStream {
reader: Box<dyn SectorReader>,
/// Underlying sector source wrapped in the 0.18
/// [`DecryptingSectorSource`] decorator. Every `read_sectors`
/// call yields plaintext, so `fill_extents` no longer needs an
/// inline `decrypt::decrypt_sectors` step. `DecryptKeys::None`
/// (raw / unencrypted disc) makes the decorator a pass-through.
reader: DecryptingSectorSource<Box<dyn SectorReader>>,
title: DiscTitle,
disc: Option<Disc>,
/// Mirror of the keys handed in at construction. The decorator
/// owns the cryptographic state; this field is kept for
/// metadata-side callers (`info()` and friends) that want to
/// know whether the disc was encrypted, without reaching through
/// the wrapper.
decrypt_keys: crate::decrypt::DecryptKeys,
// Extents to read
@@ -120,11 +131,13 @@ pub struct DiscStream {
adaptive: AdaptiveBatch,
pub errors: u64,
pub skip_errors: bool,
/// When set and the flag is raised, fill_extents returns Err(Halted) at the
/// next retry boundary. Unlike skip_errors, this propagates the error up so
/// the rip terminates cleanly. Share the Arc with Drive::halt_flag() to get
/// unified Stop behavior across drive reads and sector processing.
halt: Option<Arc<AtomicBool>>,
/// When set and the token is cancelled, fill_extents returns Err(Halted)
/// at the next retry boundary. Unlike skip_errors, this propagates the
/// error up so the rip terminates cleanly. Construct with
/// [`DiscStream::with_halt`] (preferred) or set post-hoc via the
/// deprecated [`DiscStream::set_halt`] bridge — both populate this same
/// field and either entry point yields one source of truth.
halt: Option<Halt>,
event_fn: Option<Box<dyn Fn(Event) + Send>>,
eof: bool,
@@ -189,7 +202,11 @@ impl DiscStream {
}
Self {
reader,
// Wrap the input reader in DecryptingSectorSource so the
// internal fill_extents path sees plaintext bytes. For
// DecryptKeys::None (unencrypted / raw / test fixtures)
// the decorator is a pass-through.
reader: DecryptingSectorSource::new(reader, decrypt_keys.clone()),
title,
disc: None,
decrypt_keys,
@@ -219,19 +236,39 @@ impl DiscStream {
self.event_fn = Some(Box::new(f));
}
/// Share a halt flag — typically from `Drive::halt_flag()`. When raised,
/// the next read-retry boundary inside fill_extents returns Err(Halted)
/// instead of continuing. Required for Stop to work during dense bad-sector
/// regions (where read() loops internally waiting for enough clean data to
/// emit a frame and would otherwise never check an external stop signal).
/// Constructor-time builder: attach a [`Halt`] token so that when
/// any clone is cancelled, the next read-retry boundary inside
/// `fill_extents` returns `Err(Halted)`. Required for Stop to work
/// during dense bad-sector regions (where the outer PES read() loop
/// can spend minutes inside fill_extents before emitting a frame).
///
/// Preferred over the post-hoc [`DiscStream::set_halt`] bridge —
/// pass the same `Halt` clone you hand to sweep / patch / mux so
/// every phase observes a single Stop signal.
pub fn with_halt(mut self, halt: Halt) -> Self {
self.halt = Some(halt);
self
}
/// Bridge for callers that haven't migrated to the
/// [`DiscStream::with_halt`] constructor-time path yet. Wraps the
/// supplied `Arc<AtomicBool>` as a [`Halt`] (`Halt::from_arc`) and
/// stores it in the same internal slot, so a halt installed via
/// either entry point goes through one halt-check inside
/// `fill_extents`. Calling `set_halt` after `with_halt` (or vice
/// versa) replaces the previous token with the new one.
#[deprecated(
since = "0.18.0",
note = "use `DiscStream::with_halt(Halt)` at construction instead"
)]
pub fn set_halt(&mut self, flag: Arc<AtomicBool>) {
self.halt = Some(flag);
self.halt = Some(Halt::from_arc(flag));
}
fn is_halted(&self) -> bool {
self.halt
.as_ref()
.map(|h| h.load(Ordering::Relaxed))
.map(|h| h.is_cancelled())
.unwrap_or(false)
}
@@ -241,9 +278,12 @@ impl DiscStream {
}
}
/// Skip decryption — return raw encrypted bytes.
/// Skip decryption — return raw encrypted bytes. Updates both
/// the metadata-side key field and the wrapped reader's keys so
/// subsequent `read_sectors` calls become a pass-through.
pub fn set_raw(&mut self) {
self.decrypt_keys = crate::decrypt::DecryptKeys::None;
self.reader.set_keys(crate::decrypt::DecryptKeys::None);
}
/// Get the scanned Disc (for listing all titles).
@@ -347,6 +387,7 @@ impl DiscStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for DiscStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
if let Some(frame) = self.pending_frames.pop_front() {
@@ -419,11 +460,10 @@ impl crate::pes::Stream for DiscStream {
}
let bytes = self.buf_valid;
if let Err(e) =
crate::decrypt::decrypt_sectors(&mut self.read_buf[..bytes], &self.decrypt_keys, 0)
{
return Err(e.into());
}
// Plaintext: the wrapped reader (DecryptingSectorSource)
// applied AACS / CSS in-place during fill_extents'
// read_sectors call. The pre-0.18 inline decrypt step
// lived here.
if let Some(ref mut demuxer) = self.ts_demuxer {
let packets = demuxer.feed(&self.read_buf[..bytes]);
@@ -525,3 +565,150 @@ impl crate::pes::Stream for DiscStream {
true
}
}
#[cfg(test)]
mod tests {
//! `DiscStream` is the only meaningful `FrameSource` impl in tree (every
//! other concrete `pes::Stream` impl in `mux/*` is a sink). The 0.18
//! round-1 blanket `impl<T: pes::Stream + Send> pes::FrameSource for T`
//! covers `DiscStream` for free as long as it is `Send`. These tests
//! lock that down: a static `Send` assertion plus a `Box<dyn FrameSource>`
//! round trip exercising every `FrameSource` method through the trait
//! object, so future Send-breaking edits to `DiscStream`'s interior
//! types fail at compile time and the trait-bridge dispatch is verified
//! at runtime.
#![allow(deprecated)] // exercising the 0.18 deprecation-window blanket bridge.
use super::*;
use crate::disc::{ContentFormat, DiscTitle};
use crate::pes::FrameSource;
/// Static-assert `DiscStream: Send`. The blanket
/// `impl<T: pes::Stream + Send> pes::FrameSource for T` only fires for
/// `Send` types — if a future field on `DiscStream` is non-`Send` (e.g.
/// a `Box<dyn Read>` instead of `Box<dyn SectorReader>`), this fails
/// at compile time, before the runtime trait-object test below.
fn _assert_disc_stream_is_send() {
fn requires_send<T: Send>() {}
requires_send::<DiscStream>();
}
/// Trivial `SectorReader` that yields zeroed sectors. Empty title means
/// the demuxer produces no PES frames, so `read()` walks the extents to
/// EOF and returns `Ok(None)`. That's enough to exercise the trait-object
/// dispatch — the goal here is the bridge, not the demuxer.
struct ZeroReader {
capacity: u32,
}
impl crate::sector::SectorReader for ZeroReader {
fn read_sectors(
&mut self,
_lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> crate::error::Result<usize> {
let bytes = count as usize * 2048;
buf[..bytes].fill(0);
Ok(bytes)
}
fn capacity(&self) -> u32 {
self.capacity
}
}
fn synthetic_title(sector_count: u32) -> DiscTitle {
DiscTitle {
extents: vec![crate::disc::Extent {
start_lba: 0,
sector_count,
}],
..DiscTitle::empty()
}
}
/// Smallest credible witness that `DiscStream` flows through the
/// `FrameSource` blanket impl: build a `Box<dyn FrameSource>`, drive
/// `read()` to EOF, exercise `info()` / `headers_ready()` /
/// `codec_private()` through the trait object. The trait-bridge
/// correctness is what's being verified — not demuxer behaviour.
#[test]
fn frame_source_via_dyn_object() {
let reader = ZeroReader { capacity: 8 };
let title = synthetic_title(8);
let stream = DiscStream::new(
Box::new(reader),
title,
crate::decrypt::DecryptKeys::None,
8,
ContentFormat::BdTs,
);
let mut src: Box<dyn FrameSource> = Box::new(stream);
// Empty-title fixture has no streams configured, so headers are
// trivially ready and codec_private() yields nothing on track 0.
assert!(src.headers_ready());
assert!(src.codec_private(0).is_none());
let _ = src.info();
// Drive read() to EOF through the trait object — empty-title fixture
// produces no frames, but the call still routes through the blanket
// dispatch into Stream::read.
let mut frames = 0usize;
loop {
match src.read().expect("read") {
Some(_) => frames += 1,
None => break,
}
if frames > 1024 {
panic!("unexpected unbounded frame stream from empty title");
}
}
assert_eq!(frames, 0);
}
/// `is_halted()` must observe a cancellation signal regardless of
/// which entry point installed the token. The deprecated
/// `set_halt(Arc<AtomicBool>)` and the new `with_halt(Halt)` are
/// two views over one slot — flipping either bit must cause the
/// next `fill_extents` retry boundary to bail.
#[test]
fn halt_via_with_halt_observed_by_is_halted() {
let halt = Halt::new();
let stream = DiscStream::new(
Box::new(ZeroReader { capacity: 8 }),
synthetic_title(8),
crate::decrypt::DecryptKeys::None,
8,
crate::disc::ContentFormat::BdTs,
)
.with_halt(halt.clone());
assert!(!stream.is_halted());
halt.cancel();
assert!(
stream.is_halted(),
"with_halt token cancellation must be observed by is_halted()"
);
}
#[test]
fn halt_via_set_halt_bridge_observed_by_is_halted() {
let arc = Arc::new(AtomicBool::new(false));
let mut stream = DiscStream::new(
Box::new(ZeroReader { capacity: 8 }),
synthetic_title(8),
crate::decrypt::DecryptKeys::None,
8,
crate::disc::ContentFormat::BdTs,
);
stream.set_halt(arc.clone());
assert!(!stream.is_halted());
arc.store(true, std::sync::atomic::Ordering::Relaxed);
assert!(
stream.is_halted(),
"set_halt(Arc<AtomicBool>) bridge must observe Arc-side flips"
);
}
}
+32 -7
View File
@@ -18,10 +18,10 @@ const SCAN_SIZE: usize = 1024 * 1024;
enum Mode {
Write {
muxer: super::tsmux::TsMuxer<Box<dyn Write>>,
muxer: super::tsmux::TsMuxer<Box<dyn Write + Send>>,
},
Read {
reader: Box<dyn Read>,
reader: Box<dyn Read + Send>,
},
}
@@ -57,7 +57,7 @@ pub struct M2tsStream {
impl M2tsStream {
/// Create for writing PES frames → BD-TS output.
/// Writes FMKV metadata header, then muxes PES frames into BD transport stream.
pub fn create(mut writer: impl Write + 'static, title: &DiscTitle) -> io::Result<Self> {
pub fn create(mut writer: impl Write + Send + 'static, title: &DiscTitle) -> io::Result<Self> {
// Write FMKV metadata header
if !title.streams.is_empty() {
let m = meta::M2tsMeta::from_title(title);
@@ -72,7 +72,7 @@ impl M2tsStream {
DiscStream::Subtitle(s) => s.pid,
})
.collect();
let boxed: Box<dyn Write> = Box::new(writer);
let boxed: Box<dyn Write + Send> = Box::new(writer);
let mut muxer = super::tsmux::TsMuxer::new(boxed, &pids);
for (i, cp) in title.codec_privates.iter().enumerate() {
if let Some(data) = cp {
@@ -111,7 +111,7 @@ impl M2tsStream {
/// Open an M2TS stream for reading. Takes any Read source — file, pipe, socket.
///
/// Tries FMKV metadata header first. Falls back to PMT scan of first 1 MB.
pub fn open(mut reader: impl Read + 'static) -> io::Result<Self> {
pub fn open(mut reader: impl Read + Send + 'static) -> io::Result<Self> {
// Read first chunk — enough for FMKV header or PMT scan
let mut head = vec![0u8; SCAN_SIZE];
let head_len = read_fill(&mut reader, &mut head)?;
@@ -125,7 +125,7 @@ impl M2tsStream {
let (pids, parsers, pid_to_track) = Self::setup_pes(&title.streams);
// Chain: remaining head bytes + rest of reader
let remaining_head = &head[header_end..];
let chain: Box<dyn Read> =
let chain: Box<dyn Read + Send> =
Box::new(io::Cursor::new(remaining_head.to_vec()).chain(reader));
return Ok(Self {
disc_title: title.clone(),
@@ -150,7 +150,7 @@ impl M2tsStream {
let (pids, parsers, pid_to_track) = Self::setup_pes(&streams);
// Chain: full head (it's all TS data) + rest of reader
let chain: Box<dyn Read> = Box::new(io::Cursor::new(head).chain(reader));
let chain: Box<dyn Read + Send> = Box::new(io::Cursor::new(head).chain(reader));
Ok(Self {
disc_title: DiscTitle {
@@ -173,6 +173,7 @@ impl M2tsStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for M2tsStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
if let Some(frame) = self.pending_frames.pop_front() {
@@ -283,3 +284,27 @@ impl crate::pes::Stream for M2tsStream {
true
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `M2tsStream::create(writer, title)` to construct the write half;
/// calling `FrameSink::write` on an `M2tsStream::open(reader)` instance
/// returns `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for M2tsStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
+34 -4
View File
@@ -18,7 +18,7 @@ use crate::disc::*;
use std::io::{self, Read};
struct ReadState {
reader: Box<dyn Read>,
reader: Box<dyn Read + Send>,
cluster_ts_ms: i64,
/// Codec private data per track (track_number, hvcC/avcC bytes).
codec_privates: Vec<(u16, Vec<u8>)>,
@@ -26,7 +26,7 @@ struct ReadState {
enum Mode {
Write {
muxer: Option<MkvMuxer<Box<dyn WriteSeek>>>,
muxer: Option<MkvMuxer<Box<dyn WriteSeek + Send>>>,
},
Read(ReadState),
}
@@ -40,7 +40,7 @@ pub struct MkvStream {
impl MkvStream {
/// Create for writing PES frames → MKV container.
/// Codec privates come from title.codec_privates (populated by input stream).
pub fn create(writer: Box<dyn WriteSeek>, title: &DiscTitle) -> io::Result<Self> {
pub fn create(writer: Box<dyn WriteSeek + Send>, title: &DiscTitle) -> io::Result<Self> {
let mut tracks = Vec::new();
let mut has_default_video = false;
let mut has_default_audio = false;
@@ -79,7 +79,7 @@ impl MkvStream {
}
/// Open an MKV file for reading → PES frames.
pub fn open(mut reader: impl Read + 'static) -> io::Result<Self> {
pub fn open(mut reader: impl Read + Send + 'static) -> io::Result<Self> {
let (disc_title, codec_privates) = parse_mkv_header(&mut reader)?;
Ok(Self {
disc_title,
@@ -92,6 +92,7 @@ impl MkvStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for MkvStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
let rs = match self.mode {
@@ -188,6 +189,35 @@ impl crate::pes::Stream for MkvStream {
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `MkvStream::create(writer, title)` to construct the write half;
/// calling `FrameSink::write` on a `MkvStream::open(reader)` instance returns
/// `StreamReadOnly`. `finish` is where the Cues index is written, so it must
/// be called for the resulting MKV to be seekable.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for MkvStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
// The inner `MkvMuxer` is owned via `Option`, and `Stream::finish`
// already takes it via `Option::take()` — moving `*self` out of the
// box hands it the same field by-value, so the muxer's own
// by-value `finish()` runs correctly.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
// ── MKV header parsing (read side) ────────────────────────────
/// Returns (DiscTitle, codec_privates: Vec<(track_number, codec_private_bytes)>)
+26
View File
@@ -70,6 +70,7 @@ impl NetworkStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for NetworkStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
match &mut self.mode {
@@ -108,6 +109,30 @@ impl crate::pes::Stream for NetworkStream {
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `NetworkStream::connect(addr).meta(title)` to construct the write
/// half; calling `FrameSink::write` on `NetworkStream::listen(addr)` returns
/// `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for NetworkStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
// NetworkStream is PES-only — no IOStream/Read/Write byte interface.
#[cfg(test)]
@@ -157,6 +182,7 @@ mod tests {
#[test]
#[ignore] // Requires TCP; may be flaky in CI environments
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
fn network_pes_roundtrip() {
use crate::pes;
+53
View File
@@ -16,6 +16,7 @@ impl NullStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSink in follow-up commit.
impl crate::pes::Stream for NullStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
Ok(None)
@@ -30,3 +31,55 @@ impl crate::pes::Stream for NullStream {
&self.disc_title
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// The deprecation-window callers eventually migrate; this impl exists so
/// new callers can target `FrameSink` without waiting for the rest of the
/// migration to complete.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for NullStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::pes::FrameSink;
/// Smallest credible witness that the new FrameSink impl on a concrete
/// `mux/*` sink works through the trait object: build a boxed
/// `dyn FrameSink`, write a frame, finish it. The trait-bridge correctness
/// is what's being verified — not NullStream-specific behaviour.
#[test]
fn frame_sink_via_dyn_object_writes_and_finishes() {
let title = DiscTitle::empty();
let mut sink: Box<dyn FrameSink> = Box::new(NullStream::new(&title));
let frame = crate::pes::PesFrame {
track: 0,
pts: 0,
keyframe: true,
data: vec![0x01, 0x02, 0x03],
};
sink.write(&frame).unwrap();
// info() routes through the trait object.
let _ = sink.info();
// finish() consumes the Box<Self> — must compile and run.
sink.finish().unwrap();
}
}
+16 -5
View File
@@ -167,6 +167,7 @@ pub struct InputOptions {
}
/// Open a PES input stream (produces PES frames).
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource in follow-up commit.
pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::Stream>> {
let parsed = parse_url(url);
match parsed {
@@ -234,6 +235,7 @@ pub fn input(url: &str, opts: &InputOptions) -> io::Result<Box<dyn crate::pes::S
}
/// Open a PES output stream (consumes PES frames).
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSink in follow-up commit.
pub fn output(
url: &str,
title: &crate::disc::DiscTitle,
@@ -242,15 +244,24 @@ pub fn output(
match parsed {
StreamUrl::Mkv { ref path } => {
validate_file_path(path, "mkv")?;
let file = std::fs::File::create(path)?;
let writer: Box<dyn super::WriteSeek> =
Box::new(std::io::BufWriter::with_capacity(IO_BUF_SIZE, file));
// Wrap the output in `crate::io::WritebackFile` (bounded-cache
// writeback) so a UHD-scale MKV mux to slow / network-attached
// staging doesn't hit the dirty-page burst pathology that
// sweep already side-steps. BufWriter sits on top to coalesce
// mux's many small EBML element writes.
let writer: Box<dyn super::WriteSeek + Send> =
Box::new(std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
));
Ok(Box::new(MkvStream::create(writer, title)?))
}
StreamUrl::M2ts { ref path } => {
validate_file_path(path, "m2ts")?;
let file = std::fs::File::create(path)?;
let writer = std::io::BufWriter::with_capacity(IO_BUF_SIZE, file);
let writer = std::io::BufWriter::with_capacity(
IO_BUF_SIZE,
crate::io::WritebackFile::create(path)?,
);
Ok(Box::new(M2tsStream::create(writer, title)?))
}
StreamUrl::Network { ref addr } => {
+24
View File
@@ -60,6 +60,7 @@ impl StdioStream {
}
}
#[allow(deprecated)] // 0.18 trait split: migrate to FrameSource/FrameSink in follow-up commit.
impl crate::pes::Stream for StdioStream {
fn read(&mut self) -> io::Result<Option<crate::pes::PesFrame>> {
self.ensure_header_read()?;
@@ -104,3 +105,26 @@ impl crate::pes::Stream for StdioStream {
self.header_read || self.writer.is_some()
}
}
/// FrameSink sibling to the deprecated Stream impl; both coexist during the
/// 0.18 deprecation window. Caller may pick either at the trait-object
/// boundary — `Box<dyn Stream>` (deprecated) or `Box<dyn FrameSink>` (new).
/// Use `StdioStream::output(title)` to construct the write half; calling
/// `FrameSink::write` on a `StdioStream::input()` returns `StreamReadOnly`.
#[allow(deprecated)] // delegating to deprecated Stream during the 0.18 deprecation window so callers don't see the deprecation twice.
impl crate::pes::FrameSink for StdioStream {
fn write(&mut self, frame: &crate::pes::PesFrame) -> io::Result<()> {
<Self as crate::pes::Stream>::write(self, frame)
}
fn finish(self: Box<Self>) -> io::Result<()> {
// Why: Stream::finish takes &mut self, FrameSink::finish takes Box<Self>.
// Re-borrow inside the box, call Stream::finish, drop the box.
let mut s: Self = *self;
<Self as crate::pes::Stream>::finish(&mut s)
}
fn info(&self) -> &DiscTitle {
<Self as crate::pes::Stream>::info(self)
}
}
+293 -34
View File
@@ -78,43 +78,19 @@ impl PesFrame {
}
}
/// A PES frame source or sink. Each implementor is **either** read-only or
/// write-only — never both.
///
/// Implementors fall into two camps:
///
/// - **Read sources**: `DiscStream` (drive or ISO), `M2tsStream` (when
/// constructed from an existing file), `MkvStream` (demux), `NetworkStream`
/// (TCP listener), `StdioStream::input()`. These return frames from
/// `read()` and surface `StreamWriteOnly` (E9001) from `write()`.
/// - **Write sinks**: `MkvStream::create`, `M2tsStream::create`,
/// `NetworkStream::connect`, `StdioStream::output()`, `NullStream`.
/// These accept frames in `write()` and surface `StreamReadOnly` (E9000)
/// from `read()`. Always call `finish()` when done — that's where MKV
/// writes its `Cues` index and `M2tsStream` flushes the TS muxer.
///
/// Direction is established at construction; mixing produces an error code,
/// not a panic. Most consumers don't construct streams directly — call
/// `mux::input(url, opts)` / `mux::output(url, title)` and let URL parsing
/// pick the right type.
///
/// `info()` returns the stream's `DiscTitle` metadata (track list, codec
/// info, duration). For sources it's parsed from the input; for sinks it's
/// the metadata supplied at creation. Stable across all reads.
///
/// `codec_private(track)` exposes per-track initialization data
/// (H.264 SPS/PPS, HEVC VPS/SPS/PPS, AC-3 fscod, etc.) that some output
/// formats need before any frame can be written. `headers_ready()` returns
/// false until enough input frames have been seen to populate every video
/// track's codec-private blob — callers buffer frames they read until
/// `headers_ready()` returns true.
/// Deprecated; use [`FrameSource`] for read-only sources or [`FrameSink`]
/// for write-only sinks. The runtime direction-error semantics
/// (`StreamReadOnly` / `StreamWriteOnly` from a wrong-direction call) are
/// removed in 0.18 — direction is type-checked.
#[deprecated(
since = "0.18.0",
note = "use FrameSource (read-only) or FrameSink (write-only) instead"
)]
pub trait Stream {
/// Read the next frame, or `Ok(None)` at end of stream. Returns
/// `StreamWriteOnly` (E9001) on a write-only sink.
/// Read the next frame, or `Ok(None)` at end of stream.
fn read(&mut self) -> std::io::Result<Option<PesFrame>>;
/// Write a frame to the sink. Returns `StreamReadOnly` (E9000) on a
/// read-only source.
/// Write a frame to the sink.
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()>;
/// Finalize the stream: flush buffered frames, write any container
@@ -140,6 +116,111 @@ pub trait Stream {
}
}
/// Read-only source of PES frames.
///
/// Replaces the read half of the deprecated [`Stream`] trait. Implementors
/// produce frames via [`read`](FrameSource::read) and never accept writes —
/// passing a `FrameSource` where a sink is expected is a compile error,
/// not a runtime `E9001`.
///
/// `info()` returns the source's `DiscTitle` metadata (track list, codec
/// info, duration). It must be stable across the lifetime of the source.
///
/// `codec_private(track)` exposes per-track codec initialization data
/// (H.264 SPS/PPS, HEVC VPS/SPS/PPS, AC-3 fscod, etc.) that downstream
/// muxers may need before any frame is written. `headers_ready()` returns
/// false until enough input frames have been seen to populate every video
/// track's codec-private blob — callers buffer frames they read until
/// `headers_ready()` returns true.
pub trait FrameSource: Send {
/// Read the next frame, or `Ok(None)` at end of stream.
fn read(&mut self) -> std::io::Result<Option<PesFrame>>;
/// Source metadata. Stable across reads — implementors must return a
/// consistent reference for the lifetime of the source.
fn info(&self) -> &crate::disc::DiscTitle;
/// Codec initialization data for a track (SPS/PPS, AC-3 fscod, etc.).
/// `None` for tracks that don't need codec_private (raw passthrough).
fn codec_private(&self, _track: usize) -> Option<Vec<u8>> {
None
}
/// True when `codec_private` is available for every video track —
/// callers buffer input frames until this flips, since some output
/// formats (MKV) can't write frames without codec init data.
fn headers_ready(&self) -> bool {
true
}
}
/// Write-only sink of PES frames.
///
/// Replaces the write half of the deprecated [`Stream`] trait. Implementors
/// accept frames via [`write`](FrameSink::write) and finalize via
/// [`finish`](FrameSink::finish) — passing a `FrameSink` where a source is
/// expected is a compile error, not a runtime `E9000`.
///
/// `finish` takes `Box<Self>` (rather than `&mut self` like `Stream::finish`)
/// so that finalization is a one-shot terminal operation: callers cannot use
/// the sink after `finish` returns. This is the standard idiom for terminal
/// methods on `dyn Trait` objects.
pub trait FrameSink: Send {
/// Write a frame to the sink.
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()>;
/// Finalize the sink: flush buffered frames, write any container index
/// (e.g. MKV `Cues`), close the underlying file/socket. Consumes the
/// sink — callers cannot use it afterwards.
fn finish(self: Box<Self>) -> std::io::Result<()>;
/// Sink metadata. Stable across writes — implementors must return a
/// consistent reference for the lifetime of the sink.
fn info(&self) -> &crate::disc::DiscTitle;
}
// Bridge: any **`Send`** type implementing the deprecated `Stream` trait
// is also a `FrameSource`. This lets existing concrete `Stream` impls in
// `mux/*` satisfy `FrameSource` bounds without per-type migration during
// the 0.18 deprecation window.
//
// **Send caveat (read me before tightening `Stream` itself).** This
// blanket carries a `T: Send` bound rather than promoting `Send` to a
// supertrait of `Stream`, because not every concrete in-tree `Stream`
// impl is `Send`: `MkvStream` and `M2tsStream` carry `Box<dyn Read>`
// and `Box<dyn Write>` fields whose trait objects don't include `Send`.
// Adding `Stream: Send` would force a wider audit (every `Box<dyn Read>`
// becomes `Box<dyn Read + Send>`) than this commit is taking on, and
// the type-level migration target is `FrameSource` / `FrameSink`
// directly anyway. Consequence: coercing a non-Send `Box<dyn Stream>`
// (the return shape of `crate::mux::input` / `output`) to
// `Box<dyn FrameSource>` will fail with a `T: Send` trait-bound error.
// The fix on the consumer side is to construct a Send-compliant
// `FrameSource` / `FrameSink` directly rather than relying on this
// bridge for non-Send streams.
//
// Note: `FrameSink` cannot be blanket-impl'd from `Stream` because
// `Stream::finish` takes `&mut self` while `FrameSink::finish` takes
// `Box<Self>`; concrete types will be migrated in a follow-up commit.
#[allow(deprecated)]
impl<T: Stream + Send + ?Sized> FrameSource for T {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
<Self as Stream>::read(self)
}
fn info(&self) -> &crate::disc::DiscTitle {
<Self as Stream>::info(self)
}
fn codec_private(&self, track: usize) -> Option<Vec<u8>> {
<Self as Stream>::codec_private(self, track)
}
fn headers_ready(&self) -> bool {
<Self as Stream>::headers_ready(self)
}
}
/// Wraps any output stream and counts bytes written.
///
/// Progress tracking is a CLI concern — streams don't know their size.
@@ -152,11 +233,13 @@ pub trait Stream {
/// let pct = output.bytes_written() as f64 / total as f64;
/// }
/// ```
#[allow(deprecated)]
pub struct CountingStream {
inner: Box<dyn Stream>,
written: u64,
}
#[allow(deprecated)]
impl CountingStream {
pub fn new(inner: Box<dyn Stream>) -> Self {
Self { inner, written: 0 }
@@ -168,6 +251,7 @@ impl CountingStream {
}
}
#[allow(deprecated)]
impl Stream for CountingStream {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
self.inner.read()
@@ -194,3 +278,178 @@ impl Stream for CountingStream {
self.inner.headers_ready()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::disc::DiscTitle;
fn make_frame(track: usize, pts: i64) -> PesFrame {
PesFrame {
track,
pts,
keyframe: track == 0 && pts == 0,
data: vec![track as u8, (pts & 0xff) as u8, 0xAA],
}
}
/// Direct `FrameSource` impl (not via the deprecated `Stream` blanket).
struct MockSource {
frames: std::vec::IntoIter<PesFrame>,
title: DiscTitle,
}
impl MockSource {
fn new(frames: Vec<PesFrame>) -> Self {
Self {
frames: frames.into_iter(),
title: DiscTitle::empty(),
}
}
}
impl FrameSource for MockSource {
fn read(&mut self) -> std::io::Result<Option<PesFrame>> {
Ok(self.frames.next())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
/// Direct `FrameSink` impl (not via the deprecated `Stream` blanket).
struct MockSink {
collected: Vec<PesFrame>,
title: DiscTitle,
}
impl MockSink {
fn new() -> Self {
Self {
collected: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl FrameSink for MockSink {
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()> {
self.collected.push(frame.clone());
Ok(())
}
fn finish(self: Box<Self>) -> std::io::Result<()> {
// Drop self; in real sinks this is where flush/fsync/close happens.
Ok(())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
/// Variant of `MockSink` whose `finish` returns the collected frames so
/// the test can assert on them after the consuming `Box<Self>` call.
struct CollectingSink {
collected: Vec<PesFrame>,
title: DiscTitle,
}
impl CollectingSink {
fn new() -> Self {
Self {
collected: Vec::new(),
title: DiscTitle::empty(),
}
}
}
impl FrameSink for CollectingSink {
fn write(&mut self, frame: &PesFrame) -> std::io::Result<()> {
self.collected.push(frame.clone());
Ok(())
}
fn finish(self: Box<Self>) -> std::io::Result<()> {
// Real CollectingSink consumers would expose `take()` before
// finish; this trait method just confirms the boxed signature
// compiles and runs.
Ok(())
}
fn info(&self) -> &DiscTitle {
&self.title
}
}
#[test]
fn frame_source_yields_frames_then_eof() {
let frames = vec![make_frame(0, 0), make_frame(1, 1_000), make_frame(0, 2_000)];
let mut src = MockSource::new(frames.clone());
let f0 = src.read().unwrap().expect("first frame");
assert_eq!(f0.track, frames[0].track);
assert_eq!(f0.pts, frames[0].pts);
assert!(f0.keyframe);
let f1 = src.read().unwrap().expect("second frame");
assert_eq!(f1.track, frames[1].track);
assert_eq!(f1.pts, frames[1].pts);
let f2 = src.read().unwrap().expect("third frame");
assert_eq!(f2.track, frames[2].track);
assert_eq!(f2.pts, frames[2].pts);
assert!(src.read().unwrap().is_none());
assert!(src.read().unwrap().is_none()); // idempotent at EOF
}
#[test]
fn frame_sink_collects_then_finishes() {
let mut sink = MockSink::new();
let frames = [make_frame(0, 0), make_frame(1, 100), make_frame(2, 200)];
for f in &frames {
sink.write(f).unwrap();
}
assert_eq!(sink.collected.len(), 3);
assert_eq!(sink.collected[0].pts, 0);
assert_eq!(sink.collected[1].pts, 100);
assert_eq!(sink.collected[2].pts, 200);
// Box-and-finish — the `self: Box<Self>` shape must compile and run.
Box::new(sink).finish().unwrap();
}
#[test]
fn frame_sink_via_dyn_object() {
let frames = [make_frame(0, 0), make_frame(0, 33)];
let mut sink: Box<dyn FrameSink> = Box::new(CollectingSink::new());
for f in &frames {
sink.write(f).unwrap();
}
// info() routes through the trait object.
let _ = sink.info();
sink.finish().unwrap();
}
/// The deprecated blanket impl: any concrete `Stream` should also act as
/// a `FrameSource`. `NullStream` has the simplest constructor of every
/// concrete `Stream` impl in `mux/*`, so it's the smallest credible
/// witness that the bridge compiles and dispatches correctly.
#[test]
#[allow(deprecated)]
fn deprecated_stream_satisfies_frame_source() {
let title = DiscTitle::empty();
let mut null = crate::mux::NullStream::new(&title);
let src: &mut dyn FrameSource = &mut null;
// NullStream::read returns Ok(None) — it's a write-only sink.
assert!(src.read().unwrap().is_none());
// info() forwards through the blanket impl.
let _ = src.info();
assert!(src.headers_ready());
}
}
-82
View File
@@ -1,82 +0,0 @@
//! SectorReader — trait for reading 2048-byte disc sectors.
//!
//! Implemented by Drive (SCSI) and IsoFile (file-backed).
//! Used by UDF parser, disc scanner, label parsers — anything that
//! reads sectors doesn't need to know where they come from.
use crate::error::Result;
/// Read 2048-byte sectors from a disc or disc image.
pub trait SectorReader: Send {
/// Read `count` sectors starting at `lba` into `buf`.
/// `buf` must be at least `count * 2048` bytes.
/// `recovery`: true = full retry/reset loop (ripping), false = single attempt (verify).
/// File-backed readers ignore the flag.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Total capacity in sectors, if known.
fn capacity(&self) -> u32 {
0
}
fn set_speed(&mut self, _kbs: u16) {}
}
/// SectorReader backed by a file (ISO image).
/// Seeks to lba * 2048, reads count * 2048 bytes.
pub struct FileSectorReader {
file: std::io::BufReader<std::fs::File>,
capacity: u32,
}
impl FileSectorReader {
pub fn open(path: &str) -> std::io::Result<Self> {
let file = std::fs::File::open(path)?;
let len = file.metadata()?.len();
let sectors = len / 2048;
if sectors > u32::MAX as u64 {
// ~8 TB hard cap (u32::MAX × 2048 bytes). Path lives in the
// typed Error variant — no English in the message.
return Err(crate::error::Error::IsoTooLarge {
path: path.to_string(),
}
.into());
}
let capacity = sectors as u32;
Ok(Self {
file: std::io::BufReader::with_capacity(4 * 1024 * 1024, file),
capacity,
})
}
}
impl SectorReader for FileSectorReader {
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
use std::io::{Read, Seek, SeekFrom};
let offset = lba as u64 * 2048;
let bytes = count as usize * 2048;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| crate::error::Error::IoError { source: e })?;
self.file
.read_exact(&mut buf[..bytes])
.map_err(|e| crate::error::Error::IoError { source: e })?;
Ok(bytes)
}
fn capacity(&self) -> u32 {
self.capacity
}
}
+200
View File
@@ -0,0 +1,200 @@
//! `DecryptingSectorSource` — wrap any [`SectorSource`] to apply
//! AACS / CSS in-place decryption on every read.
//!
//! This is the 0.18 single-source-of-truth for decrypt-on-read. The
//! actual cipher code lives in [`crate::aacs`] and [`crate::css`];
//! we just call the existing [`crate::decrypt::decrypt_sectors`]
//! helper that already drives both of them. In follow-up commits
//! `sweep_pipeline` and `DiscStream` migrate onto this decorator
//! and delete their duplicate decrypt call sites.
//!
//! Composition: `Drive` → `DecryptingSectorSource` → caller sees
//! plaintext. For `DecryptKeys::None` discs the decorator is a
//! pass-through, so callers can wire it unconditionally and keep
//! their pipeline shape uniform regardless of encryption state.
use crate::decrypt::{DecryptKeys, decrypt_sectors};
use crate::error::Result;
use super::SectorSource;
/// Decorator: read from `inner`, then run the configured
/// AACS / CSS decrypt over the bytes that landed in `buf`.
///
/// `unit_key_idx` selects the AACS unit key for the disc (0 for
/// the vast majority of titles; the rare multi-CPS-unit discs pick
/// the index that covers the title being read). For
/// [`DecryptKeys::None`] and [`DecryptKeys::Css`] the index is
/// ignored.
pub struct DecryptingSectorSource<S: SectorSource> {
inner: S,
keys: DecryptKeys,
unit_key_idx: usize,
}
impl<S: SectorSource> DecryptingSectorSource<S> {
/// Wrap `inner` with the given keys. The default unit-key
/// index is 0; use [`with_unit_key_idx`] for the multi-CPS-unit
/// case.
///
/// [`with_unit_key_idx`]: Self::with_unit_key_idx
pub fn new(inner: S, keys: DecryptKeys) -> Self {
Self {
inner,
keys,
unit_key_idx: 0,
}
}
/// Override the AACS unit-key index. Only meaningful for
/// [`DecryptKeys::Aacs`]; other variants ignore it.
pub fn with_unit_key_idx(mut self, idx: usize) -> Self {
self.unit_key_idx = idx;
self
}
/// Replace the configured keys without unwrapping the decorator.
/// Used by `DiscStream::set_raw()` to flip from encrypted-disc
/// decryption to a pass-through after the inner reader is already
/// owned by the wrapper. For new construction prefer [`new`].
///
/// [`new`]: Self::new
pub fn set_keys(&mut self, keys: DecryptKeys) {
self.keys = keys;
}
/// Borrow the inner source. Useful for tests and for adapters
/// that want to introspect the underlying drive / file without
/// unwrapping the decorator.
pub fn inner(&self) -> &S {
&self.inner
}
/// Mutable borrow of the inner source.
pub fn inner_mut(&mut self) -> &mut S {
&mut self.inner
}
/// Consume the decorator and return the underlying source.
pub fn into_inner(self) -> S {
self.inner
}
}
impl<S: SectorSource> SectorSource for DecryptingSectorSource<S> {
fn capacity_sectors(&self) -> u32 {
self.inner.capacity_sectors()
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
let n = self.inner.read_sectors(lba, count, buf, recovery)?;
// Reuse the existing crate-wide decrypt entry point — same
// path the 0.17 sweep_pipeline and DiscStream call, so we
// inherit their AACS / CSS / None semantics verbatim. The
// helper is a no-op for DecryptKeys::None.
decrypt_sectors(&mut buf[..n], &self.keys, self.unit_key_idx)?;
Ok(n)
}
fn set_speed(&mut self, kbs: u16) {
self.inner.set_speed(kbs)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::Result;
/// Synthetic SectorSource that yields a deterministic byte
/// pattern keyed by LBA. Used to verify the decorator's
/// pass-through behaviour for `DecryptKeys::None`.
struct PatternedSource {
capacity: u32,
}
impl PatternedSource {
fn fill(lba: u32, count: u16, buf: &mut [u8]) {
let bytes = count as usize * 2048;
for (i, slot) in buf[..bytes].iter_mut().enumerate() {
let abs = lba as u64 * 2048 + i as u64;
*slot = ((abs.wrapping_mul(2654435761) >> 16) & 0xff) as u8;
}
}
}
impl SectorSource for PatternedSource {
fn capacity_sectors(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Self::fill(lba, count, buf);
Ok(count as usize * 2048)
}
}
#[test]
fn passthrough_with_no_keys() {
let src = PatternedSource { capacity: 16 };
let mut wrapped = DecryptingSectorSource::new(src, DecryptKeys::None);
// capacity_sectors delegates.
assert_eq!(wrapped.capacity_sectors(), 16);
let mut got = vec![0u8; 4 * 2048];
let n = wrapped.read_sectors(3, 4, &mut got, false).unwrap();
assert_eq!(n, 4 * 2048);
let mut expected = vec![0u8; 4 * 2048];
PatternedSource::fill(3, 4, &mut expected);
assert_eq!(got, expected);
}
#[test]
fn passthrough_set_speed_delegates() {
struct SpeedRecorder {
last: Option<u16>,
}
impl SectorSource for SpeedRecorder {
fn capacity_sectors(&self) -> u32 {
0
}
fn read_sectors(
&mut self,
_lba: u32,
_count: u16,
_buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
Ok(0)
}
fn set_speed(&mut self, kbs: u16) {
self.last = Some(kbs);
}
}
let mut wrapped =
DecryptingSectorSource::new(SpeedRecorder { last: None }, DecryptKeys::None);
wrapped.set_speed(7200);
assert_eq!(wrapped.inner().last, Some(7200));
}
// TODO: AACS round-trip test — needs a fixture-encrypted unit
// (6144-byte aligned) plus the matching unit key. The cipher
// path itself is exercised by `crate::aacs` unit tests; here
// we only assert the decorator wires the existing helper, not
// that AES-128 is correct.
}
+243
View File
@@ -0,0 +1,243 @@
//! File-backed sector I/O — read and write 2048-byte sectors against
//! an ISO image on disk.
//!
//! [`FileSectorSource`] is the read side (open-only). [`FileSectorSink`]
//! is the write side (create or open-rw); writes go through
//! [`crate::io::WritebackFile`] so big sequential ISO writes share
//! the same bounded-cache writeback pipeline used by sweep / patch /
//! mux.
use std::fs::{File, OpenOptions};
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::Path;
use crate::error::{Error, Result};
use super::{SectorReader, SectorSink};
/// SectorSource backed by a file (ISO image).
///
/// Seeks to `lba * 2048`, reads `count * 2048` bytes per call. The
/// file is held directly: every `read_sectors` call performs an
/// absolute seek, so a wrapping `BufReader` would have its buffer
/// invalidated on every call (its internal cursor moves with the
/// `Seek` impl) — pure overhead. Callers that benefit from buffered
/// reads should compose their own `BufReader` at the `read_sectors`
/// granularity they care about.
pub struct FileSectorSource {
file: File,
capacity: u32,
}
impl FileSectorSource {
/// Open an existing ISO file for reading. Capacity is derived
/// from `metadata().len() / 2048`. Returns
/// [`Error::IsoTooLarge`] if the file would exceed the 32-bit
/// LBA address space (~8 TB).
pub fn open(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let len = file.metadata()?.len();
let sectors = len / 2048;
if sectors > u32::MAX as u64 {
return Err(Error::IsoTooLarge {
path: path.to_string_lossy().into_owned(),
}
.into());
}
let capacity = sectors as u32;
Ok(Self { file, capacity })
}
}
// Implement the legacy `SectorReader` trait. The blanket impl in
// `super` produces the `SectorSource` impl automatically — no need
// to write both, and writing both would conflict. This keeps the
// 0.17 method-resolution path intact (callers with `SectorReader`
// in scope can still write `fsr.read_sectors(..)` against a
// `FileSectorSource`).
impl SectorReader for FileSectorSource {
fn capacity(&self) -> u32 {
self.capacity
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
_recovery: bool,
) -> Result<usize> {
let offset = lba as u64 * 2048;
let bytes = count as usize * 2048;
self.file
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.file
.read_exact(&mut buf[..bytes])
.map_err(|e| Error::IoError { source: e })?;
Ok(bytes)
}
}
/// SectorSink backed by a file (ISO image).
///
/// Writes go through [`crate::io::WritebackFile`], which on Linux drives
/// continuous `sync_file_range` + `posix_fadvise(DONTNEED)` to keep
/// the kernel dirty page cache bounded during multi-GB sequential
/// writes. macOS / Windows fall through to a no-op pipeline.
///
/// `finish` runs `sync_all` before dropping the underlying file.
pub struct FileSectorSink {
inner: crate::io::WritebackFile,
}
impl FileSectorSink {
/// Create a new ISO file at `path`, truncating any existing
/// file. The file is opened read-write so the same handle can
/// later be reused for verification reads if needed (sweep
/// doesn't, but it costs nothing here).
pub fn create(path: &Path) -> std::io::Result<Self> {
let file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(path)?;
Ok(Self {
inner: crate::io::WritebackFile::new(file)?,
})
}
/// Open an existing ISO file for in-place updates (e.g. patch
/// pass writing recovered sectors over zero-filled holes).
/// Does not truncate.
pub fn open(path: &Path) -> std::io::Result<Self> {
let file = OpenOptions::new().read(true).write(true).open(path)?;
Ok(Self {
inner: crate::io::WritebackFile::new(file)?,
})
}
}
impl SectorSink for FileSectorSink {
fn write_sectors(&mut self, lba: u32, buf: &[u8]) -> Result<()> {
debug_assert!(
buf.len() % 2048 == 0,
"FileSectorSink::write_sectors: buf len {} not a multiple of 2048",
buf.len()
);
let offset = lba as u64 * 2048;
self.inner
.seek(SeekFrom::Start(offset))
.map_err(|e| Error::IoError { source: e })?;
self.inner
.write_all(buf)
.map_err(|e| Error::IoError { source: e })?;
Ok(())
}
fn finish(mut self: Box<Self>) -> Result<()> {
self.inner
.sync_all()
.map_err(|e| Error::IoError { source: e })?;
Ok(())
}
}
#[cfg(test)]
mod tests {
// Bring the 0.18 trait into scope (not super::*: the super
// module also re-exports the legacy `SectorReader`, and
// having both `SectorReader::read_sectors` and
// `SectorSource::read_sectors` visible would force every
// call site to disambiguate). External consumers see the
// same surface this test exercises.
use super::{FileSectorSink, FileSectorSource};
use crate::sector::{SectorSink, SectorSource};
use tempfile::tempdir;
#[test]
fn round_trip_single_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("rt.iso");
let mut sink = FileSectorSink::create(&path).unwrap();
// Pre-extend the file to 4 sectors of zeros so we can write
// sector 2 in place. Easiest way: write zeros first.
let zeros = [0u8; 4 * 2048];
sink.write_sectors(0, &zeros).unwrap();
let mut payload = [0u8; 2048];
for (i, b) in payload.iter_mut().enumerate() {
*b = (i as u8).wrapping_mul(17);
}
sink.write_sectors(2, &payload).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
let mut got = [0u8; 2048];
let n = src.read_sectors(2, 1, &mut got, false).unwrap();
assert_eq!(n, 2048);
assert_eq!(got, payload);
// Sectors 0,1,3 still zero.
let mut z = [0xffu8; 2048];
src.read_sectors(0, 1, &mut z, false).unwrap();
assert!(z.iter().all(|b| *b == 0));
}
#[test]
fn round_trip_multi_sector() {
let dir = tempdir().unwrap();
let path = dir.path().join("multi.iso");
let mut sink = FileSectorSink::create(&path).unwrap();
let mut payload = vec![0u8; 8 * 2048];
for (i, b) in payload.iter_mut().enumerate() {
*b = ((i * 31) ^ (i >> 7)) as u8;
}
sink.write_sectors(0, &payload).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 8);
let mut got = vec![0u8; 8 * 2048];
let n = src.read_sectors(0, 8, &mut got, false).unwrap();
assert_eq!(n, 8 * 2048);
assert_eq!(got, payload);
}
#[test]
fn open_existing_does_not_truncate() {
let dir = tempdir().unwrap();
let path = dir.path().join("open.iso");
// Create with 4 sectors of pattern A.
let mut sink = FileSectorSink::create(&path).unwrap();
let pat_a = [0xaau8; 4 * 2048];
sink.write_sectors(0, &pat_a).unwrap();
Box::new(sink).finish().unwrap();
// Reopen and overwrite sector 1 only.
let mut sink = FileSectorSink::open(&path).unwrap();
let pat_b = [0xbbu8; 2048];
sink.write_sectors(1, &pat_b).unwrap();
Box::new(sink).finish().unwrap();
let mut src = FileSectorSource::open(&path).unwrap();
assert_eq!(src.capacity_sectors(), 4);
let mut got = [0u8; 2048];
src.read_sectors(0, 1, &mut got, false).unwrap();
assert_eq!(got, [0xaau8; 2048]);
src.read_sectors(1, 1, &mut got, false).unwrap();
assert_eq!(got, [0xbbu8; 2048]);
src.read_sectors(2, 1, &mut got, false).unwrap();
assert_eq!(got, [0xaau8; 2048]);
}
}
+202
View File
@@ -0,0 +1,202 @@
//! Sector-level I/O traits.
//!
//! 0.18 splits the unidirectional read trait from a write trait at
//! the sector layer, so the type system catches "wrong direction"
//! mistakes at compile time instead of runtime. See
//! `(internal)/memory/0_18_redesign.md`.
//!
//! - [`SectorSource`] reads 2048-byte sectors. Implemented by
//! `Drive` (via the legacy [`SectorReader`] alias) and
//! [`FileSectorSource`] (ISO-backed).
//! - [`SectorSink`] writes 2048-byte sectors. Implemented by
//! [`FileSectorSink`] (ISO-backed) and, in later commits, by
//! sweep/patch consumer adapters.
//! - [`DecryptingSectorSource`] is a decorator that wraps any
//! `SectorSource` and applies the existing AACS / CSS in-place
//! decrypt to plaintext-out.
//!
//! [`SectorReader`] is the 0.17 read trait. It stays on through
//! the 0.18 migration window so existing call sites
//! (`Drive`, `IsoSectorReader`, `BufferedSectorReader`,
//! `DiscStream`, `verify`) compile unchanged. A blanket impl
//! forwards every `SectorReader` impl to `SectorSource`, so new
//! code should target `SectorSource` / `SectorSink` directly. The
//! formal `#[deprecated]` attribute lands once the internal
//! callers have migrated; see the comment on `SectorReader` for
//! why this commit holds it back.
pub mod decrypting;
pub mod file;
use crate::error::Result;
/// Read 2048-byte sectors from a disc, image, or composed source.
///
/// Direction-typed: a `SectorSource` cannot be written to. Wrap the
/// inner source in [`DecryptingSectorSource`] to get plaintext
/// sectors out of an encrypted disc.
pub trait SectorSource: Send {
/// Total capacity in sectors, if known. Returns 0 when unknown
/// (e.g. live drives that haven't completed `READ CAPACITY` yet).
fn capacity_sectors(&self) -> u32;
/// Read `count` sectors starting at `lba` into `buf`.
/// `buf` must be at least `count * 2048` bytes.
/// `recovery`: true = full retry/reset loop (ripping),
/// false = single attempt (verify). File-backed sources ignore
/// the flag.
///
/// Returns the number of bytes written into `buf` on success.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Optional speed control for sources that map to a physical
/// drive. No-op for everything else.
fn set_speed(&mut self, _kbs: u16) {}
}
/// Write 2048-byte sectors to a disc image or composed sink.
///
/// Direction-typed: a `SectorSink` cannot be read from. The
/// terminal [`finish`] takes `Box<Self>` so it can run on `dyn
/// SectorSink` and consume the sink (`fsync` + close).
///
/// [`finish`]: SectorSink::finish
pub trait SectorSink: Send {
/// Write the sectors in `buf` starting at `lba`. `buf.len()`
/// must be a multiple of 2048; the implementation seeks to
/// `lba * 2048` before writing.
fn write_sectors(&mut self, lba: u32, buf: &[u8]) -> Result<()>;
/// Flush, fsync, and close. Consumes the sink. Always called
/// last; subsequent operations are not defined.
fn finish(self: Box<Self>) -> Result<()>;
}
/// 0.17 read trait. Slated for removal once internal call sites
/// migrate to [`SectorSource`] in follow-up commits; until then
/// it remains the trait that `Drive`, `IsoSectorReader`,
/// `BufferedSectorReader`, and existing `&mut dyn SectorReader`
/// signatures use unchanged.
///
/// New code should implement [`SectorSource`] directly. The
/// blanket impl below makes any `SectorReader` automatically
/// usable wherever a `SectorSource` is expected, so a one-way
/// migration off `SectorReader` is possible per-callsite without
/// touching the impls.
//
// NOTE: not marked `#[deprecated]` in this commit — `cargo clippy
// -- -D warnings` (the CI gauntlet) treats deprecation as an
// error, and the existing `Drive` / `udf::BufferedSectorReader` /
// `mux::DiscStream` / `verify` call sites all go through this
// trait. The deprecation attribute lands together with the
// migration commits that move those call sites to
// `SectorSource`. The behavioural contract — "this trait is
// going away in 0.18" — is documented above and tracked in
// `(internal)/memory/0_18_redesign.md`.
pub trait SectorReader: Send {
/// Read `count` sectors starting at `lba` into `buf`.
/// See [`SectorSource::read_sectors`] for semantics.
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize>;
/// Total capacity in sectors, if known.
fn capacity(&self) -> u32 {
0
}
fn set_speed(&mut self, _kbs: u16) {}
}
// Blanket impl: anything implementing the legacy `SectorReader`
// trait automatically satisfies `SectorSource`. This is what keeps
// existing impls (`Drive`, `IsoSectorReader`, `BufferedSectorReader`,
// etc.) compiling without source changes during the migration. The
// reverse direction (impl SectorReader for SectorSource) is
// intentionally NOT provided — new code targets the new trait.
impl<T: SectorReader + ?Sized> SectorSource for T {
fn capacity_sectors(&self) -> u32 {
<T as SectorReader>::capacity(self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<T as SectorReader>::read_sectors(self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<T as SectorReader>::set_speed(self, kbs)
}
}
// Forwarding impls so callers can wrap `&mut dyn SectorReader` /
// `Box<dyn SectorReader>` in [`DecryptingSectorSource`] without
// having to unbox or re-borrow inside the lib's hot paths. The
// generic `&mut T` / `Box<T>` blankets would conflict with the
// `SectorReader → SectorSource` blanket above (a downstream crate
// could `impl SectorReader for &mut U`); the specific
// `dyn SectorReader` instantiations are unambiguous because
// `SectorReader` is the very trait whose `dyn` we're targeting.
impl SectorSource for &mut (dyn SectorReader + '_) {
fn capacity_sectors(&self) -> u32 {
<dyn SectorReader as SectorReader>::capacity(*self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<dyn SectorReader as SectorReader>::read_sectors(*self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<dyn SectorReader as SectorReader>::set_speed(*self, kbs)
}
}
impl SectorSource for Box<dyn SectorReader> {
fn capacity_sectors(&self) -> u32 {
<dyn SectorReader as SectorReader>::capacity(&**self)
}
fn read_sectors(
&mut self,
lba: u32,
count: u16,
buf: &mut [u8],
recovery: bool,
) -> Result<usize> {
<dyn SectorReader as SectorReader>::read_sectors(&mut **self, lba, count, buf, recovery)
}
fn set_speed(&mut self, kbs: u16) {
<dyn SectorReader as SectorReader>::set_speed(&mut **self, kbs)
}
}
pub use decrypting::DecryptingSectorSource;
pub use file::{FileSectorSink, FileSectorSource};
// Backwards-compat alias for the public API. `FileSectorReader` is
// the 0.17 name; new code uses `FileSectorSource`. Both point at
// the same type. The `#[deprecated]` attribute lands together with
// the migration commits that retire the alias from internal uses.
pub type FileSectorReader = FileSectorSource;
+6 -1
View File
@@ -1,6 +1,11 @@
//! Integration tests for progress reporting, halt behavior, drop safety,
//! and the file-backed sector reader round trip.
// 0.18 trait split: this suite still drives the deprecated `pes::Stream`
// trait directly. It will be migrated to `FrameSource`/`FrameSink` in the
// follow-up that ports concrete impls.
#![allow(deprecated)]
use libfreemkv::disc::{CopyOptions, DiscRegion};
use libfreemkv::error::Result;
use libfreemkv::pes::Stream as PesStream;
@@ -319,7 +324,7 @@ fn test_file_sector_reader_round_trip() {
tmp.write_all(&data).expect("write data");
tmp.flush().expect("flush");
let path = tmp.path().to_str().expect("path utf-8").to_string();
let path = tmp.path().to_path_buf();
let mut fsr = FileSectorReader::open(&path).expect("open FileSectorReader");
assert_eq!(fsr.capacity(), N_SECTORS as u32, "capacity mismatch");
+111
View File
@@ -286,3 +286,114 @@ fn patch_recovers_multiple_good_middles() {
pr.bytes_total,
);
}
/// 0.18 Pass N pipeline split: exercises the new producer/consumer
/// path end-to-end on a synthetic patterned reader. Bad range layout
/// is small (5 bad LBAs surrounded by good middle) so the producer
/// emits a mix of `Recovered` and `Unreadable` items and the consumer
/// thread must apply both kinds. Verifies:
///
/// - `bytes_good` advances (good sectors flow producer→consumer→file
/// →mapfile with the data preserved).
/// - The recovered LBAs end up Finished; the bad LBAs end up Unreadable.
/// - Bytes written at the recovered offsets match what the producer
/// read from the patterned source (proves the channel hand-off
/// didn't drop or reorder buffers, and the consumer's seek+write
/// landed at the right offsets).
#[test]
fn patch_pipeline_split_recovers_and_records_correctly() {
let capacity_sectors: u32 = 512;
let total_bytes: u64 = capacity_sectors as u64 * SECTOR_SIZE as u64;
// Layout: LBAs 200-204 inclusive are bad (5 sectors), 205-249 good.
// The pre-existing range is LBAs 200-249 NonTrimmed (100 KB).
let mut bad_lbas = HashSet::new();
for lba in 200..205 {
bad_lbas.insert(lba);
}
let (mut reader, _trace) = PatternedSectorReader::new(capacity_sectors, bad_lbas.clone());
let disc = synthetic_disc(capacity_sectors);
let tmp = tempfile::NamedTempFile::new().unwrap();
let iso_path = tmp.path().to_path_buf();
drop(tmp);
let finished = [
(0, 200 * 2048),
(250 * 2048, (capacity_sectors as u64 - 250) * 2048),
];
let nontrimmed = [(200 * 2048, 50 * 2048)];
prep_iso_and_mapfile(&iso_path, total_bytes, &finished, &nontrimmed);
let opts = CopyOptions {
decrypt: false,
multipass: true,
..Default::default()
};
let pr = disc
.copy(&mut reader, &iso_path, &opts)
.expect("copy returns Ok");
// Bytes_good_total should advance — the good LBAs in the bad range
// (205-249, 45 sectors) are all reachable via per-sector retry.
// Initial bytes_good = 200 * 2048 + (512-250) * 2048 = 462 sectors.
// After patch, bytes_good should be ≥ 462 + 45 = 507 sectors worth.
let initial_good_sectors: u64 = 200 + (capacity_sectors as u64 - 250);
let min_expected_good_bytes = (initial_good_sectors + 30) * 2048;
assert!(
pr.bytes_good >= min_expected_good_bytes,
"patch should have recovered most good LBAs in the bad range via the pipeline. \
bytes_good={} (expected ≥ {}); bytes_total={}",
pr.bytes_good,
min_expected_good_bytes,
pr.bytes_total,
);
// Verify the mapfile records: every good LBA is Finished, every
// bad LBA is either Unreadable or NonTrimmed (not Finished).
let map_path = libfreemkv::disc::mapfile_path_for(&iso_path);
let map = Mapfile::load(&map_path).unwrap();
let finished_ranges = map.ranges_with(&[SectorStatus::Finished]);
let in_finished = |lba: u32| -> bool {
let pos = lba as u64 * 2048;
finished_ranges
.iter()
.any(|&(p, sz)| pos >= p && pos < p + sz)
};
for lba in 205..250 {
assert!(
in_finished(lba),
"good LBA {lba} should be Finished after pipeline patch run"
);
}
for lba in 200..205 {
assert!(
!in_finished(lba),
"bad LBA {lba} should NOT be Finished after pipeline patch run"
);
}
// Verify the consumer wrote the producer's bytes at the right
// offsets. PatternedSectorReader fills each sector with `(lba & 0xff)
// as u8` — picking LBA 220 (well inside the recovered region) gives
// a clean signature byte to check.
use std::io::{Read, Seek, SeekFrom};
let mut iso = std::fs::File::open(&iso_path).unwrap();
iso.seek(SeekFrom::Start(220 * 2048)).unwrap();
let mut sector = [0u8; 2048];
iso.read_exact(&mut sector).unwrap();
let expected_byte = (220u32 & 0xff) as u8;
let _ = std::fs::remove_file(&iso_path);
let _ = std::fs::remove_file(&map_path);
assert!(
sector.iter().all(|&b| b == expected_byte),
"consumer should have written PatternedSectorReader's pattern \
(byte {expected_byte:#x} for LBA 220) to the recovered offset; \
got first 8 bytes = {:?}",
&sector[..8]
);
}
+25 -19
View File
@@ -1,5 +1,10 @@
//! Integration tests for the PES stream pipeline.
// 0.18 trait split: this suite still drives the deprecated `pes::Stream`
// trait directly. It will be migrated to `FrameSource`/`FrameSink` in the
// follow-up that ports concrete impls.
#![allow(deprecated)]
use libfreemkv::mux::meta::M2tsMeta;
use libfreemkv::pes::Stream as PesStream;
use libfreemkv::*;
@@ -302,12 +307,12 @@ fn m2ts_stream_write_read() {
keyframe: i == 0,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
// Verify the info is correct
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 4);
assert_eq!(info.duration_secs, 7200.0);
}
@@ -356,7 +361,7 @@ fn m2ts_read_returns_error_on_write_stream() {
let dt = sample_disc_title();
let output = Cursor::new(Vec::new());
let mut stream = M2tsStream::create(output, &dt).unwrap();
assert!(stream.read().is_err());
assert!(PesStream::read(&mut stream).is_err());
}
// ── DiscTitle::empty ──────────────────────────────────────────
@@ -578,7 +583,7 @@ fn meta_all_stream_types() {
#[test]
fn mkvstream_write_finish() {
let dt = sample_disc_title();
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let mut stream = MkvStream::create(writer, &dt).unwrap();
// Write some fake PES frames (they won't produce valid MKV content
@@ -590,20 +595,20 @@ fn mkvstream_write_finish() {
keyframe: i == 0,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
// finish should not panic even without valid codec data
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
}
#[test]
fn mkvstream_meta_sets_title() {
let dt = sample_disc_title();
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let stream = MkvStream::create(writer, &dt).unwrap();
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.playlist, "Test Movie");
assert_eq!(info.duration_secs, 7200.0);
assert_eq!(info.streams.len(), 4);
@@ -640,7 +645,7 @@ fn mkvstream_roundtrip_bdts() {
codec_privates: Vec::new(),
};
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let mut stream = MkvStream::create(writer, &dt).unwrap();
// Write PES frames targeting the audio track
@@ -651,13 +656,13 @@ fn mkvstream_roundtrip_bdts() {
keyframe: true,
data: vec![i; 100],
};
stream.write(&frame).unwrap();
PesStream::write(&mut stream, &frame).unwrap();
}
stream.finish().unwrap();
PesStream::finish(&mut stream).unwrap();
// Verify the info is correct
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 1);
assert_eq!(info.playlist, "Audio Only");
}
@@ -724,10 +729,10 @@ fn mkvstream_meta_preserves_all_streams() {
codec_privates: Vec::new(),
};
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(Cursor::new(Vec::new()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> = Box::new(Cursor::new(Vec::new()));
let stream = MkvStream::create(writer, &dt).unwrap();
let info = stream.info();
let info = PesStream::info(&stream);
assert_eq!(info.streams.len(), 5, "all 5 streams should be preserved");
assert_eq!(info.playlist, "Stream Test");
assert_eq!(info.duration_secs, 3600.0);
@@ -832,7 +837,8 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
}
}
let writer: Box<dyn libfreemkv::mux::WriteSeek> = Box::new(SharedWriter(output2.clone()));
let writer: Box<dyn libfreemkv::mux::WriteSeek + Send> =
Box::new(SharedWriter(output2.clone()));
let mut stream2 = MkvStream::create(writer, &dt).unwrap();
// Write the ES data (SPS+PPS+IDR) as a keyframe PES frame.
@@ -842,7 +848,7 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
keyframe: true,
data: es_data,
};
stream2.write(&frame1).unwrap();
PesStream::write(&mut stream2, &frame1).unwrap();
// Write a second non-IDR frame
let frame2 = libfreemkv::pes::PesFrame {
@@ -851,8 +857,8 @@ fn mkvstream_e2e_h264_produces_valid_mkv() {
keyframe: false,
data: es_data2,
};
stream2.write(&frame2).unwrap();
stream2.finish().unwrap();
PesStream::write(&mut stream2, &frame2).unwrap();
PesStream::finish(&mut stream2).unwrap();
let data = output2.lock().unwrap().clone().into_inner();