MattJackson
f98f07b2d3
Sweep was the original producer/consumer split that motivated the
generic Pipeline primitive (round 1, commit 198268b). Now that
Pipeline + Sink exist, sweep stops shipping its own bespoke
threading.
- New SweepSink: Sink<WorkItem> impl in src/disc/sweep.rs. Owns
WritebackFile + Mapfile + ProgressSnapshot back-channel. apply()
carries the file-write + mapfile.record per WorkItem; close()
drains writeback, fsyncs, flushes mapfile.
- Disc::sweep: constructs SweepSink, calls Pipeline::spawn_named
(so the consumer thread keeps showing up as
freemkv-sweep-consumer), sends WorkItems, calls pipe.finish().
The producer-side ReadCtx state machine, decrypt, set_speed,
halt — all unchanged.
- Pipeline gains spawn_named(name, depth, sink) so callers can
preserve identifiable thread names without the primitive baking
one in. Also adds Pipeline::try_send for the throttled
StatsRequest path that must not block the producer.
- Deleted src/disc/sweep_pipeline.rs entirely. WorkItem,
ProgressSnapshot, ConsumerSummary moved into disc/sweep.rs as
module-private types. WorkItem::Finish dropped — dropping the
channel is the end-of-stream signal Pipeline already uses.
Behaviour-preserving: the sweep algorithm, mapfile invariants,
back-pressure via channel depth (DEFAULT_PIPELINE_DEPTH = 4) all
match the 0.17.13 implementation. New synthetic regression test
(sweep_pipeline_full_good_100_batches) exercises ~100 batches of
clean reads end-to-end through the new Pipeline path and verifies
bytes_good and ISO file size.
See (internal)/memory/0_18_redesign.md.
32 lines
1.4 KiB
Rust
32 lines
1.4 KiB
Rust
//! File I/O helpers that bound kernel cache pressure on big writes.
|
|
//!
|
|
//! `WritebackFile` is a drop-in wrapper around `std::fs::File` for any
|
|
//! call site that performs large sequential writes (sweep, patch, mux,
|
|
//! etc.). It implements `Write` and `Seek` so existing code paths can
|
|
//! swap `File` for `WritebackFile` with no body changes. Internally it
|
|
//! drives a `WritebackPipeline` that, on Linux, drains dirty pages
|
|
//! continuously at 32 MB granularity to avoid the kernel's
|
|
//! accumulate-then-burst flush behaviour. macOS and Windows use a
|
|
//! no-op pipeline — their default cache policies have not been shown
|
|
//! to exhibit the same pathology for this access pattern.
|
|
//!
|
|
//! `Pipeline` + `Sink` (0.18) is the generic producer/consumer primitive
|
|
//! used by sweep, patch, and mux to overlap reads with writes via a
|
|
//! bounded channel + dedicated consumer thread.
|
|
|
|
mod writeback;
|
|
mod writeback_file;
|
|
|
|
pub mod pipeline;
|
|
|
|
pub(crate) use writeback_file::WritebackFile;
|
|
|
|
// Re-exports for the 0.18 redesign. Sweep is wired up in
|
|
// `disc/sweep.rs`; patch and mux migrate in later 0.18 slices.
|
|
// `WRITE_THROUGH_DEPTH` is patch-only and currently has no in-tree
|
|
// caller — the targeted `#[allow]` keeps the re-export visible
|
|
// without dragging the rest of the module under `dead_code`.
|
|
#[allow(unused_imports)]
|
|
pub use pipeline::WRITE_THROUGH_DEPTH;
|
|
pub use pipeline::{DEFAULT_PIPELINE_DEPTH, Flow, Pipeline, Sink};
|