recovery: decayed (EWMA) handler scorecard so the ranking flips
Grade handlers by an exponentially-decayed recovery rate (bytes/second, alpha=0.5) instead of a cumulative rate. The residual hardens mid-pass, so the best technique changes: cumulative froze the early winner in the lead forever; the EWMA re-prices continuously — a handler that stops earning decays down, a late-starting specialist climbs. Keeps rank()'s untried->top one-shot calibration and attempted-but-zero-time->bottom. New flip fixture proves a handler that recovers a lot early then nothing loses its lead to one that starts recovering later.
This commit is contained in:
+89
-20
@@ -722,12 +722,25 @@ impl SectionHandler for Jump {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per-rip handler scorecard. Grades each handler by the recovery RATE it has
|
/// EWMA smoothing factor for the decayed recovery rate. Each new attempt is
|
||||||
/// achieved so far (bytes recovered per second of wall time) so the coordinator
|
/// weighted `α`, the running average `1-α`, so a handler's score tracks its
|
||||||
/// runs the best-performing handler FIRST on later sections and lets a proven
|
/// RECENT performance and forgets its distant past at a rate set by `α`. Higher
|
||||||
/// dud fall to the back. Ephemeral — reset each rip, no persistence. A handler
|
/// = more reactive (leadership flips sooner); lower = steadier. 0.5 halves the
|
||||||
/// not yet tried ranks top (`u64::MAX`) so every handler is calibrated once
|
/// weight of the previous score on every attempt — reactive enough that a proven
|
||||||
/// before the ranking narrows to the winners ("try each quick, then prioritise").
|
/// early winner whose territory is exhausted decays out of the lead within a few
|
||||||
|
/// barren attempts, while a late-starting specialist climbs as it earns.
|
||||||
|
const SCORE_EWMA_ALPHA: f64 = 0.5;
|
||||||
|
|
||||||
|
/// Per-rip handler scorecard. Grades each handler by a DECAYED recovery rate (an
|
||||||
|
/// EWMA of bytes-recovered-per-second, [`SCORE_EWMA_ALPHA`]) so the coordinator
|
||||||
|
/// runs whoever is winning *now* FIRST on later sections. The residual shrinks
|
||||||
|
/// and hardens mid-pass, so the best technique CHANGES: the fast scouts clean
|
||||||
|
/// the range-fronts, then the leftovers are exactly the marginal sectors where
|
||||||
|
/// the specialists win — and the ranking must FLIP. A cumulative rate froze the
|
||||||
|
/// early winner in the lead forever; the EWMA re-prices continuously — a handler
|
||||||
|
/// that stops earning decays down, one that starts earning climbs. Ephemeral —
|
||||||
|
/// reset each rip, no persistence. A handler not yet tried ranks top
|
||||||
|
/// (`u64::MAX`) so every handler is calibrated once before the ranking narrows.
|
||||||
#[derive(Default)]
|
#[derive(Default)]
|
||||||
pub(super) struct HandlerScoreboard {
|
pub(super) struct HandlerScoreboard {
|
||||||
stats: std::collections::HashMap<String, ScoreStat>,
|
stats: std::collections::HashMap<String, ScoreStat>,
|
||||||
@@ -735,26 +748,41 @@ pub(super) struct HandlerScoreboard {
|
|||||||
|
|
||||||
#[derive(Default, Clone, Copy)]
|
#[derive(Default, Clone, Copy)]
|
||||||
struct ScoreStat {
|
struct ScoreStat {
|
||||||
|
/// Decayed recovery rate (bytes/second), the ranking signal. `None` until
|
||||||
|
/// the first attempt that spent measurable time (a zero-elapsed call proves
|
||||||
|
/// no rate). Seeded to the first timed sample, then EWMA'd.
|
||||||
|
ewma_rate: Option<f64>,
|
||||||
|
// Cumulative totals — for the operator log line only, NOT for ranking.
|
||||||
recovered: u64,
|
recovered: u64,
|
||||||
nanos: u128,
|
nanos: u128,
|
||||||
attempts: u64,
|
attempts: u64,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl HandlerScoreboard {
|
impl HandlerScoreboard {
|
||||||
fn rate(s: &ScoreStat) -> u64 {
|
/// Fold one timed sample (bytes/second) into the decayed rate.
|
||||||
if s.nanos == 0 {
|
fn decay(prev: Option<f64>, sample: f64) -> f64 {
|
||||||
0
|
match prev {
|
||||||
} else {
|
None => sample,
|
||||||
((s.recovered as u128 * 1_000_000_000) / s.nanos).min(u64::MAX as u128) as u64
|
Some(p) => SCORE_EWMA_ALPHA * sample + (1.0 - SCORE_EWMA_ALPHA) * p,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Record one attempt: bytes recovered over `elapsed`.
|
/// Record one attempt: `recovered` bytes over `elapsed`. A timed attempt
|
||||||
|
/// (elapsed > 0) decays a fresh bytes/second sample into `ewma_rate` — a
|
||||||
|
/// barren attempt (recovered = 0) contributes a 0 sample that decays the
|
||||||
|
/// score DOWN, which is exactly what lets an exhausted early winner lose its
|
||||||
|
/// lead. A zero-elapsed call (handler yielded before any timed read)
|
||||||
|
/// contributes no rate sample.
|
||||||
fn record(&mut self, name: &str, recovered: u64, elapsed: std::time::Duration) {
|
fn record(&mut self, name: &str, recovered: u64, elapsed: std::time::Duration) {
|
||||||
let e = self.stats.entry(name.to_string()).or_default();
|
let e = self.stats.entry(name.to_string()).or_default();
|
||||||
e.recovered = e.recovered.saturating_add(recovered);
|
e.recovered = e.recovered.saturating_add(recovered);
|
||||||
e.nanos = e.nanos.saturating_add(elapsed.as_nanos());
|
e.nanos = e.nanos.saturating_add(elapsed.as_nanos());
|
||||||
e.attempts += 1;
|
e.attempts += 1;
|
||||||
|
let secs = elapsed.as_secs_f64();
|
||||||
|
if secs > 0.0 {
|
||||||
|
let sample = recovered as f64 / secs;
|
||||||
|
e.ewma_rate = Some(Self::decay(e.ewma_rate, sample));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Ranking key (higher runs earlier). Untried → top, so it gets calibrated.
|
/// Ranking key (higher runs earlier). Untried → top, so it gets calibrated.
|
||||||
@@ -762,13 +790,14 @@ impl HandlerScoreboard {
|
|||||||
match self.stats.get(name) {
|
match self.stats.get(name) {
|
||||||
// Never attempted → top, so every handler is calibrated once.
|
// Never attempted → top, so every handler is calibrated once.
|
||||||
None => u64::MAX,
|
None => u64::MAX,
|
||||||
// Attempted but recorded no measurable time — e.g. it returned
|
// Attempted but no timed sample yet — e.g. it returned `Halted` on
|
||||||
// `Halted` on its first check or did zero reads. It proved nothing,
|
// its first check or did zero reads. It proved nothing, so rank it at
|
||||||
// so rank it at the BOTTOM (0), not the top: otherwise a called-but-
|
// the BOTTOM (0), not the top: otherwise a called-but-idle handler
|
||||||
// idle handler perpetually crowds out proven performers. (An entry
|
// perpetually crowds out proven performers.
|
||||||
// exists only after `record`, so `Some` always means attempts ≥ 1.)
|
Some(s) => match s.ewma_rate {
|
||||||
Some(s) if s.nanos == 0 => 0,
|
None => 0,
|
||||||
Some(s) => Self::rate(s),
|
Some(r) => r.max(0.0).min(u64::MAX as f64) as u64,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -776,7 +805,8 @@ impl HandlerScoreboard {
|
|||||||
/// handler is pulling the weight and which is a dud on this drive/disc.
|
/// handler is pulling the weight and which is a dud on this drive/disc.
|
||||||
pub(super) fn log(&self) {
|
pub(super) fn log(&self) {
|
||||||
let mut rows: Vec<_> = self.stats.iter().collect();
|
let mut rows: Vec<_> = self.stats.iter().collect();
|
||||||
rows.sort_by_key(|(_, s)| std::cmp::Reverse(Self::rate(s)));
|
// Rank by the decayed rate (the live signal), highest first.
|
||||||
|
rows.sort_by(|a, b| self.rank(b.0).cmp(&self.rank(a.0)));
|
||||||
for (name, s) in rows {
|
for (name, s) in rows {
|
||||||
let mbps = s.recovered as f64 / (s.nanos as f64 / 1e9).max(1e-9) / 1_048_576.0;
|
let mbps = s.recovered as f64 / (s.nanos as f64 / 1e9).max(1e-9) / 1_048_576.0;
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
@@ -785,6 +815,7 @@ impl HandlerScoreboard {
|
|||||||
handler = name.as_str(),
|
handler = name.as_str(),
|
||||||
recovered_mb = s.recovered as f64 / 1_048_576.0,
|
recovered_mb = s.recovered as f64 / 1_048_576.0,
|
||||||
attempts = s.attempts,
|
attempts = s.attempts,
|
||||||
|
decayed_bytes_per_s = s.ewma_rate.unwrap_or(0.0),
|
||||||
mb_per_s = mbps,
|
mb_per_s = mbps,
|
||||||
"handler scorecard (this rip)"
|
"handler scorecard (this rip)"
|
||||||
);
|
);
|
||||||
@@ -1567,4 +1598,42 @@ mod tests {
|
|||||||
assert_eq!(out, HandlerOutcome::Halted);
|
assert_eq!(out, HandlerOutcome::Halted);
|
||||||
assert_eq!(h.read_count(), 0, "halt must precede any read");
|
assert_eq!(h.read_count(), 0, "halt must precede any read");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn scorecard_decays_so_a_late_starter_overtakes_an_early_winner() {
|
||||||
|
// The whole point of the DECAYED rate: the residual hardens mid-pass, so
|
||||||
|
// leadership must hand off. A cumulative rate would freeze "early" in the
|
||||||
|
// lead forever; the EWMA re-prices continuously.
|
||||||
|
let mut sb = HandlerScoreboard::default();
|
||||||
|
let dt = Duration::from_secs(1);
|
||||||
|
|
||||||
|
// Round 1 — "early" cleans the easy bulk; "late" finds nothing yet.
|
||||||
|
sb.record("early", 1_000_000, dt);
|
||||||
|
sb.record("late", 0, dt);
|
||||||
|
assert!(
|
||||||
|
sb.rank("early") > sb.rank("late"),
|
||||||
|
"early must lead once it's the only one recovering"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The bulk is gone. Now "early"'s technique no longer fits the hardened
|
||||||
|
// residual (barren attempts) while "late"'s specialist starts winning.
|
||||||
|
for _ in 0..4 {
|
||||||
|
sb.record("early", 0, dt);
|
||||||
|
sb.record("late", 1_000_000, dt);
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
sb.rank("late") > sb.rank("early"),
|
||||||
|
"a handler that stops earning must LOSE its lead to a late starter \
|
||||||
|
(late={}, early={})",
|
||||||
|
sb.rank("late"),
|
||||||
|
sb.rank("early")
|
||||||
|
);
|
||||||
|
|
||||||
|
// Calibration invariants preserved: an untried handler still ranks top
|
||||||
|
// (one-shot calibration), and a handler attempted with no timed read
|
||||||
|
// (zero elapsed) ranks bottom rather than crowding out proven performers.
|
||||||
|
assert_eq!(sb.rank("never_tried"), u64::MAX, "untried → top");
|
||||||
|
sb.record("idle", 0, Duration::ZERO);
|
||||||
|
assert_eq!(sb.rank("idle"), 0, "attempted-but-zero-time → bottom");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user