Review fixes, red half: a Stop during the host lookup must never reach the
service (never call the key service twice), and a pending Stop returns
before the body is built. KT7a's u64::MAX check now asserts saturation.
Commit 2 of 2 (stop-design-v5 §2.7, T15-T17, D3, D4):
- query_with builds the body on the caller's thread and runs DNS + POST on
a worker, capped at 4 per host. The caller waits for a slot and for the
result halt-aware in 20 ms slices, so Stop returns Halted within a slice
at any stage. The abandoned worker frees its slot at its own stall bound.
- Reachability is recorded on the caller's thread, only when the worker's
result arrives. A Stop records nothing.
- IdleConnector, chained after DefaultConnector, owns every post-connect
bound. Each write and read gets a fresh 60 s stall bound (T16), and the
wait for a reply's first byte gets 60 s + request bytes / 256 KiB/s (T17).
The 180 s phase totals are gone: no total is configured.
- A reply body that stalls or drops records Transport reachability (T16).
An over-cap body still records its Status.
The trait path passes a fresh Halt. ctx.halt()/ctx.progress() is ST-K1b,
after KU-X2. No public API change.
Commit 1 of 2 (stop-design-v5 §5.0): KT1-KT10 plus guards G12/G13, with
signatures that keep today's behaviour (query_with ignores the halt,
hardened_agent_with maps idle onto the old phase totals, the first-byte
budget does not scale), so each test compiles and fails on its assertion.
KT9 replaces the 180 s totals test. The ctx.halt()/ctx.progress() wiring
is ST-K1b (after KU-X2).
Correct the rationale: the env proxy was resolved to the key service's pinned address, so every lookup failed.
Gate set_last_decode_reachability behind a test-hooks feature, mark the verdict types non_exhaustive, test a .invalid host, add CHANGELOG.
The key-service agent disables ureq's env proxy, which would re-resolve the host and bypass the pinned, SSRF-guarded addresses.
check_keyserver_url returns a Permanent/Temporary KeyserverUrlRejection; set_last_decode_reachability plants a verdict for caller tests.
ureq 3.4.1 (#1194, "Fix timeout budgets restarting and applying to later
phases") made timeout_recv_response bound only the response HEADERS and
turned timeout_recv_body into an ABSOLUTE total-body deadline. The online
key-service POST set recv_response only, so under 3.4.2 the reply body read
had NO deadline at all — a server that sent headers then stalled mid-reply
would hang the key request forever.
The /decode reply is a small JSON key answer that arrives in one read (a
few hundred bytes; the 1 MiB cap is a defensive anti-OOM ceiling, not an
expected size), so this is a fine total-cap case, not a large-body slow-link
download: no rolling-idle connector is warranted. Set timeout_recv_body to
the same TIMEOUT_SECS total ceiling so the POST can never hang forever.
Regression guard the_reply_body_read_is_bounded_not_only_the_headers asserts
recv_body is now set. Full suite + precommit (fmt/clippy/deny/tests) green.
Restructure 10 over-cap comment blocks in keydb.rs and online.rs to fit
comment-guard's caps (inline // asides <=3 lines, /// item-doc prose <=8)
by tightening wording. No code changes; behavior unchanged.
online.rs:
- validate_keyserver_url/resolve_and_guard now reject any non-https:// scheme
as a Config fault, so a cleartext misconfig is caught ONCE at config time
instead of failing every rip via query()'s runtime https check.
- is_blocked_ip: add 198.18.0.0/15 (RFC 2544 benchmarking) and 192.0.0.0/24
(RFC 6890 IETF protocol assignments, incl. 192.0.0.170/171) to the SSRF guard.
- DNS in-flight cap is now PER HOST, not process-global, so black-holed lookups
to one dead keyserver can't starve resolution of a healthy different keyserver.
- reject a non-empty, non-":port" tail after a bracketed IPv6 authority
(e.g. https://[::1]extra/) instead of silently dropping it.
keydb.rs / keydb_format.rs:
- MAX_KEYDB_BYTES defined once in keydb_format and re-used by keydb.rs.
- CacheEntry::is_settled falls back to inode identity + stamped_at age when the
mtime is absent (mtime-less FS) or in the future (clock skew), so the ~62 MiB
keydb no longer re-parses on every lookup.
- save()'s entry-count guard counts DK/PK/HC/HC2 rows via the real parsers
(is_parseable_entry_line), so a prefixed-but-malformed row can't pass entries>0.
Tests added for each fix.
OnlineSource now records the HTTP outcome of its actual /decode POST — a
Status(code) for any answer (200/404/422/429/5xx) or Transport for a
connect/timeout/DNS/TLS failure — in a per-thread slot, readable via the new
take_last_decode_reachability() / DecodeReachability. This lets a caller
(autorip) classify a no-key result from the REAL decode instead of firing a
second, redundant probe against the POST-only endpoint. No change to key
resolution itself. Targets 1.7.2.
Several tests asserted only key[0] (the first byte), which a byte-transposition bug could pass — assert the full 16-byte key everywhere, including a KAT that recomputes the expected VUK-derived keys directly. test_parse_disc_entry used identical all-zero placeholders for M/I/V/U, which couldn't catch a field mix-up; give each field a distinct byte pattern and assert every field.
Docs: keydb-format.md claimed DiscEntry::title was genuinely unused dead code, but it's read by to_keydb_cfg and the redacting Debug impl — correct the comment to only list empty/find_vuk as truly unused. Also cross-reference the two independent MAX_KEYDB_BYTES constants (keydb.rs's decompression cap and keydb_format.rs's on-disk load cap) so they're kept in sync, and bump Cargo.lock's version metadata.
Each DNS lookup for the SSRF guard spawns a thread that is never joined and can hang for the full OS resolver timeout, so a black-holed keyserver leaks one thread+stack per attempt. Track outstanding resolver threads with an atomic counter; over a cap of 4, report the host as unreachable instead of spawning another. Decrement only when the (possibly long-hung) lookup actually returns, so the cap reflects threads truly in flight.
The IPv6 SSRF guard already re-checked IPv4-mapped/-compatible addresses, but 6to4 (2002::/16) and Teredo (2001:0000::/32) also tunnel an embedded IPv4 and were passed through untouched, letting an attacker wrap 127.0.0.1 or a cloud metadata address in either encapsulation to bypass the block list. Decode both forms and re-check the embedded IPv4 through is_blocked_ip. Add a test covering 6to4/Teredo-wrapped loopback and metadata addresses, plus a 6to4-wrapped public IP that must still be allowed.
CI/lint hardening since v1.6.14: clippy on stable with the MSRV single-sourced
from Cargo.toml, cargo-deny dependency-audit gate with first-party git sources
allowed (#15), and audience-based comment-guard (#17). No library code changes.
Adds targeted unit tests for previously-uncovered branches: keydb load
failure classification (invalid-UTF-8 / unreadable path), zip extraction
success/no-cfg-member paths, write_atomic create/rename failure modes,
keydb_format hex-digit-parsing edge cases, iter_disc_entries, to_keydb_cfg
HC2 emission and empty-title/comment-field combinations, MultiSource::label,
and the online key-service client's SSRF-guard malformed-authority and
unresolvable-host paths, UK-as-string and VUK-derivation reply handling,
and get_fmts_indexes. Local line coverage moves from ~90.3% to ~92.1%.
The round-2 guard tests used `keyserver.invalid` — RFC 6761 reserved and
guaranteed non-resolvable, which is the correct property. But the shared
leak-guard net lists `.invalid` among the private-infra TLD suffixes
(.internal/.local/.lan/.corp), so every such host tripped it and turned the
repo red on push — the exact cost the precommit script's own comment records
from 1.6.3. `.test` is equally RFC-6761-reserved and equally non-resolvable,
carries the same test guarantee, and is not in the infra net. No behaviour
change; the tests assert the identical never-resolves path.
Round 1 separated a config fault from a transient one and then routed
them to OPPOSITE severities. `GuardFail::Config` returned
`Ok(Vec::new())` from `query` while `GuardFail::Unreachable` returned
`Err`, so a mistyped port — `https://host:notaport/` — produced one
`warn!` and a benign-looking empty, which `MultiSource::first_non_empty`
cannot distinguish from "no source has a row for this disc". The disc
was reported as E7022 "no key for this disc". That is the seven-hour-502
collapse with the polarity reversed, and worse: an outage self-heals, a
typo in a config file does not.
Every way of failing to ASK is now `Err(KeyServiceUnavailable)`, logged
at error level with text that names it as a URL to fix rather than an
outage to wait out. That covers the address-guard rejection (bad port,
malformed URL, a host resolving to a non-public address) and the
cleartext `http://` refusal, which had the same silent shape.
The two input-shaped guards — an over-cap MKB, too few content samples —
deliberately stay `Ok(empty)`: they are properties of this disc, not
faults, and calling them an outage would mislabel them in the other
direction. The line between the three cases is now written down at the
guards themselves.
Not a rip-breaking change: an `Err` never blocks the chain, because
`first_non_empty` still returns any later source's key set outright and
surfaces a failure only when nothing anywhere resolved.
Also: the agent-reuse cache key is now sorted and deduped. It was an
ordered `Vec<SocketAddr>` compare, so a round-robin keyserver — the one
deployment the connection reuse exists for — reordered its A/AAAA
records and missed the cache on every query, silently paying a fresh TLS
handshake each time. Set equality is also the honest predicate for
"the same validated addresses".
Note for libfreemkv: there is no `KeyServiceMisconfigured` code, so a
permanent fault borrows the transient E7028. E7028's "the source never
answered" is exactly right here; only its "retry later" hint is wrong.
Two defect classes dominate this change, and both are the same shape: something
that should have been loud was silent.
KEY MATERIAL IN A PUBLIC `Debug`. `KeyDb` and `DiscEntry` both carried a derived
`Debug` over raw key bytes — every processing key, and every disc's Media Key,
VID, VUK and Unit Keys. Both are public API, so a single `{:?}` in a caller's
log line, a `tracing` field, or an `assert_eq!` panic message published the
whole keydb: on the real file that is 184,860 discs' key material plus the
processing-key pool. libfreemkv's equivalent types (`aacs::types`) all
hand-write a redacting `Debug` for exactly this reason; this crate's copy of the
parser dropped the convention when it was relocated. Both impls are restored,
field-for-field in the same style, and pinned by tests that fail against the
derive.
A SOURCE THAT COULD NOT ANSWER IS NOT A SOURCE THAT SAID "NO KEY". `OnlineSource`
was engineered at length to separate `Err` (the service could not answer) from
`Ok(empty)` (it answered and holds nothing) after a seven-hour run of HTTP 502s
was reported to operators as E7022 "no key for this disc". Three places
re-collapsed that distinction:
* `MultiSource` discarded every inner `Err` (`if let Ok(uks) = ..`), so a
composition whose sources had all failed returned a clean `Ok(empty)` — the
incident reproduced one layer up, in both the base and the forensic path. It
now returns the FIRST source failure when nothing resolved, matching
libfreemkv's own resolver, while a later success still wins outright.
* `KeydbSource` turned every `KeyDb::load` failure into `Ok(empty)` with no
log at all. A keydb over the 128 MiB cap or one that is not valid UTF-8 is a
corrupt, truncated or half-written file, not a disc without a key; those are
now logged and returned as errors. A MISSING keydb stays the documented
benign empty.
* A mistyped port in the key-service URL (`https://host:notaport/`) reused the
whole authority as the hostname, so it failed DNS and surfaced as
`Unreachable` — a standing misconfiguration reported as a transient outage,
the one collapse `GuardFail`'s doc says must never happen because the two
halves demand opposite operator actions. It is now a `Config` rejection, the
same verdict the bracketed-IPv6 branch already gave.
The keydb parser was equally silent: every malformed `| DK` / `| PK` / `| HC` /
`| HC2` / `0x…` row was dropped with no counter and no log, and so was a row
lost to the entry cap or overwritten by a duplicate hash. A byte-shifted mirror
download therefore passed `save()`'s validation, was atomically committed as the
new keydb, and resolved nothing for every disc past the damage — indistinguishably
from a healthy file. Rejections are now counted by reason and summarised in one
log line per parse. The real published keydb parses with zero rejections and zero
duplicates, so the summary stays quiet on healthy input.
Also fixed, with tests:
* The 62 MiB / 184,860-entry keydb was re-read and re-parsed on EVERY call —
~140 ms each in release, three or more times per AACS-cert rip. `KeydbSource`
now caches the parse behind the file's (size, mtime) stamp (~1 µs on a hit),
re-reads a file replaced underneath it, and drops its own cache on `save()`
so a same-size rewrite inside the timestamp resolution cannot be missed.
Peak RSS is unchanged; the parse simply is not repeated.
* A disc row prefixed `0X` was dropped silently, though `parse_hex` in the same
file deliberately accepts both cases. `save()`'s entry count now CALLS the
parser's own predicate instead of restating it, so the two cannot drift again.
* Each disc hash was stored twice — once as the map key, once in the entry —
~13 MB of duplicate text at real scale, now one shared allocation.
* `OnlineSource` built a fresh agent per query, paying a second TLS handshake
on every FMTS disc. The agent is reused only while the freshly re-resolved
and re-guarded address set is identical, so the anti-rebinding guarantee is
untouched.
Test coverage for defences that were never exercised: the cleartext-`http://`
refusal, the over-cap MKB skip and the too-few-samples skip (a reordering that
let a plaintext POST through — carrying the bearer token and base64 key material
— would have shipped green), and the 1 MiB reply cap. The real-scale tests used
to `return` early when `KEYDB_PATH` was unset, so CI ran them, asserted nothing
about 181k-entry behaviour and reported success; they are `#[ignore]`d with a
documented run command and now fail loudly if run without their fixture.
leak-guard's internal-infra net lists .invalid alongside .internal,
.local, .lan and .corp, so the hostname these tests use to prove the
pinned resolver is consulted tripped it. The point is only that the name
never resolves, and .test is reserved for exactly that by RFC 6761 -- it
is also what the rest of this suite already uses (example.test).
No change to what the tests assert.
The three crates move together because they must: freemkv resolves its
siblings from their git TAGS in its own CI, where keysources still wanted
ureq 2, so cargo saw no conflict and the earlier split bump went green on
a dependency graph the release never uses. The qa and hash-matrix
workflows resolve those siblings from their dev PATHS, cargo unifies to a
single ureq, and the mismatch stops compiling. Verified here the way
those workflows see it: freemkv built against the sibling paths now
resolves one ureq 3.4.0 and no ureq 2 anywhere.
Notable API changes:
AgentBuilder::new()..build() -> Config::builder()..build() + Agent
.redirects(0) -> .max_redirects(0)
.timeout_read(d) -> .timeout_recv_response(Some(d))
.resolver(closure) -> the Resolver trait
.set(k, v) -> .header(k, v)
ureq::Response -> http::Response<ureq::Body>
Error::Status(code, _) -> Error::StatusCode(code)
Error::Transport(_) -> fans out; matched with a catch-all
Two hazards the new resolver API introduces, both handled:
Agent::new_with_config compiles fine and then silently ignores a custom
resolver, so the pinned addresses would be dropped and every request
would go back to live DNS -- reopening the DNS-rebinding window the
guards exist to close, with no symptom. Only Agent::with_parts takes a
resolver, and that is what every pinned agent here uses.
ResolvedSocketAddrs is a fixed 16-slot array whose push writes straight
into it, so a host publishing more than 16 addresses would panic the
calling thread. Truncated to the first 16.
The four reply stubs used ureq::Response::new, which v3 removes; they are
rebuilt with http::Response::builder over a ureq::Body, so they stay
server-free unit tests.
resolve_and_guard validates the keyserver's addresses and hardened_agent
pins them, so a hostname cannot re-resolve to an internal address between
the guard and the POST. The validation half has seven tests; the pin had
none, and none of the existing tests makes a connection, so all 87 of them
pass even with the resolver wiring removed entirely.
That is a bad failure mode to leave uncovered, because a mis-wired
resolver fails open: connections fall back to live DNS, the rebinding
window reopens, and nothing looks wrong.
Pin the agent at a loopback listener the test owns, then ask it for a host
that cannot resolve (.invalid, reserved by RFC 2606). Only a consulted
resolver can turn that name into a connection. Asserts the connection
arrives, the round-trip completes, and the original host still travels in
the request.
Verified by removing the resolver wiring: this test fails on the timeout
while the other 87 stay green.
`query` returned `Vec::new()` for every outcome: a 502, a refused token, a DNS
failure, an unreadable body, and a service that answered 200 holding no key for
this disc. Only the last one means what the caller then reported -- E7022, "no
key source has a decryption key for this disc". The module already knew the
difference and logged it; the distinction died at the `return`, and the tracing
line that carried it is not on by default (RUST_LOG=info did not surface it
through the CLI in practice).
`query` now returns `Result<Vec<UnitKey>, Error>`:
Ok(non-empty) the service answered with a key
Ok(empty) the service ANSWERED and holds nothing, or there was nothing
to ask with (no service configured, blocked URL, over-cap MKB,
too few samples) -- a genuine miss, E7022 is correct
Err(..) the service could not answer -- nothing is known
`classify_http_status` is the table the fix exists for: 401/403 ->
KeyServiceUnauthorized, 429 -> KeyServiceRateLimited, 5xx -> unavailable, and
anything else non-2xx -> unavailable too. Deliberately: the genuine miss is a
**200 with no key in the body**, so a status code is never evidence about this
disc, and guessing "no key" from one is the original bug. An unreadable,
over-cap or unparseable body is the service failing mid-answer, so those are
failures as well.
`resolve_and_guard` now reports WHY it rejected a URL. A host that did not
resolve (DNS failure or DNS timeout) is the service being unreachable, and was
being folded in with "this URL is not allowed" -- one is transient, the other
never fixes itself. `GuardFail::Config` still falls through to the next source
as before.
The reply-to-verdict mapping moved into `interpret_reply` so it is testable
without a network. A stub HTTP server is not an option here: the SSRF guard
blocks loopback by design, so 127.0.0.1 can never be reached through `query`.
Nothing new is logged. The request body carries base64 key material and the
reply carries keys; the added fields are the error code and the status already
being recorded.
Red before green: http_5xx_is_a_source_failure_not_a_missing_key,
transport_failure_is_a_source_failure and unparseable_reply_is_a_source_failure
all fail against the old collapse-to-empty behaviour and pass after.
service_with_a_key_still_resolves_it_in_order is a regression pin -- it passes
either way, and is here because the happy path went through a signature change.
The Windows UI needs current winsafe, whose real minimum is 1.89 (its manifest
under-declares 1.87 while it uses NonNull::from_ref). Rather than stop at the
minimum, this goes to current stable and fixes what that costs.
The counter-intuitive result: 1.97 is CHEAPER than 1.89. libfreemkv had 54
clippy errors at 1.89 and 6 at 1.97, because clippy tightened the noisy
collapsible_if lint in between. Stopping at the minimum would have been the most
expensive choice available.
Roughly 47 lints across the eight repos, the large majority auto-fixed:
libfreemkv 6, freemkv-engine 14, bdemu 8, freemkv-keysources 7, autorip 6,
freemkv-unlock 3, freemkv-i18n 3. The hand-fixed ones are a descending sort to
sort_by_key(Reverse), four manual checked-division sites, a loop counter replaced
by enumerate, and a loop whose first let-else became a while-let.
Worth recording for whoever bumps next: clippy is MSRV-AWARE. Those 54 lints only
appear once the crate DECLARES 1.89 or later, because let-chains become
available. A bare `cargo +1.89 clippy` against a manifest still pinned at 1.87
reports clean and is meaningless — gate with the real precommit script, which is
also the only thing that covers build scripts.
The pin still sits below the Mac default, so it keeps doing its job: catching
lint drift locally before CI sees it.
Every request to the key service carries base64 AACS key material in its body —
the unit-key file, the MKB, content samples, the volume ID — and a bearer token
in a header when one is configured. Over http:// all of that goes to any on-path
observer, and the token is replayable.
There is no plaintext deployment to preserve. The address guard already rejects
loopback, RFC1918, ULA and link-local, so an http:// URL can only ever reach the
public internet, which is the worst case for cleartext; a self-hosted keyserver
on a LAN is impossible under that guard whatever its scheme. A non-https URL now
skips the online source with a warning rather than transmitting, so a
misconfiguration falls through to the local keydb and says why instead of
leaking quietly.
A key-service failure was indistinguishable from "this disc has no key".
Five of the seven exits in the online query returned an empty result with no
log at all, and the one that did log threw away the HTTP status the service
had already handed us — so a 502, an expired token, a rate limit and a
genuine miss all produced the same single line, and the disc was reported as
having no key. Each exit now says which one it was, and the genuine miss is
logged distinctly from every failure.
Nothing new is disclosed: the address that failed the guard, the response
body and the bearer token are all still kept out of the log. The HTTP status
is the service's own response code, and the size cap is a constant.
A malformed key in a returned UK set no longer skips that one element. The
mux trusts any non-empty result as the complete forensic set, so a short set
silently omits an index; the whole reply is rejected instead.
Also make the on-disk keydb cap real. It was a metadata pre-check followed by
an unbounded read, which is not a cap: it is skipped when the stat fails, and
a FIFO or character device reports zero length, so --keydb /dev/zero passed
the guard and then read until the process was OOM-killed (NUL is valid UTF-8,
so it never even failed the decode). Read through the same take() cap the
download path already used, and add the boundary test the load path never had.
Bump to 1.6.0 (workspace sync, no source change). Fix stale get_uk ->
get_unit_keys references in the crate description + comments, correct the
libfreemkv version note, and drop a phantom Mapfile mention from the tests.
Base per-CPS-unit keys and the AACS 2.1 forensic index set are now two explicit
trait operations instead of one overloaded get_uk whose return count was implicit.
keydb serves unit keys (forensic opts out via the default); online implements both;
MultiSource drives both in order. Teed up for 1.4.5. Local WIP baseline.
The key service now always returns {"UK":[...]} - an array of one for a
plain movie sample, or all 32 index-ordered variant keys for a forensic
sample. OnlineSource::query accepts both the legacy string form and the
array form, emitting one UnitKey per element (index = array position).
Re-export MIN_SAMPLE_UNITS so callers size their samples correctly.
leak-guard bans the .invalid TLD (grouped with private .internal/.local/etc);
example.test is the RFC 6761 reserved test TLD — equally unreachable, not on
the ban list — so the leak-guard CI check passes.
libfreemkv deleted the aacs::boil veneer, so switch the resolve path to the
raw primitives: derive_media_key_from_{pk,dk}, derive_vuk, decrypt_unit_key
from aacs::derive, and the newtypes from aacs::types. Add a local uks_from_vuk
helper composing decrypt_unit_key. No behaviour change; fmt/clippy/test green
on Rust 1.86.
The online source dropped any MKB over 10 MiB while libfreemkv's reader
captures up to 64 MiB — an MKB in that band was silently un-forwardable
(no key, no surfaced cause). Match the cap and log when it is exceeded.
- KeydbSource implements KeySource::host_certs(), delegating to the
inherent host_certs() — surfaces the | HC | / | HC2 | certs already
parsed from keydb.cfg by libfreemkv's parser, so the OEM cert route
collects them across the keysource layer. No new parsing.
- OnlineSource::host_certs() is a no-op stub: returns empty with zero
network access (no client fetch, no server endpoint). Online host-cert
serving is deferred. TODO(owner) marker left in place.
Tests: trait host_certs returns the keydb HC row; empty when keydb
missing; online host_certs is an empty no-op without network.
OnlineSource::query() now includes the disc's volume_label (UDF/ISO volume id)
as a plain-text `title` field in the /decode POST, so the key service can build
a disc_hash → title catalog from real rips. Depends on libfreemkv 0.30.
Each source implements next_key (a cursor over its candidates) instead of
returning them all at once. The keydb hands its per-disc candidates out
UK-first (UK > VK > MK > DK) so a stale/wrong VUK never pre-empts a good UK in
the same entry; online and mapfile are one-shot. MultiSource composes sources
in the caller's chosen order and resolve_and_apply drives the
next_key -> decrypt_with loop, stopping at the first key that decrypts.
read_sample_units moves here so the CLI and autorip share one content sampler.
OnlineSource: the remote key-service client (moved out of autorip), posting the
disc's Unit_Key_RO.inf + MKB + Volume ID + encrypted content samples to the
service and returning the resolved unit key as a terminal Key::Unit candidate.
Kept out of libfreemkv so the library stays network-free. Source-internal
failures (unreachable / status / parse) are logged and surface as "no
candidate" so the next source is tried.
MapfileSource: reads a rip mapfile's persisted unit keys (the resume / deferred-
mux fast path) and offers them as a Key::Unit candidate. Keyed by mapfile path.
DiscInputs gained an app-populated `samples` field for sources that validate
against ciphertext server-side (OnlineSource); local sources ignore it.
Matthew Jackson