Per keys-upfront-design KU-K1 (§8.2): KeydbSource::answer_depends_on_samples
is false (keyed by disc hash, asked once per resolve, N-KU10); MultiSource's
is any(inner) (KU3-6), so a nested sample-dependent source still gets the
per-piece ask. OnlineSource::last_failure_was_transport (J15) is true only
when the most recent query recorded DecodeReachability::Transport (DNS,
connect refused, or an idle stall — never an HTTP answer, 5xx included), and
resets on the next success, so resolve's J13 retry (stop-design-v5 T31)
targets exactly the requests that never got an answer.
KeydbSource::answer_depends_on_samples (KSK1) and MultiSource::answer_depends_on_samples
(KSK2, KU3-6 "any(inner)") both default to `true` today, so a keydb-then-online
composition would be re-asked per piece instead of once. OnlineSource::last_failure_was_transport
(J15) defaults to `false`, so a transient DNS/connect/idle failure would never be
retried by `resolve` (J13). Red: these six assertions fail against today's defaults.
Implement KeySource::resolve_unit_keys for KeydbSource by splitting
unit_keys_from into resolve_from, which reports: whether find_disc
matched; the specific miss reason (NoVid when a Media Key has no VID,
else NoDerivableKey); the matched entry's shape (vuk / unit_keys count /
media_key / keydb_vid / enc_title_keys len / vid_available); and the
loaded entry count for a self-diagnosing true-miss verdict.
MultiSource forwards resolve_unit_keys so a matched inner source's richer
verdict survives composition. No derivation-logic change: the locked
VID-per-path rule is intentional and the investigation found no
correctness bug there — only the conflated verdict. Tests cover
matched-no-VID, matched-no-material, true miss, and the success path.
- Remove the MAX_DISC_ENTRIES cap: a keydb is a database and must hold every
disc row it contains (a count-based drop silently lost real discs).
- Strip a leading UTF-8 BOM before parsing so a Windows-saved keydb's FIRST
disc row survives (U+FEFF is not trim()-able whitespace).
- Gate the disc branch on the 0x/0X prefix alone and COUNT a malformed 0x row
(missing " = ") as rejected instead of silently ignoring it — surfaces the
"hash is in the file but nothing resolves" class of bug.
- Tests: BOM, CRLF, leading whitespace, case/prefix-insensitive lookup,
duplicate-hash last-wins, unit-key-only rows, 600k-entry no-drop, and the
load_failure verdicts (permission / invalid / missing).
[skip-media]
Restructure 10 over-cap comment blocks in keydb.rs and online.rs to fit
comment-guard's caps (inline // asides <=3 lines, /// item-doc prose <=8)
by tightening wording. No code changes; behavior unchanged.
online.rs:
- validate_keyserver_url/resolve_and_guard now reject any non-https:// scheme
as a Config fault, so a cleartext misconfig is caught ONCE at config time
instead of failing every rip via query()'s runtime https check.
- is_blocked_ip: add 198.18.0.0/15 (RFC 2544 benchmarking) and 192.0.0.0/24
(RFC 6890 IETF protocol assignments, incl. 192.0.0.170/171) to the SSRF guard.
- DNS in-flight cap is now PER HOST, not process-global, so black-holed lookups
to one dead keyserver can't starve resolution of a healthy different keyserver.
- reject a non-empty, non-":port" tail after a bracketed IPv6 authority
(e.g. https://[::1]extra/) instead of silently dropping it.
keydb.rs / keydb_format.rs:
- MAX_KEYDB_BYTES defined once in keydb_format and re-used by keydb.rs.
- CacheEntry::is_settled falls back to inode identity + stamped_at age when the
mtime is absent (mtime-less FS) or in the future (clock skew), so the ~62 MiB
keydb no longer re-parses on every lookup.
- save()'s entry-count guard counts DK/PK/HC/HC2 rows via the real parsers
(is_parseable_entry_line), so a prefixed-but-malformed row can't pass entries>0.
Tests added for each fix.
Several tests asserted only key[0] (the first byte), which a byte-transposition bug could pass — assert the full 16-byte key everywhere, including a KAT that recomputes the expected VUK-derived keys directly. test_parse_disc_entry used identical all-zero placeholders for M/I/V/U, which couldn't catch a field mix-up; give each field a distinct byte pattern and assert every field.
Docs: keydb-format.md claimed DiscEntry::title was genuinely unused dead code, but it's read by to_keydb_cfg and the redacting Debug impl — correct the comment to only list empty/find_vuk as truly unused. Also cross-reference the two independent MAX_KEYDB_BYTES constants (keydb.rs's decompression cap and keydb_format.rs's on-disk load cap) so they're kept in sync, and bump Cargo.lock's version metadata.
Two disc-side reads were silently swallowed with unwrap_or(&[]): a failure reading the disc's encrypted title keys or its MKB looked identical to a disc that genuinely had none, hiding real I/O/parse errors from the operator. Log a warning (mirroring online.rs's existing pattern) before falling back to empty in both cases.
Also, a stored unit key numbered 0 (an invalid CPS unit number; valid numbering is >= 1) was mapped via num.saturating_sub(1) onto idx 0, silently colliding with and shadowing a legitimate unit 1. Skip unit number 0 instead of mis-mapping it, and add a regression test.
cached_db held self.cache locked across the full KeyDb::load_counted parse of the ~62 MiB keydb, so a reparse (triggered by a changed mtime) stalled every other worker waiting on the cache. Take a brief lock for the fast-path settled-cache-hit check, release it, parse outside the lock, then re-acquire and double-check: if a peer installed the same settled stamp while we parsed, adopt their entry and drop our redundant parse rather than racing a lost update. Stamp and bytes still come from the one already-open file handle, preserving the one-open-one-identity invariant.
keydb.cfg carries AACS key material plus the host private key/cert. write_atomic renamed a File::create()'d temp file onto it, so the final mode depended on umask and could leave the keys world-readable. On Unix, open the temp file with mode 0600 explicitly via OpenOptionsExt; non-Unix keeps the plain create().
CI/lint hardening since v1.6.14: clippy on stable with the MSRV single-sourced
from Cargo.toml, cargo-deny dependency-audit gate with first-party git sources
allowed (#15), and audience-based comment-guard (#17). No library code changes.
Adds targeted unit tests for previously-uncovered branches: keydb load
failure classification (invalid-UTF-8 / unreadable path), zip extraction
success/no-cfg-member paths, write_atomic create/rename failure modes,
keydb_format hex-digit-parsing edge cases, iter_disc_entries, to_keydb_cfg
HC2 emission and empty-title/comment-field combinations, MultiSource::label,
and the online key-service client's SSRF-guard malformed-authority and
unresolvable-host paths, UK-as-string and VUK-derivation reply handling,
and get_fmts_indexes. Local line coverage moves from ~90.3% to ~92.1%.
Round 1's parse cache keyed on `(len, modified)`, which is not a file
identity. mtime is stored at whole-second resolution on HFS+, on many
container overlays and on most network filesystems, so an external
writer — the daily-refresh job, a second freemkv process, an editor, a
sync tool — replacing keydb.cfg within one second with a file of the
same byte length left the stamp bit-identical and the source served the
superseded parse for the rest of the process's life. Retired or
since-corrected keys, reported as success. The doc claiming "an
externally replaced keydb is still picked up on the next call" was false
in exactly that case.
Two discriminators, chosen so neither costs a 62 MiB content hash (the
141 ms x 3+ loads per rip the cache removed):
1. dev + ino. Every atomic publisher (temp + fsync + rename, including
this crate's save()) creates a new inode, so the stamp differs
unconditionally, whatever the clock did.
2. A settle window. An entry is trusted only once the file has been
quiet for the mtime granularity (2 s); the proof that this closes
the residual in-place case, and the network-clock caveats, are in
CacheEntry::is_settled. Cost: re-parsing for up to 2 s after each
write of a file that is rewritten daily.
What it still does NOT catch, now stated in the doc instead of implied
away: a writer that forges the timestamp back to the cached value
(touch -t, rsync --times, a backup restore) into the same inode at the
same length. save() therefore keeps invalidating directly.
Also fixed, all in the same cache:
- The corruption warning was swallowed. ParseStats::log runs inside
KeyDb::parse and a cache hit skips parse, so a damaged keydb warned
once per process and was then served in silence to every later rip.
It is re-emitted on every hit that serves the file; an intact keydb
still says nothing.
- TOCTOU. cached_db stat'd the path and then separately loaded the
path — two resolutions of one name with no atomicity, so a rename in
the gap filed one file's stamp beside another file's keys. The file
is opened once; the stamp is an fstat on that handle and the bytes
come from it.
- Memory residency is recorded rather than quietly widened: a hit
holds every disc's key material plus host-cert private keys for the
process's life, unzeroized. Nothing in this crate zeroizes on drop
and that predates the cache; closing it means zeroize-on-drop across
KeyDb, DiscEntry and libfreemkv's aacs::types, which is cross-crate
work and not smuggled in behind a cache fix.
On KeydbHostCert's derived Debug: audited, NOT a leak. libfreemkv's
HostCert hand-writes a redacting Debug (private_key: "<redacted>",
certificate_len), so `{:?}` on a host cert prints no key bytes. But that
is an external guarantee this crate inherits silently, and the existing
redaction test only covered the outer types. It is now pinned here, for
host certs and device keys, verified red against a libfreemkv patched to
derive Debug instead.
Two KEYDB_PATH-gated tests that could not fail are fixed too:
test_decrypt_real_unit fell off the end after trying every key (a no-op
decrypt_unit passed it), and test_resolve_keys_vuk_path returned early
with zero assertions when the fixture had no qualifying entry.
CHANGELOG: entries for this and for dd9a5dc, including the previously
unrecorded Arc<str> break to KeyDb::disc_entries / DiscEntry::disc_hash.
Two defect classes dominate this change, and both are the same shape: something
that should have been loud was silent.
KEY MATERIAL IN A PUBLIC `Debug`. `KeyDb` and `DiscEntry` both carried a derived
`Debug` over raw key bytes — every processing key, and every disc's Media Key,
VID, VUK and Unit Keys. Both are public API, so a single `{:?}` in a caller's
log line, a `tracing` field, or an `assert_eq!` panic message published the
whole keydb: on the real file that is 184,860 discs' key material plus the
processing-key pool. libfreemkv's equivalent types (`aacs::types`) all
hand-write a redacting `Debug` for exactly this reason; this crate's copy of the
parser dropped the convention when it was relocated. Both impls are restored,
field-for-field in the same style, and pinned by tests that fail against the
derive.
A SOURCE THAT COULD NOT ANSWER IS NOT A SOURCE THAT SAID "NO KEY". `OnlineSource`
was engineered at length to separate `Err` (the service could not answer) from
`Ok(empty)` (it answered and holds nothing) after a seven-hour run of HTTP 502s
was reported to operators as E7022 "no key for this disc". Three places
re-collapsed that distinction:
* `MultiSource` discarded every inner `Err` (`if let Ok(uks) = ..`), so a
composition whose sources had all failed returned a clean `Ok(empty)` — the
incident reproduced one layer up, in both the base and the forensic path. It
now returns the FIRST source failure when nothing resolved, matching
libfreemkv's own resolver, while a later success still wins outright.
* `KeydbSource` turned every `KeyDb::load` failure into `Ok(empty)` with no
log at all. A keydb over the 128 MiB cap or one that is not valid UTF-8 is a
corrupt, truncated or half-written file, not a disc without a key; those are
now logged and returned as errors. A MISSING keydb stays the documented
benign empty.
* A mistyped port in the key-service URL (`https://host:notaport/`) reused the
whole authority as the hostname, so it failed DNS and surfaced as
`Unreachable` — a standing misconfiguration reported as a transient outage,
the one collapse `GuardFail`'s doc says must never happen because the two
halves demand opposite operator actions. It is now a `Config` rejection, the
same verdict the bracketed-IPv6 branch already gave.
The keydb parser was equally silent: every malformed `| DK` / `| PK` / `| HC` /
`| HC2` / `0x…` row was dropped with no counter and no log, and so was a row
lost to the entry cap or overwritten by a duplicate hash. A byte-shifted mirror
download therefore passed `save()`'s validation, was atomically committed as the
new keydb, and resolved nothing for every disc past the damage — indistinguishably
from a healthy file. Rejections are now counted by reason and summarised in one
log line per parse. The real published keydb parses with zero rejections and zero
duplicates, so the summary stays quiet on healthy input.
Also fixed, with tests:
* The 62 MiB / 184,860-entry keydb was re-read and re-parsed on EVERY call —
~140 ms each in release, three or more times per AACS-cert rip. `KeydbSource`
now caches the parse behind the file's (size, mtime) stamp (~1 µs on a hit),
re-reads a file replaced underneath it, and drops its own cache on `save()`
so a same-size rewrite inside the timestamp resolution cannot be missed.
Peak RSS is unchanged; the parse simply is not repeated.
* A disc row prefixed `0X` was dropped silently, though `parse_hex` in the same
file deliberately accepts both cases. `save()`'s entry count now CALLS the
parser's own predicate instead of restating it, so the two cannot drift again.
* Each disc hash was stored twice — once as the map key, once in the entry —
~13 MB of duplicate text at real scale, now one shared allocation.
* `OnlineSource` built a fresh agent per query, paying a second TLS handshake
on every FMTS disc. The agent is reused only while the freshly re-resolved
and re-guarded address set is identical, so the anti-rebinding guarantee is
untouched.
Test coverage for defences that were never exercised: the cleartext-`http://`
refusal, the over-cap MKB skip and the too-few-samples skip (a reordering that
let a plaintext POST through — carrying the bearer token and base64 key material
— would have shipped green), and the 1 MiB reply cap. The real-scale tests used
to `return` early when `KEYDB_PATH` was unset, so CI ran them, asserted nothing
about 181k-entry behaviour and reported success; they are `#[ignore]`d with a
documented run command and now fail loudly if run without their fixture.
Both MAX_KEYDB_BYTES comments justified the 128 MiB ceiling by naming
the public keydb they were sized against. The number and the reasoning
are the useful part; the name is not, and it trips the pre-commit
scan. Keep the justification, drop the name.
The public UHD keydb is ~62 MiB and growing, leaving only ~2 MiB
under the old 64 MiB ceiling — updates would start failing as it grows.
128 MiB restores years of headroom while still bounding a decompression
bomb. Both cap sites (keydb.rs download/save, keydb_format.rs on-disk load)
bumped; the unrelated MKB cap in online.rs is untouched.
Bump to 1.6.0 (workspace sync, no source change). Fix stale get_uk ->
get_unit_keys references in the crate description + comments, correct the
libfreemkv version note, and drop a phantom Mapfile mention from the tests.
Base per-CPS-unit keys and the AACS 2.1 forensic index set are now two explicit
trait operations instead of one overloaded get_uk whose return count was implicit.
keydb serves unit keys (forensic opts out via the default); online implements both;
MultiSource drives both in order. Teed up for 1.4.5. Local WIP baseline.
leak-guard bans the .invalid TLD (grouped with private .internal/.local/etc);
example.test is the RFC 6761 reserved test TLD — equally unreachable, not on
the ban list — so the leak-guard CI check passes.
- to_keydb_cfg emits the sibling `| HC2 |` line (AACS 2.0 host cert),
the inverse of parse_host_cert_v2, so a round-trip never silently
drops v2 host certs.
- save-validation mirrors the parser's real disc-entry rule exactly: a
`0x` line counts as an entry only if it also contains ` = `, so
validating + persisting content that parses to zero usable entries
(e.g. a stray `0xDEADBEEF` line) can no longer succeed.
- disc-entry title is kept verbatim (parens and all) on parse, matching
the emit path so it round-trips.
libfreemkv deleted the aacs::boil veneer, so switch the resolve path to the
raw primitives: derive_media_key_from_{pk,dk}, derive_vuk, decrypt_unit_key
from aacs::derive, and the newtypes from aacs::types. Add a local uks_from_vuk
helper composing decrypt_unit_key. No behaviour change; fmt/clippy/test green
on Rust 1.86.
libfreemkv's AACS work changes mk_from_dk from (dk, mkb, vid) to (dk, mkb) —
the VID enters at the VUK step, not the MK step. Adopt the 2-arg call so the DK
pool is tried unconditionally (no longer gated on a VID being present), matching
libfreemkv. Without this keysources fails to compile against the new libfreemkv.
(Equivalent to the stranded b80d4cb; applied directly since that branch had
diverged surrounding context.)
unit_keys_from now loads all stored unit keys AND the VUK/MK-derived ones, deduping by key value instead of first-hit, so a disc with partial stored UKs plus a VUK yields every declared CPS unit key. Plus keydb.cfg format round-trip (to_keydb_cfg inverse of parse).
Central round-trippable codec: parse + to_keydb_cfg in one place.
Emits HC, DK, PK, then disc entries sorted by hash. Round-trip test
(parse -> serialize -> re-parse) covers HC/DK/PK/disc. disc_id -> vid
names the field for what it is (the I-field volume ID).
Move the keydb save/validation/atomic-write path out of libfreemkv onto
KeydbSource. New KeydbSource::save(bytes) validates + decompresses (zip /
gz / plain, decompressed-size capped) and crash-safely writes to the
source's OWN path (sibling-temp + fsync + rename + parent-dir fsync) —
not a hardcoded default, so the caller chooses the destination.
KeydbSource::update(fetch, url) calls an INJECTED fetch closure then
save, keeping this crate transport-agnostic on the update path (the app
supplies its own TLS / SSRF-guarded transport). UpdateResult moves here
and is re-exported. Add flate2 + zip (already in the resolved graph via
libfreemkv) for decompression; no new HTTP stack.
unit_keys_from now resolves the Media Key in order: stored per-disc MK
-> keydb Processing Key pool (mk_from_pk vs this disc's own MKB) ->
device-key pool (mk_from_dk), then MK+VID -> VUK -> UK. MK/VUK entries
still honored directly; cross-disc MK-pool brute stays retired. Fixes the
factually-wrong justifying comment + adds PK-pool KATs.
A keydb can hand out a per-disc terminal Key::Unit (a UK entry keyed on
disc_hash). Unlike a derived key (Device/Processing/Media/Volume), a
terminal UK is applied as-is by Disc::decrypt_with: it is NOT re-derived
through the MKB-verified AACS resolver, so a UK entry whose hash matches
the disc but whose key bytes are wrong is only disproved by descrambling
real ciphertext.
KeydbSource inherited the default needs_samples() == false, so on the
autorip auto-resume / mux-worker path (which samples units only when some
source reports needs_samples()) a keydb-only resolve ran with empty
samples and committed a wrong UK as success, muxing undecryptable video
while reporting done. The CLI was unaffected because it always samples.
Override needs_samples() to true on KeydbSource so every consumer samples
encrypted units before resolving and a wrong keydb UK is rejected on all
paths. Regression test asserts the override.
- KeydbSource implements KeySource::host_certs(), delegating to the
inherent host_certs() — surfaces the | HC | / | HC2 | certs already
parsed from keydb.cfg by libfreemkv's parser, so the OEM cert route
collects them across the keysource layer. No new parsing.
- OnlineSource::host_certs() is a no-op stub: returns empty with zero
network access (no client fetch, no server endpoint). Online host-cert
serving is deferred. TODO(owner) marker left in place.
Tests: trait host_certs returns the keydb HC row; empty when keydb
missing; online host_certs is an empty no-op without network.
OnlineSource::query() now includes the disc's volume_label (UDF/ISO volume id)
as a plain-text `title` field in the /decode POST, so the key service can build
a disc_hash → title catalog from real rips. Depends on libfreemkv 0.30.
Each source implements next_key (a cursor over its candidates) instead of
returning them all at once. The keydb hands its per-disc candidates out
UK-first (UK > VK > MK > DK) so a stale/wrong VUK never pre-empts a good UK in
the same entry; online and mapfile are one-shot. MultiSource composes sources
in the caller's chosen order and resolve_and_apply drives the
next_key -> decrypt_with loop, stopping at the first key that decrypts.
read_sample_units moves here so the CLI and autorip share one content sampler.
OnlineSource: the remote key-service client (moved out of autorip), posting the
disc's Unit_Key_RO.inf + MKB + Volume ID + encrypted content samples to the
service and returning the resolved unit key as a terminal Key::Unit candidate.
Kept out of libfreemkv so the library stays network-free. Source-internal
failures (unreachable / status / parse) are logged and surface as "no
candidate" so the next source is tried.
MapfileSource: reads a rip mapfile's persisted unit keys (the resume / deferred-
mux fast path) and offers them as a Key::Unit candidate. Keyed by mapfile path.
DiscInputs gained an app-populated `samples` field for sources that validate
against ciphertext server-side (OnlineSource); local sources ignore it.
The published key-source layer for libfreemkv. libfreemkv does no lookup; it
is handed a Key and derives down. This crate provides the KeySource impls that
do the lookup and hand a Key in. Applications choose and order the sources.
This first cut ships:
- KeydbSource: parses a local keydb.cfg and enumerates its material as ordered
candidate keys (per-disc VUK/unit/media first, then the universal device-key,
processing-key, and media-key pools). It does no derivation — the library
walks the MKB and verifies media keys. Candidate ordering lets the library
try each path a keydb can satisfy.
- resolve_first: tries each source's candidates in order and returns the first
the caller's validator accepts (validate-before-return), so a stale entry
falls through to the next source.
OnlineSource (remote key service) and MapfileSource (cached unit key) land with
the application wiring, where the sample-read and mapfile paths already live.
Matthew Jackson