Version bump and changelog for the 1.6.3 sync. All eight crates ship the
same version, so the crates with no functional change this cycle say so
rather than carrying an empty section.
Two problems, both invisible until the whole graph is looked at together.
DEAD: num-bigint, sha2, num-traits, num-integer, cmac and cbc are
declared here and referenced nowhere -- not in src, tests or benches.
They were being compiled, audited and offered version bumps forever for
no reason. Removing beats bumping.
cbc nearly survived the sweep: a substring search for "cbc" matches 44
occurrences of ycbcr_to_rgb in the DVD subtitle decoder, so it looked
used. Only a word-boundary search exposed it.
SKEW: this crate was the outlier on every shared dependency -- aes 0.8,
rand 0.8, base64 0.22.1 and zip 2 against 0.9 / 0.10 / 0.23 / 8
elsewhere. Cargo cannot unify across a major version, so it compiled
BOTH: 32 duplicated crates in the freemkv binary's graph, including two
complete AES implementations (aes 0.8 + 0.9, cipher 0.4 + 0.5), two
digest stacks and two getrandom. Two crypto stacks in one product is
worth removing on its own.
The aes bump is an API rename -- BlockCipher-prefixed traits, Array for
GenericArray -- and the obvious translation uses Array::from_slice,
which the new version deprecates and clippy's -D warnings would reject.
These use the From<[T; N]> conversion the crate points at instead.
3441 tests pass in debug and release. The AACS crypto here is covered by
known-answer tests, so a byte-order or sizing mistake in that rename
could not have passed.
Version updates were never configured here -- only security alerts, which
report but never open a pull request. So the absence of Dependabot PRs was
not "nothing to update", it was "nothing was checking".
Updates target `dev`, never `main`. main is a release pointer that
release.sh moves to each tag, so a bot commit on it would put work there
that no tag contains -- the exact state that aborted the 1.6.2 cascade at
its last step, and pointing a scheduled bot at main would recreate it
weekly.
Minor and patch bumps are grouped into one PR rather than one per crate:
eight repos times a handful of dependencies is a volume nobody reads, and
an unread PR queue is indistinguishable from no updates at all.
The freemkv crates are excluded. They depend on each other by git tag,
re-pinned by release.sh inside the release commit, and Dependabot cannot
see that cascade -- a PR bumping one could pin a version whose tag does
not exist yet.
It fired on `release: published` and committed to main, the branch the
release points at its tag, so it could only ever leave main ahead of the
tag. release.sh now makes the same edit inside the release commit.
In this repo it was a no-op regardless: the pattern it rewrote does not
appear in this README, so it has been running and changing nothing.
A clip's span is measured over the title's full extents when the disc is
scanned, and a frame is placed by the offset it was read from. Those only
agree while the mux reads every byte the scan counted.
On a disc carrying a forensic segment it does not: the read plan omits the
units belonging to another device group, so fewer bytes are fed than the
spans describe, and the shortfall grows through the title. Every frame past
the first segment then looks earlier than it is — placed in a clip it did
not come from, or dropped at a join for failing marks it was never inside.
The spans still tile one another perfectly, so the check that asks whether
they can be trusted cannot see any of it.
The plan is now compared against the extents it was built from. When they
differ the spans are dropped and placement falls back to timestamps, which
is what that path is for and what the surrounding comment already promised
would happen when an offset stops meaning anything. Ordinary discs are
untouched: with no forensic segment the plan IS the extents.
The provenance guard knew one way to lose a frame's source offset —
writing it as absent. Omitting the field entirely does the same thing,
because the frame type fills it in by default, and the guard read straight
past that. A parser rewritten into the second spelling would have kept its
green light while its track silently went back to being placed by guesswork.
It now reports which spelling it found and where.
The title-count clamp was asserted over a fixture too small to hold more
titles than the cap allows, so the walk stopped when the buffer ran out and
the clamp was never what bounded it — the assertion held with the clamp
deleted. The fixture now carries more entries than the cap.
The subdirectory limit was checked by restating the constant's own
definition; the guard itself had never run, and deleting it changed
nothing. The limit is lowered under test so a real folder can exceed it,
and the test now walks one and requires the refusal.
One test also carried two unrelated grounding notes while the test they
described had none, so reading the note above a test told you about a
different one.
A playlist may point at the same .m2ts from several play items — a
seamless split, a looped segment, multiple angles. The file has one set of
bytes, so it has one span, and the byte a frame was read from therefore
identifies the FILE, not which play item's range it falls in. Placement
took the first of them and judged every frame against its marks, so
everything past that range's end was treated as material the playlist
excludes and dropped: half the clip missing from the rip, with the
timeline still charged for its full duration.
The offset narrows a frame to the run; only its timestamp can finish the
job, and each play item carries its own marks. A frame is now matched
against the marks of the entry it actually falls in, falling back to the
first when it falls in none, which leaves genuinely-excluded material
dropped as before.
The test for this construction asserted only that such a playlist is
trusted, and never placed a frame from the second range — so the loss it
described in prose was invisible to it. It now places one from every range
and requires them all to survive.
The deferred-mux replay of buffered frames wrote them without their source
offset, quietly sending the head of every such title down the timestamp
heuristic the rest of this work exists to retire. It passes the offset
through now, and the frame writer that omits it is compiled out of the
library entirely: no production path can discard provenance any more.
Also: the AC-3 buffer-reuse test could not see the regression it named.
Feeding equal-sized packets, a fresh allocation per call yields the same
capacity as a reused one. It now feeds a large packet then a small one,
where only a reused buffer keeps the larger capacity.
An ADTS header that declares a CRC follows must be at least nine bytes —
seven of header plus the two the CRC occupies — because the declared frame
length counts them. The structural gate compared against a flat seven and
never read the bit that says whether a CRC is there at all, so a frame
whose own header describes something impossible was accepted and handed to
the muxer as decodable.
The pipeline's spawn doc named Sweep, and the thread Sweep would have
created, as callers to look for. Neither has been in this crate since the
recovery passes moved out. The same paragraph already records fixing this
once, for a different departed caller — it simply drifted again a sentence
later, so it now says to name callers that live here or name none.
Whether a disc is structurally AACS-encrypted was spelled out by hand in
both the fast identify and the full scan. They agreed today; nothing made
them agree tomorrow, and disagreeing would mean the same disc reported
encrypted by one path and clear by the other. There is one definition now,
and the comment that pointed at it by line number points at its name.
The AC-3 parser built a fresh buffer on every packet — of the order of a
hundred thousand times per title — to work around a borrow it cannot
avoid. The copy stays; the allocation does not. The buffer is now lent out
and handed back, and a test pins that, because reverting it would be
invisible in behaviour.
Left alone deliberately: send/send_with_halt and finish/finish_with_halt
look like one action under two names, and are not. After the consumer
fails, one must still accept items and the other must refuse them; that
difference is what stops a producer reading an entire disc for a write
that died on its first frame. Collapsing them was tried here and the
existing test caught it. Both now say so where the choice is made.
A recovery resumed against an image that has been truncated since the
previous pass — a full disk, an interrupted transfer, a remount — cannot
repair it: the pass only revisits the ranges the recovery data calls bad,
so everything past the cut stays a hole while the counts still describe a
whole disc. There was no way to say that. Reusing the recovery-data error
would have been wrong, because the recovery data is intact; the image is
not.
Registered in the uniqueness list, the code table and the range check, so
a future variant cannot silently reuse 6015 or map to the wrong code.
The `consumers` job checked out all five dependents against this commit and
ran `cargo check --all-targets`. That proves they still COMPILE, which
catches a changed signature and nothing else. The failures worth catching
here keep every signature intact and change behaviour: the library still
builds, the dependent still builds, and the dependent's tests are what go
red. Those never ran.
Run their suites instead. Same checkouts, same patched paths — only the
verb changes.
The demuxer interleaves, so at a boundary the streams do not reset on the
same frame: audio for the next segment can reach the muxer before the video
frame that opens the epoch it belongs to. Only the primary video track may
open one, so those frames rode the just-ended offset and landed a whole
segment in the past. Downstream the strictly-monotonic block nudge then
crushed the entire run onto one instant a tick apart, which is audible.
Such a frame is recognised on its OWN raw PTS, which within an epoch only
advances, so a large backward step is unambiguous — a different signal from
the shared frontier, which is what the old false-positive ratchet keyed on.
It then rides a provisional offset private to its track, computed the same
way the video path computes a real one, and drops it the moment the video
retires an epoch, so the run rejoins with no seam.
The offset is deliberately private: it never writes offset_ns, never
advances the frontier and never retires an epoch, so it cannot move the
video timeline. Letting a passive track open a REAL epoch was tried first
and inflated a 476.776 s title to 656.216 s, because every track meets a
boundary at its own pace and the video path rebased again on top.
Measured on a real DVD title with 8 cell boundaries: audio frames stamped
inside one cadence 120 -> 0, outsized gaps 7 -> 0, subtitle 1 -> 0, and the
video span byte-identical at 476.484. On the HD-DVD title the remaining
boundary gap on its first audio track falls from 0.999 s to 0.275 s.
Acceptance: 76 pass, 0 fail, 0 skip.
A non-video frame can reach the muxer after the video that opened the next
epoch has already been processed. It carries an old-epoch PTS, so adding
the new offset flings it forward by a whole clip. The remap that exists to
catch this compared the frame against the CURRENT frontier and only
accepted it within one backstep, which is the wrong yardstick: by the time
a straggler is seen the frontier has moved on into a new epoch, and how far
below it the frame lands says nothing about where it belongs.
Retiring an epoch now records the frontier it closed at alongside its
offset, and a straggler is judged against the end of the epoch its PTS came
from. Keeping the whole history rather than one previous offset also fixes
the case a single `prev_offset_ns` cannot express at all: a frame from two
or more epochs back.
Measured on a real HD-DVD title whose second audio track put its final
packet at 12834.587 s in a 6434.100 s file. That frame sits 23 s below the
current frontier — outside the old window, so it was refused — but 0.15 s
below the end of the clip it actually came from. It now lands at 6417.216,
one 32 ms cadence step after its neighbour, and the track spans 6416.160
with no outsized gaps. The other two streams are unchanged.
A title that never rebases retires no epoch, so the lookup finds nothing
and the mapping is bit-identical to before.
Four parsers (dvdsub, flac, lpcm, mpegaudio) stamp a source offset on
every frame but had no test that read one back, so a regression to
`source: None` would have been caught only by the brace-balanced audit
in codec/mod.rs — a lint, not a behavioural check. Each now asserts the
emitted frame carries the offset of the packet that supplied its first
byte.
The Blu-ray feed spans had no direct test at all. Add one that walks a
multi-item playlist and requires the spans to tile the feed with no gap
or overlap; it catches a one-sector-per-clip drift, which is exactly the
error class that would misattribute frames near a seam.
`no_provenance_still_places_by_marks` asserted only that placement
returned something, which passes for a frame placed in the wrong clip.
Its probe timestamp lands in an overlap between two clips in the real
mark table, so pinning one clip would assert a coin-flip; instead
require the offset to be one that a clip actually containing that
timestamp would produce.
The TT_SRPT pointer at 0xC4 is a SECTOR offset from the start of
VIDEO_TS.IFO. Reading it as a byte offset lands 1/2048th of the way in and
does not error — it silently returns a different title list, which is how
a disc enumerating 38 titles and an image enumerating 10 looked like a
scan difference rather than a failure.
The table already had tests for its entry layout, dedup and the 99-title
clamp. What it did not have was anything pinning the offset unit itself,
or the grouping of titles by title set, or a truncated table keeping the
rows it can read rather than failing a whole disc.
release.sh dates the heading when it cuts the tag, and refuses a bare
'## [1.6.1]' because it cannot tell an undated heading from one it has
already stamped. Shipping a tag whose public changelog says UNRELEASED is
permanent and cannot be corrected inside that tag, so the check is right
to stop rather than guess.
Every emitted frame must carry the source byte offset of the packet it
came from. That held only for video for as long as it existed, and
nothing asserted it, so nothing caught it: ten parsers built frames with
`source: None` and a multi-clip title could not place audio or subtitles
by byte at all.
Finding them took a brace-balanced scan of the tree by hand — a regex
cannot do it, because a Frame literal contains nested braces and a
non-greedy match stops at the first `}`, which is how five sites survived
the first pass. This is that scan, as a test.
A second test fails if a codec module is added and not listed, since an
unchecked parser is exactly how the gap persists. Modules that emit no
Frame are named explicitly rather than skipped silently.
The guard tripped on itself twice while being written — first on its own
doc comment, then on its own string literals — so it strips line comments
and excludes its own module. A check that matches prose about the defect
rather than the defect is the same mistake in a different place.
The entries had grown into narrative paragraphs; each is now the outcome
first and the mechanism in a sentence or two. The website changelog page
mirrors libfreemkv's, so it inherits this directly.
Two things that shipped in this cycle were not in it at all:
dir:// as a source. It was pulled from 1.6.1 on 2026-08-05 because the
folder reader produced a wrong title list on a real DVD. That defect was
the CSS descramble bug below, fixed since — the acceptance suite now
shows a folder reporting the same 38 titles as its ISO, with matching
streams, languages and runtime, and the CLI has been wired for it all
along. It ships, so it is listed.
The CSS title-table corruption itself: a disc enumerating 38 titles
produced a decrypted image enumerating 10, silently, at exit 0. That is
a data-integrity fix a user needs to know about.
A real rip caught these. The warning added for a track placed from
timestamps under a seam plan fired once, on one subtitle track of a
23-clip title, eighteen minutes in — a display set that arrives as a lone
non-PCS segment is emitted straight through rather than accumulated, and
that path still built its frame with source: None.
Auditing the rest the same way found four more: flac, lpcm, mpegaudio and
the passthrough parser. All are one-PES-one-frame, so the packet's own
facts are the unit's facts.
Every Frame construction in every codec parser now carries a source,
checked by walking balanced braces rather than by eye — the earlier
count was taken with a regex that cannot see a block containing nested
braces, which is how these survived the first pass.
Marks that do not advance across a title are normal, not a defect: each
clip file carries its own STC, so one clip's IN has no ordering
relationship to the previous clip's. Refusing those tables dropped
exactly the branched titles this type exists for onto the inference path
that cannot read them, and that is the overrun.
Lifting it was tried once before every track carried a source offset, and
nine audio and subtitle tracks with nothing to place them by pinned
themselves to clip 0 and dropped most of the title. Every parser stamps
provenance now, so the clip comes from the byte offset and the marks keep
one job: whether a frame lies inside its own clip's [in, out].
Still gated on spans_trusted. Without usable spans there is no offset to
place by and inference is all that is left, so a table inference cannot
read is still refused rather than silently truncating a title.
A frame reaching the timestamp path under a plan now says so once per
track. It is not expected any more, and it is how a mostly-dropped track
reached a user without a single line in the log.
dts, ac3 and truehd assembled access units across PES packets with three
private implementations of the same bookkeeping, and none of them carried
the source byte offset. They now hold a PesBuf, so a unit takes the
timestamp AND the source of the packet covering its first byte, from the
same mark, and no codec can answer that question its own way again.
dts is the reference case: its pts_marks already implemented the rule
correctly, and all 61 of its existing tests -- including the PTS
attribution ones -- pass unchanged on the shared type. That is the
evidence the type preserves the behaviour dts had right.
ac3 kept a single carry-over timestamp and one anchor offset, so it could
only attribute the first unit in a call; it now resolves each unit at its
own offset. truehd had no attribution at all beyond a running clock.
New tests cover the case that motivated this: a unit whose first bytes
arrive in one packet and whose remainder arrives in the next keeps the
FIRST packet's offset. At a clip seam those two packets belong to
different clips, and taking the later one places the audio in the wrong
one.
Clippy on the pinned toolchain caught an empty `if` block left where dts
used to clear stale marks -- restored as an explicit clear, with why it
is still needed once drain keeps the covering mark.
A parser that assembles an access unit across PES packets has to answer
one question for every unit it emits: which packet carried this unit's
FIRST byte? Its timestamp comes from that packet, and so does the source
byte offset that says which clip of a multi-clip title it belongs to.
The packets that complete the unit carry later values that must not
override it.
That question was being answered three ways. DTS kept a deque of
(offset, pts) markers and took the one covering offset 0. AC-3 kept a
single carry-over timestamp. TrueHD kept its own. PGS and DVD subtitles
each held a pending unit with just a start time. None of them carried
the source offset at all, which is why provenance existed only for video
and why nine audio and subtitle tracks on a branched title had nothing
to place them by.
So it lives in one place now. PesBuf owns the bytes AND the marks;
PesFacts returns a packet's timestamp, source and discontinuity
together, so a parser cannot take one from one packet and another from
the next, because it does not assemble them itself.
The type answers WHICH packet. How a codec reads a timestamp out of that
packet stayed the codec's business at first, and that turned out to be
the same drift one level down: dvdsub read pts alone and returned 0 for
a packet carrying only dts, while everything else took pts.or(dts). Now
there is one derivation. It is not a choice between two fields — for
audio and subtitles there is no reordering, so dts IS the presentation
time and reading it is reading the same value from whichever field the
packet used. Reordering video never calls it; that path reconstructs
display order instead.
Migrated: adts, pgs, dvdsub. dts, ac3 and truehd follow.
Accepting a table whose marks do not advance was wrong, and a real
22-clip title showed exactly how. The rip failed the drop-volume gate at
84 percent with more frames dropped than kept.
The cause is a fact the code never stated: provenance is VIDEO-ONLY.
Every audio and subtitle parser -- dts, ac3, adts, truehd, pgs -- builds
its frames with `source: None`, while hevc, h264 and vc1 propagate it.
So on that title the video track placed correctly by byte offset and
nine audio and subtitle tracks arrived with nothing to place them by.
The branch added to keep an unprovenanced frame from stranding did the
stranding itself. It held such a frame on its track's current clip,
reasoning that the cursor only ever advances under provenance and so
could not be wrong -- which is true only for a track that eventually
receives some. A track that never receives any stays pinned to clip 0
for the whole title, and every frame past clip 0's OUT mark is dropped.
That is what the disc showed: nine tracks pinned to clip 0, the first
drop nine milliseconds past its OUT.
So the refusal goes back, now with the real reason recorded, and the
test asserts it rather than asserting the behaviour that failed.
Lifting it needs the audio and subtitle parsers to carry provenance
first, stamped from the PES that STARTED each access unit -- the same
rule au_assembly already applies to video. The diagnostics that found
this are kept.
The heuristic path has always logged its first drop per track with the
frame's timestamp and the clip marks it was judged against. The
provenance path did not log at all, so a title that dropped MOST of its
frames there failed the volume gate without a single line saying which
frame, which clip, or which marks — a full-length rip produced zero drop
events and an error, which is not a diagnosable failure.
Also pins the all-identical-spans hazard as a test. Every PlayItem
referencing one clip file gives every clip the same span, which passes
the tiling check and is therefore "trusted" while carrying no
information at all about which PlayItem a byte belongs to. That is not
what the discs on hand do — theirs have one distinct span per clip — but
the check conflates "the spans tile" with "the spans distinguish", and
only the second justifies placing a frame by its byte offset.
A seamlessly branched title re-references one clip file from several
PlayItems with different mark ranges, and those references share a single
feed span because the bytes are read once. A byte offset alone cannot
then tell them apart.
Whether that is happening on a given title is the fact that decides
whether provenance can identify a clip on its own, and it was not
observable. distinct_spans < clips says it directly.
Whether a title was placed by the seam plan or fell back to inference is
the single most useful fact about a branched rip, and it was invisible.
The two looked identical in the output, so telling them apart meant
rebuilding and re-ripping — which is exactly what it cost to find that
most branched discs were silently on the inference path.
Log both flags once at plan construction, and log each refusal with the
reason that caused it.
A playlist whose clips each restart their own STC has marks that all
cover the same low values, so they are not points on one title-wide
clock. from_clips refused those tables outright and fell back to PTS-jump
inference.
That refusal predates provenance, and it is only about inference. With
the feed spans tiling the title the clip comes from the frame's byte
offset and the marks are never read across clips: each clip's offset_ns
maps its own private clock onto the output timeline, which is exactly
the right operation for a restarting STC.
Refusing unconditionally therefore turned the seam plan off on most of
the branched discs on hand — the very titles it was written for. They
ran on inference with a timeline minutes past the title's real length:
one 2h29m title came out with its audio and video both spanning 3h15m.
So the refusal now applies only when the spans cannot be trusted, where
it is still load-bearing: without a usable clock or a usable byte offset
there is nothing to place with.
Accepting these tables opens one hole, and it is the same stranding the
refusal guarded against. A frame arriving with no byte offset would fall
through to the mark heuristics, which on this kind of table are
meaningless and can strand a track on a clip it has already left,
dropping the rest of the title. Such a frame now stays on its track's
current clip; the cursor only ever advances under provenance, which is
never wrong about which clip a byte came from.
The provenance lookup runs once per frame per track. A linear scan over the
clip list is fine for the 11-PlayItem fixture and is not fine for the real
hoard: two discs there carry 900 PlayItems in one title, and several carry
250-450, so a scan would be hundreds of comparisons on every one of millions of
frames.
spans_trusted already guarantees the spans tile the feed contiguously and in
order, which makes this a partition point. A repeated clip reuses its first
reference span, so the search walks back to the FIRST entry sharing it — the
bytes are read once, so the material is emitted once, at that entry offset, and
the answer is stable regardless of which duplicate the partition lands on.
Also adds a test at the real 900-clip scale: every clip places, output stays
monotonic across all 900, an arbitrary byte resolves to the right clip out of
ascending order, and a byte past the end belongs to no clip.
Four audit rounds each fixed one rule in SeamPlan::place and broke another,
because the question the rules were answering has no answer. Inside a
seamless-branching overlap clip k OUT comes AFTER clip k+1 IN — 57.8s of
overlap on the real fixture table — so a single timestamp is legitimately
inside two clips, and a clip file is not trimmed to its marks, so it also
carries material from before its own IN. No rule over timestamps can say which
clip a frame came from, and each attempt was right for one disc layout and
silently wrong for another: 65s of rewind, 17 minutes stranded, 28 minutes
dropped, 55s refused.
Frames already carry the byte offset they were read from (PesFrame::source,
stamped by the TS demuxer). Clip now carries the byte span its stream occupies
in the title feed, recorded while the extents are gathered. So the clip is a
LOOKUP: the offset falls in exactly one span. There is no decision to get wrong.
Every track of a clip lives in the same stream file and therefore shares one
span, so video, audio and subtitles agree by construction. Divergence between
them — each track guessing separately under its own tolerance — is how audio
and video ended up on different clips and drifted apart in the first place.
spans_trusted gates the whole path: unless the spans tile the feed contiguously
from zero, an offset means nothing and provenance is ignored in favour of the
mark heuristics, which is the 1.6.0 behaviour. A broken map degrades instead of
confidently selecting a wrong clip for every frame. A clip referenced twice
reuses its first span (the bytes are read once) and is still trusted.
Sources that stamp no provenance — a mkv:// remux, the deserialize hop — take
the heuristics, which is what they have always used and where they have always
been right, because they have no overlapping clips to be ambiguous about.
36 timeline tests, six of them new and covering: the overlap case marks cannot
see, all tracks agreeing, out-of-marks material dropped AND counted, a holed
span map, a discontiguous one, a repeated clip, and no provenance at all.
A symlink to a directory — a normal way to keep tens of gigabytes of streams
off the system disk — was dropped from the plan silently. `entry.file_type()`
reports the LINK, so such a subtree never enters `dirs`, and `metadata()` then
follows it and reports a directory, so it fails the is_file() test and was
skipped with nothing said.
The result is the worst class this release exists to close: PLAYLIST and
CLIPINF still synthesize, so the folder scans and enumerates titles, every clip
resolves to no extents, the mux takes its clean-EOF path, and a near-empty MKV
is written at exit 0.
It is still skipped rather than followed — following link targets invites
cycles and escapes from the folder — but at the same volume as the
unrepresentable-name skip eight lines above.
Round 7 enumerated the state space of SeamPlan::place and found three more
holes in the mark heuristics — the third consecutive round to find a defect
here, each time in the guard the previous round added:
- a clip file opening ON its own IN mark, after a tail frame past the previous
clip OUT had already advanced the cursor, read as another crossing and jumped
a clip too far: 28 minutes of clip 6 dropped;
- the last clip was unguarded entirely (`clip + 1 < len` skipped the check), so
the 9->10 seam rewound 75.6s with dropped_total() at 0 — the exact round-5
signature, reachable without any corrupt input;
- a glitch mid-clip satisfies "inside the current clip", so the round-6
stranding survived for the rest of that clip.
Three patches to the same heuristic, three wrong in a new way. So stop patching
it. Everything above the placement is still heuristics over marks, but the
property they exist to serve is now checked DIRECTLY: a track output must not
run backwards. A placement that would rewind by more than
DISCONTINUITY_BACKSTEP_NS is refused; the scan then looks for a LATER clip
containing the frame that does not rewind, which is exactly what a genuine
overlap crossing looks like; and if none exists the frame is dropped and
counted, because emitting it anywhere moves the track backwards.
The tolerance is the backstep constant because B-frame reorder legitimately
emits out of order by a fraction of a second, while a rewind that matters is
orders of magnitude larger.
The cursor moves ONLY when a later clip actually accepts the frame. My first
version of this advanced on a failed search, which strands a track exactly as
before — caught immediately by the round-6 regression test.
All 29 previous timeline tests still pass, including
continuity_preserves_forward_gap. The three round-7 scenarios now have a test
each, with the real 00801.mpls numbers.
The round-5 fix advanced the clip cursor on ANY backward step over 3s. That
also fires for a corrupt PTS, and for a legitimate STC discontinuity inside one
clip — and nothing moves the cursor back, because a forward step matches
neither past_out nor stepped_back. Every later frame then sits below the new
clip IN and is dropped: on the fixture table that is ~17 minutes of one track
gone, and the only volume gate compares total drops against ALL tracks frames,
so it exits 0.
The branch now also requires the frame to be INSIDE the current clip marks,
which is the only case that would otherwise be silently placed at the old
offset — the rewind. A frame outside them needs no help: the containment check
drops and counts it, the cursor stays put, and the next good frame is placed
normally. Both behaviours have a test, each confirmed to fail without the guard.
Separately, the dir:// PES input path scanned the folder and never applied the
encryption verdict scan_dir exists to produce, so the same folder ripped
through one door and failed through the other asking for a key it does not
need. That logic now lives in one function, session::apply_folder_encryption_verdict,
called by both.
The previous commit bounded the crack scan inspection by the bytes actually
read, which stopped it examining stale buffer bytes from an earlier batch. But
the cursor still advanced by the REQUESTED count, so those sectors were skipped
outright — trading "scans the wrong data" for "silently scans less than it
thinks", on exactly the damaged media where a title key is hardest to find.
The cursor now advances by what was read, so the next iteration resumes where
the read stopped. Floored at one sector so a source returning Ok(0) cannot spin.
Found by reading the fix again rather than by the next audit round.
read_sectors returns the number of bytes written and a source may return Ok
with fewer than asked — a recovery read over a damaged region does exactly
that. The crack loop discarded the count and inspected all n sectors of a
buffer that is REUSED across batches, so the tail still held the previous
batch sectors.
Cracking a key from those means cracking from data belonging to a different
extent, possibly a different VTS, while crack_span records the CURRENT one. A
key that opens nothing in this region is then installed, and its wrong
descrambles are only partly caught by the per-sector crib.
descramble_region was fixed to require the MPEG-2 pack start code, but
descramble_sector — the PUBLIC per-sector entry point, and the one the
module-level example tells callers to use — still keyed on the byte 0x14 flag
bits alone. The crate own documented guidance therefore led straight back into
the defect this release exists to fix: a VIDEO_TS.IFO sector holding 0x15 at
0x14 while starting 00 26 00 00 loses 1912 of its 2048 bytes, and because that
sector carries TT_SRPT the disc enumerates 38 titles while an image decrypted
from it enumerates 10, at exit 0.
It has no callers inside the crate, which is exactly why it survived three
rounds: nothing exercised it. Putting the guard inside the function rather than
in each caller is what keeps the safe path the easy one.
The integration test that covers it built its sector from the flag byte alone,
which no real scrambled sector looks like, so it stopped representing the path
it names — the same fixture-realism gap already fixed in four other places this
release.
A hole in the seam fix itself, found in audit round 5 and confirmed by
measurement against the real 00801.mpls marks already in the test file.
Nothing upstream trims a clip stream to its marks, so a clip file can open with
material from BEFORE its IN. With the cursor on clip 5 (7708.99..7910.79) a
frame at 7845.00 — clip 6 pre-mark lead-in, 8s below clip 6 IN of 7853.00 —
sits further from that mark than the 250ms tolerance, so neither crossing rule
fires. The cursor stays on clip 5, and 7845.00 IS inside clip 5 range, so the
frame is PLACED with clip 5 offset.
Measured: output went from 3714.51s to 3649.51s. Backwards 65 seconds, with
dropped_total() still 0 — no counter, no gate, nothing noticed — while the
whole 57.8s overlap band was emitted a second time over clip 5 written tail.
That is the collide-and-flatten symptom this type exists to remove.
A track own PTS only ever runs forward inside a clip, so a backward step larger
than DISCONTINUITY_BACKSTEP_NS is always a clip change. Such a frame now
advances to the first clip that could contain it — which terminates on its own,
rather than running to the end of the list, which is what the next_in bound
exists to prevent — and a frame still below that clip IN is material the
playlist excludes, so the containment check drops and COUNTS it.
All 27 pre-existing timeline tests still pass, including
continuity_preserves_forward_gap, which is the one that catches an over-eager
heuristic here. The new test asserts the output never moves backwards at a
join and was confirmed to FAIL before the fix.
The same gap as the MKV muxer: DemuxSink builds its timeline with
TimelineContinuity::with_clips, but no test ever handed it a title carrying
usable PlayItem marks, so frames_mapped and dropped_total() were always zero
and both gates in finish() were unreachable.
Deleting either left the whole suite green while a demux:// export of a
seamless-branching title wrote a directory of zero-byte track files beside a
populated chapters document and reported success at exit 0.
The test gives the title two clips with marks that exclude every frame, and was
confirmed to FAIL with the gate removed.
Audit finding: both seam gates in MkvMuxer::finish were dead code under test.
set_clips was never called anywhere in the suite, so continuity.dropped_total()
was always zero and neither branch could be reached.
The ordering of the two zero-frame checks is load-bearing and the comment says
so: a title the seam plan emptied also has a zero frame count, and MkvInvalid
is classified by is_skippable_title_stub as an empty nav/menu stub — so
reporting it that way makes an all-titles rip drop a real feature and finish the
rest at exit 0. Swapping the checks left the entire suite green.
The new test builds a two-clip plan (from_clips needs at least two, strictly
increasing) whose marks exclude every frame, and asserts both that the error is
SinkWroteNothing and that it is NOT classified as skippable. Confirmed to FAIL
with the two checks swapped.
Auditing my own fixes found all three. None were in the original code.
The VTS crack sort was byte-wise case-SENSITIVE while the filters that select
those files (vts_group_of / is_title_vob) are case-insensitive. On a
case-sensitive volume a set holding vts_01_1.vob beside VTS_01_2.VOB sorted
part 2 first, because V (0x56) precedes v (0x76) — reintroducing exactly the
budget-exhaustion the ordering exists to prevent. Now sorted on the same
uppercase normalisation the filters apply.
Refusing a name that round-trips to empty aborted the WHOLE plan. A sidecar
folder named with a single emoji made a backup un-rippable that 1.6.0 handled
fine, and reported it as a collision with a file that does not exist. It is now
skipped with a warning: the entry is unaddressable either way, but one
irrelevant file should not cost the user their rip.
The mtime check now applies only to files whose CONTENT the plan read — the
IFOs, whose bytes 0xC0/0xC4 place every VOB. Everything else is planned from
size alone, which is already checked, so comparing mtime there bought nothing
and risked a real false positive: disc backups commonly live on exFAT/FAT32,
which stores local time, so a long rip spanning a DST transition would see a
whole-hour shift on an untouched multi-gigabyte VOB and abort hours in.
Two dirimage findings, both silent-wrong-output.
The per-directory uniqueness check compared raw host names, but the reader does
not see raw host names: parse_udf_name trims leading and trailing whitespace
and drops any code unit char::from_u32 rejects — which is every half of the
surrogate pairs the encoder emits for non-BMP characters. So " 00000.m2ts" and
"00000.m2ts", or "A<astral>.m2ts" and "A.m2ts", were two entries at plan time
and ONE name at read time. find/read_file take the first match, so a title
resolved to the wrong file extents and muxed the wrong bytes at exit 0 — the
exact shadowing DirNameCollision exists to prevent.
The key is now derived by round-tripping the name through the very encoder and
parser that will be used, so it cannot drift from them. A name that survives
that round trip as empty is refused outright: it would exist in the image and
be addressable by nothing. The new test builds a real folder, calls plan, and
was confirmed to FAIL with the fix reverted.
Separately, the plan-vs-read revalidation compared file LENGTH only, while the
plan depends on CONTENT: a DVD VOB placement comes from bytes 0xC0/0xC4 of its
IFO, and IFOs occupy a whole number of sectors so an in-place rewrite keeps the
length. A re-authoring tool touching the folder mid-rip would pass the size
check while every title extent pointed at stale sectors. mtime is now compared
alongside size, and only when both sides report one, so a filesystem without
timestamps falls back to the old behaviour rather than failing every read.
probe_folder_encryption sampled AACS units starting from the largest extent
anywhere in the disc. AACS units are 3 sectors and a unit boundary is only
guaranteed at the START of a clip, so that anchor is only correct when the
largest extent happens to be a clip first extent.
For any clip over ~2 GiB it is not. The planner caps an allocation descriptor
at MAX_AD_BYTES = 524287 sectors, so every full piece of a split file ties on
sector_count and max_by_key returns the LAST tie — an extent beginning
(k-1)*524287 sectors into the file. 524287 % 3 == 1, so that start misses the
unit boundary for two file sizes in three.
The 6144-byte sample windows then begin mid-source-packet and the byte read as
the CPI flag is content. Both verdicts fail in a costly direction: a decrypted
folder is rejected as encrypted (DirImageEncrypted on something perfectly
rippable — the exact case scan_dir was added to rescue), or real ciphertext
reads as clear and the mux writes it out as video at exit 0. is_unit_aligned
cannot catch it: it measures against the same wrong base.
Now the largest TITLE first extent, which is unit-aligned by construction and
is also the more meaningful sample — the main feature rather than whichever
fragment happened to be biggest.
The existing tests could not reach this: their fixture m2ts is 786,432 bytes,
a single extent, which is always its own first.
Round 1 removed a largest-first sort from resolve_vts_key because
largest-first is the 1.5.1 garbage bug. Auditing that fix showed it was only
half right: `planned` comes from walking the UDF directory, which yields File
Identifier Descriptors in on-disc authoring order with nothing sorting them. So
deleting the sort did not restore playback order, it left the order undefined —
whatever the disc happened to list first.
DVD-Video numbers a title sets VOBs in playback order by spec (VTS_xx_1.VOB ..
VTS_xx_9.VOB, single digit), so ascending filename IS playback order and is
deterministic regardless of how the directory is laid out.
Order decides correctness here: crack_key shares one sector budget across the
whole extent list, a CSS DVDs biggest cell opens with a long clear run, and CSS
recovers the title key from scrambled data itself. Starting in the wrong place
can exhaust the budget without ever meeting scrambled data, whereupon the
caller falls back to the disc-wide key and the whole VTS is descrambled wrongly
— corrupt PES behind an intact header, written out at exit 0.
Four fixes from the first audit round. Every finding was verified against a
pinned tree and read directly before being accepted.
resolve_vts_key sorted a VTS title-VOB extents largest-first. That is the 1.5.1
garbage bug, and it grew back in a new code path: the comment claimed it
"matched the scan heuristic", but that heuristic WAS the bug and had already
been fixed in decrypt_keys_for_title, which documents the rule (PLAYBACK ORDER,
never largest-cell-first) and pins it with a regression test. A CSS DVDs biggest
cell opens with a long clear run and crack_key shares one sector budget across
the extent list, so starting there can exhaust it without ever MEETING
scrambled data — and CSS recovers the key from scrambled data itself. The crack
then returns None, the caller falls back to the disc-wide key, and every VOB in
that VTS is descrambled wrongly: corrupt PES behind an intact header, written
out as a complete extract at exit 0.
parse_pgcit dropped titles silently in THREE places — an unparseable PGC, an
out-of-range PGC index, and a truncated entry table. The finder caught one; the
other two turned up on reading the function. parse_vmg already counts and warns
per skipped title SET for exactly this reason, and this was the last place a
disc could quietly report fewer titles than it has.
write_image called flush() and returned Ok. flush() only pushes bytes into the
page cache and promises nothing about durability, so a 6-90 GB image could be
reported complete while still unwritten — a crash or an unmounted volume then
leaves a truncated file the caller was told was finished. Now into_inner (so a
buffered-write error surfaces instead of being dropped by BufWriter::drop)
followed by sync_all.
timeline used abs() on a saturating_sub result. Every other comparison in that
module is saturating because the timestamps come off a disc and are not
trusted; abs() panics on i64::MIN, which saturating_sub can produce.
The 1.6.1 DVD fix removed the USE of the loose predicate but left the
predicate sitting there with the better name. Anyone asking "is this sector
scrambled?" finds is_scrambled before is_scrambled_pack, and reintroduces the
defect that destroyed 1912 bytes of a real VIDEO_TS.IFO — the sector carrying
TT_SRPT — so the disc enumerated 38 titles and an image decrypted from it
enumerated 10, silently, at exit 0.
Byte 0x14 only means "scrambling control" inside an MPEG-2 pack. In an IFO,
UDF or ISO 9660 sector it is whatever that format stores there. The new name
says what the function actually tests and nothing more, so the honest question
has the obvious name and the dangerous one has to be asked for deliberately.
Its doc comment also claimed a caller, decrypt::decrypt_sectors, that does not
exist — so the name was an invitation and the documentation was an argument
for accepting it. It has no production callers at all; it stays public because
an integration test asserts the flag extraction directly.
Matthew Jackson