libfreemkv/docs/paramount.md

# Paramount/onQ `playlists.xml` — derivation notes

`/BDMV/JAR/` is application-defined space, so this format has no spec to
look up. Every field meaning below was derived by measuring real discs and
cross-checking against per-display-set content. Treat an unfamiliar value
as unknown rather than guessing — the disc's own `forced_on_flag` is the
only authoritative forced signal.

## `ForcedSub` — the `forced_sub` CSV cell values

The attribute reads like a boolean and was once parsed as one (`cell ==
"1"` → forced). It is not. Every image in the corpus carrying this
vendor's `playlists.xml` — seven distinct discs — uses four values, and
decoding three of those discs' feature subtitle tracks and counting every
PGS display set separates them into two populations two orders of
magnitude apart:

* `0` — a subtitle track with no forced-narrative content. On the two
  discs measured that use the flag at all, not one `0` track carried a
  single `forced_on_flag` display set.
* `1` — a FULL DIALOGUE track that additionally contains some
  forced-narrative signs. On one measured disc, all nine `1` cells are
  full tracks of 949-1411 display sets, eight of them carrying 5-14
  flagged sets and the ninth none; that disc has no dedicated forced
  track at all. On another, all seven `1` cells are full tracks of
  1602-1651 display sets carrying 0-31 flagged sets. Reading `1` as
  forced is what made one language present as two identical full
  subtitle tracks with one of them flagged forced.
* `2` and `3` — a DEDICATED forced-narrative track. These take their own
  trailing STN slots, one per localized language, duplicating a language
  that already holds a full track earlier in the list. Measured: the two
  `2` slots on one disc are 15 and 10 display sets, EVERY one flagged
  forced, against ~1600 on that disc's full tracks; the four `3` slots on
  another are 7, 14, 23 and 59 display sets against 1216-2655. What
  distinguishes `2` from `3` the corpus does not reveal — both sit in the
  same trailing position, both measure the same shape, and one disc uses
  each for a different language — so both map alike.

So the old reading was wrong in BOTH directions: it flagged full dialogue
tracks forced, and it discarded the cells that name the real forced
tracks.

The `1` case is deliberately NOT carried through as a weaker "contains
forced segments" hint. There is no qualifier for that, and the asymmetry
argues against inventing one here: a wrong forced flag on a 30 MB
dialogue track is the user-visible defect, while a missing hint costs
nothing.

Nothing downstream can undo a wrong forced label on the discs that need
it most: `mux::codec::pgs::demotable` may only clear a vendor forced
label where some track on the disc demonstrably sets `forced_on_flag`,
and measured discs using this label format never set it.

## `MAX_COM_INDICES` — the CSV addressability bound

The highest CSV cell position that can ever be addressed. The labelling
loops number cells 1-based into a `u16` and `break` at
`u16::try_from(i + 1)`, so cell `MAX_COM_INDICES` and everything past it
is never visited. Two different things are measured against that bound,
and they are not the same:

* A VALUE at or beyond it cannot match any cell. This caps the set:
  values are filtered before insertion, so at most `MAX_COM_INDICES`
  distinct entries can ever be stored, however long the attribute is.
  The `HashSet` that replaced a linear scan fixed the LOOKUP cost; this
  fixes the ALLOCATION, so a disc declaring half a billion indices no
  longer costs half a billion entries.
* A POSITION at or beyond it describes nothing new. This caps the WORK,
  not the memory — the value filter already made the set small, but
  without it every one of those half a billion cells is still split and
  parsed. It is an early exit at the first position whose contents
  provably cannot matter, and it is why `forced_sub` — which holds no
  set at all, and so gets no protection from the value rule — is bounded
  too.

Real authoring is nowhere near either limit: the BD STN table admits at
most 32 streams per playlist, so nothing legitimate is lost.

## `com_indices` / `forced_subs` — why they are separate functions

Extracted so each bound is independently observable in a unit test.
Asserting the bound through `labels_from_feature` is not possible in
either case: for `com_indices`, a `HashSet` collapses repeated values and
an out-of-range index changes no label either way, so a label-level test
passes whether or not the cap exists. For `forced_subs`, the Vec is read
only as `forced.get(i)` from a loop that stops at `MAX_COM_INDICES`, so
cells past that are unreachable by construction and, again, unobservable
through the labels. Returning the raw set/Vec lets a test hand in tens of
thousands of distinct unaddressable entries and see them refused.

## `labels_from_feature`

Split out from `parse` so the per-type numbering and commentary/forced-
index logic is unit-testable without a `SectorSource`/`UdfFs`.

## Test: `a_playlists_stream_list_cannot_run_into_the_next_playlist`

Immunity pin, section-boundary half. The pixelogic parser walks a flat
string sequence and recognises its feature section's END by marker
alone, so a section with no marker behind it runs off into whatever
follows and counts it as more STN slots. Nothing here can do that: the
stream list is one attribute of one XML element, so its length is the
CSV's own cell count and its scope is the element's byte range that
`xml::find_element` returns. Text after the element — including the next
playlist's own `aud` — is not reachable from it.

And when the boundary is MISSING the failure is closed, not open:
`xml::find_element` needs a matching close tag and yields `None` without
one, so an unterminated element ends the walk rather than swallowing the
rest of the document.

Mutation: hand `labels_from_feature` the document instead of the
element, or let an unterminated element run to EOF → the bonus
playlist's languages join the feature's stream list.

## Test: `bounding_the_parse_does_not_change_a_legitimate_playlist`

`sub_com1_idx` is parsed straight out of the disc's `playlists.xml`,
which is attacker-controlled and has no length bound of its own.

This replaces a WALL-CLOCK test. That one built a 200 000 x 1 000 001
fixture and failed if it took over 10 s, to prove the membership test
was a set rather than a linear scan. Measured on the machine that wrote
this: 1.62 s alone, and OVER 10 s — a real failure — when the suite's
other 3 347 tests were running concurrently. A 6x margin against a
shared CPU is not a margin; it is a CI failure that looks like a flake
and gets re-run until it passes.

It also measured the wrong thing. Making the lookup O(1) bounded the
QUERY, not the PARSE: the set was still built from every entry the disc
declared, so a hostile playlist could still force an unbounded
allocation before any lookup happened. `MAX_COM_INDICES` bounds that.

What this test guards is that bounding did not change what a legitimate
playlist MEANS: it goes red if the bound is set too LOW (verified at 2 —
the real indices `0,2,4` stop resolving and the purposes change). It
does NOT go red if the bound is deleted entirely, because the
out-of-range filler is unobservable at the label level and a `HashSet`
collapses the repeats. Enforcement is proven separately, by
`distinct_unaddressable_indices_are_refused_not_stored`, which reads the
set itself. Two tests, two properties; neither pretends to the other's
job.

## Test: `distinct_unaddressable_indices_are_refused_not_stored`

The set REFUSES unaddressable indices, so a hostile playlist cannot
inflate it. DISTINCT values on purpose: a `HashSet` collapses repeats,
so a million copies of one index costs one entry and would prove
nothing. Fifty thousand distinct out-of-range indices cost fifty
thousand entries without the filter, and none with it — so this test
goes red if the bound is removed, which the label-level assertions
elsewhere cannot do.

## Test: `forced_sub_cells_past_the_last_addressable_one_are_not_parsed`

`forced_sub` is bounded too — the third CSV in the same function, and
the one that had no value filter to hide behind. Read through
`forced_subs` rather than through the labels for the same reason the two
tests above read the set: the subtitle loop stops at `MAX_COM_INDICES`,
so a label-level assertion cannot tell a bounded parse from an unbounded
one.

## Test: `an_index_that_cannot_address_any_cell_is_not_retained`

Asserted through `com_indices`, not through the labels: the labelling
loop never queries a cell position that high, so at the label level
retaining the value is unobservable and the assertion could not fail.
Reading the set is what makes the claim checkable.

## Test: `empty_csv_slot_still_occupies_its_stn_slot`

The `aud` / `sub` CSVs are the vendor's STN-ordered stream lists: one
slot per stream, and `aud_com1_idx` / `forced_sub` are indexed against
those same slot positions. A slot whose language cell is empty carries
nothing to label but still OCCUPIES its slot, so it must not renumber
the slots behind it.

Numbering only the slots that carry a language collapsed every later
label one position forward per empty cell, which is how a forced marker
authored for one STN slot lands on the full-subtitle track in front of
it.

## Test: `a_contains_forced_segments_cell_is_not_a_forced_track`

`forced_sub` is an enumeration, and `1` is its "full dialogue track that
also carries forced signs" value — NOT "this track is forced". Measured
on a disc whose feature declares nine `1` cells among 32 subtitle slots:
all nine are full dialogue tracks of 949-1411 display sets, and the disc
has no dedicated forced track at all. Reading `1` as forced is what
produced two identical full subtitle tracks for one language with one of
them flagged forced. See "`ForcedSub` — the `forced_sub` CSV cell
values" above for the full corpus measurement, including why nothing
downstream can undo this.

## Test: `a_dedicated_forced_narrative_cell_is_a_forced_track`

`2` and `3` are the cells that DO name a dedicated forced-narrative
track, and the old boolean reading discarded both. Measured: these cells
occupy their own trailing STN slots, one per localized language,
duplicating a language that already holds a full track earlier in the
list. On one measured disc the four `3` slots carry 7, 14, 23 and 59
display sets against 1216-2655 on the full tracks they duplicate — and
not one display set anywhere on that disc carries `forced_on_flag`, so
neither the scan probe nor the muxer can promote them from content. The
vendor cell is the only evidence there is.

## Test: `an_unrecognised_forced_sub_cell_is_not_forced`

An unrecognised cell must fall to NOT forced. Asserting forced is the
expensive mistake (a full dialogue track a player then burns on screen),
so an unknown value from a future authoring revision must not be able
to make that claim.

## Test: `find_feature_first_wins_on_audio_count_tie`

On a tie in audio-slot count, the FIRST playlist encountered wins
(consistent with `select_result`'s first-wins tiebreak elsewhere in the
registry) — later playlists only displace the current best on a
STRICTLY greater count.