diff --git a/CHANGELOG.md b/CHANGELOG.md index e39efe03..39394d37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ ### Output-format changes +- **MKV DefaultDuration of MPEG-2 video** follows the kept frames' measured period when that is another standard rate: a soft-telecined NTSC film title (DVD) declares 23.976 fps instead of the sequence's 29.97, so the frame count agrees with the duration. Video at its sequence rate is unchanged. - **LPCM** (BD and DVD) is decoded to interleaved big-endian PCM in WAVE_FORMAT_EXTENSIBLE channel order (BD pad channel removed, LFE/surround remapped, DVD 20/24-bit groups unpacked) at the source depth: 16-bit stays 16-bit, 20- and 24-bit sources output 24-bit; MKV `BitDepth` matches, and M2TS/MPG output keeps the depth. BD 20-bit packets are kept (24-bit container); reserved header codes are dropped. On M2TS output LPCM is repacked to the BD layout; LPCM BD cannot carry (e.g. 44.1 kHz) is dropped and reported. - **M2TS:** reordered video (H.264, HEVC, MPEG-2, VC-1) now carries a DTS (H.222.0 2.7.5); PTS round to the nearest 90 kHz tick (a PTS can be 1 tick later than 1.7.7), the first frame lands one second above zero, PCR never decreases with B-frames; output always declares `bd-ts` in its FMKV header. PGS `.sup` ticks use the same rounding. - **MKV:** DTS, DTS-HD HRA and DTS-HD MA are all `A_DTS`; streams with no Matroska CodecID (text subtitles, unknown codecs) are omitted rather than declared under another codec; SDR BT.2020 video is tagged with the BT.2020 transfer (14), not PQ; the timeline origin is the earliest sample of any selected track (audio and subtitles before the first video keyframe are kept at their true offsets; video starts at its first keyframe), and Blu-ray clip IN applies only to BD-TS sources, whose frames before the IN are dropped and counted; VobSub cues without a stop command end at the next subtitle (10 s cap) instead of 30 s. **MP4:** a track that starts after the earliest sample keeps its offset through an empty edit, and video before its first keyframe is not stored. @@ -38,7 +39,7 @@ - **`json://`** audio `sample_rate` is a number in Hz (`null` when unknown), not a string such as `"48kHz"`; a `sample_rates` array lists every rate a stream carries. - **FMKV (`network://`, `stdio://`) header version 2** is used when a track carries decoder timing (Opus CodecDelay/SeekPreRoll) and adds per-frame DiscardPadding; streams without timing stay version 1. Headers now also carry display aspect, CICP, stream purpose/qualifier, chapters and content format, and PIDs actually written by M2TS. A header declaring more than 256 streams is refused with E9008. An older receiver omits the MKV Channels element for the new channel-layout strings; keep both ends on one version. - **`dir://` image bytes:** the PVD declares interchange level 3, Unique IDs start at 16 and the volume id round-trips through the d-string encoder; `dir://` sources are stamped as images in the `fvi://` provenance header. -- **One output title name:** a title read from a disc layout (`disc://`, `iso://`, `dir://`, a staged image, a server live rip) is named by the disc's meta title (else its volume id) in every container's title field and in `fvi://` provenance; 1.7.7 named image and server-live rips by the MPLS playlist file. `fvi://` provenance of an `iso://`/`dir://` source now carries the volume id. +- **Output title names:** a title read from a drive (`disc://`, a server live rip) is named by the disc's meta title (else its volume id) in every container's title field and in `fvi://` provenance; one read from an image or folder (`iso://`, `dir://`, a staged image) by the disc's meta title, else its own playlist name as in 1.7.7, so an image and its extracted folder name a title alike. 1.7.7 named server-live rips by the MPLS playlist file. `fvi://` provenance of an `iso://`/`dir://` source now carries the volume id. - **TrueHD 7.1 / Atmos labels on every path:** the playlist labels a 7.1 or Atmos TrueHD track 5.1 at 48 kHz; every mux (live disc, staged image, `iso://`/`dir://`, containers) now completes the track's channels, sample rate and label from its first major sync before the output opens, and `json://` / `chapters://` get the completed title too. 1.7.7 corrected only `iso://`/`dir://` URL rips, so live and staged-image rips now label such tracks 7.1 / Atmos. - **Track selection applies to container inputs:** `selection` keeps a subset of the tracks of an `mkv://`, `mp4://`, `m2ts://`, `network://` or `stdio://` input (renumbered in order, video always kept), as it already did for disc titles; an unknown PID is refused at open. - **A live DVD extracted to `dir://` is descrambled:** `Disc::extract_tree` finds and cracks each scrambled VTS from its content on every DVD-Video layout. A live drive scan records no disc-wide CSS key, so a live DVD's title VOBs were written still scrambled. @@ -95,6 +96,7 @@ ### Fixed +- Every AACS format takes one encryption decision in `KeyRing::acquire`: content its own detector reads in the clear needs no key and asks no source, whatever key files the disc declares (an HD DVD copy in the clear declaring several keys, and an FMTS disc whose forensic segments are also clear, no longer ask for or refuse on keys); encrypted content is keyed by the format's rules and refused (E7022 / E7032) when no source has the key. - A zero read batch (`MuxOptions::default()` carries `batch_sectors` 0) is refused with `Error::MuxBatchSectorsZero` (E9085) on every path (live, session, ISO, URL) instead of spinning until Stop. - A stopped or failed read can no longer finalise a truncated MKV / M2TS as a complete title; `extract` after a drive Stop leaves the file `.partial` instead of finalising zeros. - DVDs with MPEG audio (common on PAL) ripped an empty track; anamorphic DVDs got wrong or missing subtitle languages; a partial angle-cell filter dropped cells. diff --git a/src/aacs/content.rs b/src/aacs/content.rs index d8e21bf5..8b9d66cd 100644 --- a/src/aacs/content.rs +++ b/src/aacs/content.rs @@ -1451,6 +1451,26 @@ mod tests { } } + // The CPI clear is BD-TS only: an HD DVD program-stream unit has no TP_extra_header and + // its byte 0 of each 192-byte stride is payload. + #[test] + fn clear_copy_permission_indicator_masks_bdts_and_leaves_mpegps_alone() { + let before = vec![0xFFu8; ALIGNED_UNIT_LEN]; + let mut ps = before.clone(); + clear_copy_permission_indicator(&mut ps, crate::disc::ContentFormat::MpegPs); + assert_eq!(ps, before); + let mut ts = before.clone(); + clear_copy_permission_indicator(&mut ts, crate::disc::ContentFormat::BdTs); + for (i, b) in ts.iter().enumerate() { + let want = if i % BD_SOURCE_PACKET_BYTES == 0 { + 0x3F + } else { + 0xFF + }; + assert_eq!(*b, want, "byte {i}"); + } + } + // ── is_clean / ts_sync_count edge cases ──────────────────────────────── #[test] diff --git a/src/aacs/crypto.rs b/src/aacs/crypto.rs index 55908783..1871c8ec 100644 --- a/src/aacs/crypto.rs +++ b/src/aacs/crypto.rs @@ -32,7 +32,8 @@ pub(crate) fn new_cipher_for(key: &[u8; 16]) -> Aes128 { } /// Build an AES-128 key schedule. The single construction site for the CBC -/// helpers, so `KEY_EXPANSIONS` can count them under test. +/// helpers, so `KEY_EXPANSIONS` can count them under test; the single-block ECB +/// helpers build their own and are not counted. fn new_cipher(key: &[u8; 16]) -> Aes128 { #[cfg(test)] KEY_EXPANSIONS.with(|c| c.set(c.get() + 1)); diff --git a/src/aacs/derive.rs b/src/aacs/derive.rs index 858da03e..e4174010 100644 --- a/src/aacs/derive.rs +++ b/src/aacs/derive.rs @@ -386,8 +386,8 @@ pub(crate) fn resolve_dk_node( } /// Public, side-effect-free accessors over the MKB record helpers, exposed so -/// independent reproduction harnesses (e.g. `examples/prove_hkd_aacs.rs`) can -/// exercise the exact same parser + verify primitives the production walk uses. +/// independent reproduction harnesses can exercise the exact same parser + verify +/// primitives the production walk uses. /// These are thin wrappers — no new logic. #[doc(hidden)] pub mod probe { @@ -452,7 +452,7 @@ pub fn decrypt_unit_key(vuk: &[u8; 16], encrypted_uk: &[u8; 16]) -> [u8; 16] { // Decrypt every encrypted unit key in a parsed Unit_Key_RO.inf with a VUK, // paired with its declared CPS-unit number. The single VUK->unit-keys step -// both resolvers and resolve_candidate call, so the map cannot drift. +// `resolve_candidate` calls. pub(crate) fn derive_unit_keys(uk_file: &UnitKeyFile, vuk: &[u8; 16]) -> Vec<(u32, [u8; 16])> { uk_file .encrypted_keys @@ -524,7 +524,7 @@ pub fn resolve_candidate( ) -> Option { // Boil a VUK → all unit keys, each paired with its declared CPS-unit number. // Derive the stride version from the disc's own MKB, then defer to the shared - // `derive_unit_keys` (the one place both resolvers and this path decrypt). + // `derive_unit_keys` (the one place a VUK unwraps the unit keys). let boil = |vuk: Vuk| -> Option> { let version = mkb_type(mkb) .map(|t| t.generation()) @@ -735,6 +735,31 @@ mod resolve_candidate_tests { assert_eq!(r.unit_keys, vec![(2, uk.key)]); } + /// The declared number comes from the file's own numbering (an HD DVD VTKF with a gap), and + /// falls back to the position when the file does not parse or lacks that slot. + #[test] + fn resolve_candidate_uk_declared_number_follows_the_file_with_positional_fallback() { + let mut vtkf = vec![0u8; 0x80 + 64 * 0x24]; + vtkf[..12].copy_from_slice(b"DVD_HD_V_TKF"); + // Slots 2 and 5 (1-based) are present; slot 1 is a gap. + for slot in [1usize, 4] { + vtkf[0x80 + slot * 0x24] = 0x80; + } + let at = |idx: u32| { + let uk = UnitKey::new(idx, [0x9u8; 16]); + resolve_candidate(&KeyCandidate::Uk(uk), &[], &vtkf, None) + .expect("terminal") + .unit_keys[0] + .0 + }; + assert_eq!(at(0), 2); + assert_eq!(at(1), 5, "the file's number, not idx + 1"); + assert_eq!(at(7), 7, "no such slot: the position"); + let uk = UnitKey::new(3, [0x9u8; 16]); + let r = resolve_candidate(&KeyCandidate::Uk(uk), &[], &[0u8; 4], None).unwrap(); + assert_eq!(r.unit_keys[0].0, 3, "unparseable file: the position"); + } + /// MK/PK/DK paths derive the VUK from a VID; without one, derivation stops. #[test] fn resolve_candidate_mk_requires_vid() { diff --git a/src/aacs/inf.rs b/src/aacs/inf.rs index 9d47d49b..0ff50e9b 100644 --- a/src/aacs/inf.rs +++ b/src/aacs/inf.rs @@ -217,7 +217,9 @@ const VTKF_AV_FLG: u8 = 0x80; /// An absent slot is SKIPPED (not a terminator): the 1-based slot index IS /// the CPS unit number, so `title_cps_unit` is left empty (playlist-owned). pub fn parse_vtkf(data: &[u8]) -> Option { - if data.len() < VTKF_HEADER_LEN || &data[..12] != VTKF_MAGIC { + // A file cut inside the entry table would silently lose its later slots' keys. + if data.len() < VTKF_HEADER_LEN + VTKF_MAX_ENTRIES * VTKF_ENTRY_LEN || &data[..12] != VTKF_MAGIC + { return None; } // SHA1 of the WHOLE file — the KEYDB lookup key. BackupHDDVD-family key @@ -228,9 +230,6 @@ pub fn parse_vtkf(data: &[u8]) -> Option { let mut encrypted_keys = Vec::new(); for n in 0..VTKF_MAX_ENTRIES { let pos = VTKF_HEADER_LEN + n * VTKF_ENTRY_LEN; - if pos + VTKF_ENTRY_LEN > data.len() { - break; - } // AV_FLG clear = empty slot: skip it, but keep the slot index as the CPS // number (do NOT break — a gap must not renumber the keys that follow). if data[pos] & VTKF_AV_FLG == 0 { @@ -304,6 +303,10 @@ fn read_mkb_pack( // Transport errors propagate: swallowing one would return a truncated MKB as Ok. let r = session.execute(&cdb, DataDirection::FromDevice, &mut buf, 10_000)?; + // A GOOD status with no header transferred is a drive fault, not an empty MKB. + if r.bytes_transferred < 4 { + return Err(crate::error::Error::AacsKeyRead); + } let data_len = u16::from_be_bytes([buf[0], buf[1]]) as usize; let num_packs = buf[3] as usize; if data_len < 2 { @@ -318,12 +321,17 @@ fn read_mkb_pack( /// Read MKB from drive via SCSI (REPORT DISC STRUCTURE format 0x83). /// Returns the concatenated MKB data from all packs. A short or over-long pack is a -/// drive fault (`AacsKeyRead`), never a holed MKB; a header-only pack in a multi-pack MKB is `AacsKeyRead`; a single header-only pack is an empty MKB. +/// drive fault (`AacsKeyRead`), never a holed MKB; a header-only pack in a multi-pack MKB is +/// `AacsKeyRead`; a single header-only pack is an empty MKB. pub fn read_mkb_from_drive( session: &mut dyn crate::scsi::ScsiTransport, ) -> crate::error::Result> { let (num_packs, first) = read_mkb_pack(session, 0)?; let Some(mut mkb) = first else { + // A declared multi-pack MKB whose first pack carries no data is a hole. + if num_packs > 1 { + return Err(crate::error::Error::AacsKeyRead); + } return Ok(Vec::new()); }; // A header-only pack in a multi-pack MKB is a hole, not an empty MKB. @@ -455,6 +463,13 @@ mod vtkf_tests { ); } + #[test] + fn parse_vtkf_rejects_a_file_cut_inside_the_entry_table() { + let full = synth_vtkf(&[[0x11u8; 16], [0x22u8; 16]]); + let cut = VTKF_HEADER_LEN + 10 * VTKF_ENTRY_LEN; + assert!(parse_vtkf(&full[..cut]).is_none()); + } + #[test] fn parse_vtkf_rejects_non_magic() { let mut data = synth_vtkf(&[[0x11u8; 16]]); @@ -843,6 +858,42 @@ mod read_mkb_tests { ); } + /// A header the drive never transferred, or a zero-length first pack of a multi-pack MKB, + /// is a drive fault: `AacsKeyRead`, not an empty MKB. + #[test] + fn read_mkb_from_drive_untransferred_header_and_holed_first_pack_fail() { + struct Scripted { + xfer: usize, + num_packs: u8, + } + impl ScsiTransport for Scripted { + fn execute( + &mut self, + _cdb: &[u8], + _direction: DataDirection, + data: &mut [u8], + _timeout_ms: u32, + ) -> crate::error::Result { + data[0..2].copy_from_slice(&0u16.to_be_bytes()); + data[3] = self.num_packs; + Ok(ScsiResult { + status: 0, + bytes_transferred: self.xfer, + sense: [0u8; 32], + }) + } + } + for (xfer, num_packs) in [(0, 0), (4, 2)] { + assert!( + matches!( + read_mkb_from_drive(&mut Scripted { xfer, num_packs }), + Err(crate::error::Error::AacsKeyRead) + ), + "xfer {xfer}, packs {num_packs}" + ); + } + } + /// A transport failure on the FIRST pack must propagate as an error — the /// MKB is the root of the whole AACS ladder, so an unreadable one cannot be /// downgraded to "an MKB with no records". @@ -970,3 +1021,25 @@ mod unit_key_ro_tests { ); } } + +#[cfg(test)] +mod content_cert_tests { + use super::*; + + #[test] + fn parse_content_cert_reads_cc_id_from_bytes_14_to_20() { + let mut data = vec![0u8; 20]; + data[1] = 0x80; + for (i, b) in data[14..20].iter_mut().enumerate() { + *b = 0xA0 + i as u8; + } + data[13] = 0xEE; + let cert = parse_content_cert(&data).expect("20-byte cert parses"); + assert_eq!(cert.cc_id, [0xA0, 0xA1, 0xA2, 0xA3, 0xA4, 0xA5]); + assert!(cert.bus_encryption); + assert!( + parse_content_cert(&data[..19]).is_none(), + "19 bytes is short" + ); + } +} diff --git a/src/aacs/mkb.rs b/src/aacs/mkb.rs index a424f511..5009ffa0 100644 --- a/src/aacs/mkb.rs +++ b/src/aacs/mkb.rs @@ -116,6 +116,7 @@ pub enum AacsVersion { /// the V10-vs-else stride choice it drives — lives in exactly one place. pub const AACS_MAJOR_BD: u8 = 1; +/// AACS major version 2 (UHD-BD, 2.0 / 2.1): see [`AACS_MAJOR_BD`]. pub const AACS_MAJOR_UHD: u8 = 2; impl AacsVersion { diff --git a/src/aacs/mod.rs b/src/aacs/mod.rs index 8e29c598..d69d26e2 100644 --- a/src/aacs/mod.rs +++ b/src/aacs/mod.rs @@ -152,6 +152,25 @@ mod tests { assert_eq!(ALIGNED_UNIT_LEN, 3 * 2048); } + // Any error but UdfNotFound / DiscRead is a hard failure: a good DUPLICATE copy must not + // mask it. + #[test] + fn read_first_propagates_a_hard_error_without_trying_the_duplicate() { + use crate::error::Error; + let candidates = ["/AACS/Unit_Key_RO.inf", "/AACS/DUPLICATE/Unit_Key_RO.inf"]; + let mut tried = Vec::new(); + let out = super::read_first(&candidates, |p| { + tried.push(p.to_string()); + if p == "/AACS/DUPLICATE/Unit_Key_RO.inf" { + Ok(vec![0xAA]) + } else { + Err(Error::DecryptFailed) + } + }); + assert!(matches!(out, Err(Error::DecryptFailed))); + assert_eq!(tried, vec!["/AACS/Unit_Key_RO.inf".to_string()]); + } + // AACS `/AACS/DUPLICATE/` redundancy: a `DiscRead` on the primary managed // file must fall through to the backup copy, not abort the whole read. #[test] diff --git a/src/aacs/segment.rs b/src/aacs/segment.rs index c8f95093..ca0b2c91 100644 --- a/src/aacs/segment.rs +++ b/src/aacs/segment.rs @@ -302,6 +302,48 @@ mod tests { assert!(ranges.is_empty(), "inverted segment yields no range"); } + #[test] + fn fmts_key_ranges_skips_a_segment_straddling_non_adjacent_extents() { + use crate::disc::Extent; + // Clip bytes [0, 20480) at LBA 100.., [20480, 40960) at LBA 500.. (or reversed). + // Packets 104..=109 span bytes 19968..21119, crossing the extent boundary. + let segs = vec![Segment { + index: 5, + start_spn: 104, + end_spn: 109, + }]; + for (first, second) in [(100, 500), (500, 100)] { + let extents = vec![ + Extent { + start_lba: first, + sector_count: 10, + }, + Extent { + start_lba: second, + sector_count: 10, + }, + ]; + let ranges = fmts_key_ranges(&segs, &extents, &|v| v as usize); + assert!(ranges.is_empty(), "{first}/{second}: {ranges:?}"); + } + } + + #[test] + fn segment_edges_are_inclusive() { + let seg = Segment { + index: 1, + start_spn: 100, + end_spn: 200, + }; + assert!(seg.contains_spn(100) && seg.contains_spn(200)); + assert!(!seg.contains_spn(99) && !seg.contains_spn(201)); + // A unit touching exactly one edge packet overlaps. + assert!(seg.overlaps_spn(69, 100)); + assert!(seg.overlaps_spn(200, 231)); + assert!(!seg.overlaps_spn(68, 99)); + assert!(!seg.overlaps_spn(201, 232)); + } + #[test] fn clip_byte_to_lba_walks_extents() { use crate::disc::Extent; diff --git a/src/aacs/trace.rs b/src/aacs/trace.rs index 40fd9674..04c225f3 100644 --- a/src/aacs/trace.rs +++ b/src/aacs/trace.rs @@ -146,7 +146,7 @@ pub enum KeyOutcome { /// Where an acquired key came from, for the ring's trace and the engine's key status. /// Variants are ordered root-of-trust → per-disc leaf. -#[derive(Debug, Clone, Copy, PartialEq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum KeyOrigin { /// MKB + device keys → subset-difference tree → VUK DeviceKey, @@ -197,6 +197,8 @@ mod tests { // `who` is the source's name carried verbatim. assert_eq!(t.keys[0].who, "keydb"); assert_eq!(t.unlock[0].who, "AACS cert"); + // Apps match on this id. + assert_eq!(BUS_BLOCKED, "bus_blocked"); // Default / new is empty. assert_eq!(ResolutionTrace::new(), ResolutionTrace::default()); assert!(ResolutionTrace::new().unlock.is_empty()); diff --git a/src/aacs/variant.rs b/src/aacs/variant.rs index 3808e8b1..e615e1dc 100644 --- a/src/aacs/variant.rs +++ b/src/aacs/variant.rs @@ -19,7 +19,7 @@ use super::crypto::{aes_ecb_decrypt, aes_g}; use super::mkb::*; use super::types::DeviceKey; -// ── Public constants ────────────────────────────────────────────────────── +// ── Constants ───────────────────────────────────────────────────────────── /// Zero placeholder KCD, NOT real key material — PER-LICENSEE. const KEY_CORRECTION_DATA: [u8; 16] = [0u8; 16]; @@ -419,7 +419,9 @@ fn km_from_slot_inputs( /// without its slot) against every slot and returns the Km for the slot whose full chain passes /// the Verify-Media-Key record, so an unverified key is never returned. VID-free — derive the /// VUK via [`super::derive::derive_vuk`]; `Kp` comes from [`walk_processing_key`]. Errors: -/// `NotVariantMkb`/`MkbIncomplete`/ `ProcessingKeyUnavailable` +/// `NotVariantMkb`, `MkbIncomplete` (also a slot's short tables), `ProcessingKeyUnavailable` +/// (no slot verified), or, when a slot needs it, `SoftCorrectionRequired` / +/// `OnlineChallengeRequired` / `VariantsTableUnavailable`. pub fn derive_media_key_variant( mkb_records: &[MkbRecord], pk: &[u8; 16], @@ -759,7 +761,7 @@ mod tests { fn mkb_records_matches_walk_mkb_framing() { // The lazy `mkb_records` iterator and the owning `walk_mkb` must agree on // (offset, type, len) for every record — they share the one framing - // walker, and every aacs::resolve/derive MKB walk now relies on this equivalence. + // walker, and the derive MKB walks rely on this equivalence. let mut mkb = vec![0x10, 0x00, 0x00, 0x06, 0xAA, 0xBB]; mkb.extend_from_slice(&[0x05, 0x00, 0x00, 0x08, 1, 2, 3, 4]); mkb.extend_from_slice(&[0x00, 0x00, 0x00, 0x00, 0xFF]); // terminator + trailing @@ -942,6 +944,16 @@ mod tests { ); } + #[test] + fn walk_processing_key_rejects_a_device_key_shift_of_32_or_more() { + // dk.u_mask_shift = 3 + 32 would wrap to the slot's shift 3 under wrapping_shl and + // match; a keydb byte out of u32-shift range must match nothing. + let (recs, mut dk, _, _) = synthetic_variant_setup(0x00); + assert!(walk_processing_key(&recs, &[dk.clone()]).is_some()); + dk.u_mask_shift = 3 + 32; + assert!(walk_processing_key(&recs, &[dk]).is_none()); + } + #[test] fn walk_processing_key_skips_uv_zero() { // A uv == 0 slot is skipped (`if uv == 0 { continue }`). With only a diff --git a/src/bdnav/bdjo.rs b/src/bdnav/bdjo.rs index 92f03504..8dc0a383 100644 --- a/src/bdnav/bdjo.rs +++ b/src/bdnav/bdjo.rs @@ -397,6 +397,20 @@ mod tests { } } + #[test] + fn app_strings_drop_trailing_nul_padding() { + let bytes = build_bdjo(&[AppSpec { + control_code: 1, + base_dir: "0000\0", + classpath_extension: "", + initial_class: "a.B\0\0", + }]); + let apps = parse(&bytes).expect("parses"); + assert_eq!(apps[0].base_directory, "0000"); + assert_eq!(apps[0].initial_class, "a.B"); + assert_eq!(apps[0].jar_ids(), vec!["0000"]); + } + #[test] fn parses_autostart_app_fqcn_and_jar_ids() { let bytes = build_bdjo(&[AppSpec { diff --git a/src/bdnav/vm.rs b/src/bdnav/vm.rs index d01d4313..b73faa2d 100644 --- a/src/bdnav/vm.rs +++ b/src/bdnav/vm.rs @@ -226,7 +226,7 @@ fn run(vm: &mut Vm, mut obj_id: usize, is_feature: &dyn Fn(u16) -> bool) -> Opti // CMP — skip the next command when the compare is false. 1 => { let truth = match c.cmp_opt { - 0x01 => (src & !dst) == 0, // BC: every src bit set in dst + 0x01 => (dst & !src) == 0, // BC: every dst bit set in src (hdmv_vm.c) 0x02 => dst == src, 0x03 => dst != src, 0x04 => dst >= src, @@ -522,6 +522,42 @@ mod tests { assert_eq!(resolve(&index, &mobjs, &|_| true), None); } + #[test] + fn goto_lands_on_its_target_and_break_terminates() { + // op_cnt=1, grp=BRANCH(0), sub_grp=GOTO(0), imm dst; branch_opt GOTO(1) / BREAK(2). + let goto = |line: u32| cmd(1 << 5, 0x81, 0, 0, line, 0); + let brk = cmd(1 << 5, 0x82, 0, 0, 0, 0); + let index = idx(PlaybackObj::Hdmv { id_ref: 0 }, vec![]); + // GOTO line 2 jumps over the logo PlayPL 99 (a candidate) to the feature PlayPL 1. + let d = build(&[&[goto(2), play_pl(99), play_pl(1)]]); + let mobjs = mobj::parse(&d).unwrap(); + assert_eq!(resolve(&index, &mobjs, &|_| true), Some(1)); + let d = build(&[&[brk, play_pl(1)]]); + let mobjs = mobj::parse(&d).unwrap(); + assert_eq!(resolve(&index, &mobjs, &|_| true), None); + } + + #[test] + fn only_the_play_pl_family_emits_a_playlist() { + // sub_grp=PLAY(2); branch_opt 3 is Terminate/Link, which never plays dst. + let d = build(&[&[cmd((1 << 5) | 2, 0x83, 0, 0, 11, 0)]]); + let mobjs = mobj::parse(&d).unwrap(); + let index = idx(PlaybackObj::Hdmv { id_ref: 0 }, vec![]); + assert_eq!(resolve(&index, &mobjs, &|_| true), None); + } + + #[test] + fn setsystem_does_not_execute_as_a_gpr_set() { + // grp=SET(2), sub_grp=SETSYSTEM(1), set_opt=MOVE(1): must not write GPR5. + let setsys = cmd((2 << 5) | (2 << 3) | 1, 0x40, 0, 0x01, 5, 77); + let d = build(&[&[setsys]]); + let mobjs = mobj::parse(&d).unwrap(); + let index = idx(PlaybackObj::Hdmv { id_ref: 0 }, vec![]); + let mut vm = Vm::new(&mobjs, &index); + assert_eq!(run(&mut vm, 0, &|_| false), None); + assert_eq!(vm.gpr[5], 0); + } + #[test] fn swap_immediate_operand_is_not_written_back() { // SWAP dst=GPR0 with src=IMMEDIATE 5 (imm_op1=0, imm_op2=1) must refuse @@ -625,12 +661,13 @@ mod tests { } } - // libbluray INSN_BC: true iff every bit of src is set in dst (src & ~dst == 0). + // libbluray INSN_BC skips the next command when `dst & ~src` is non-zero: + // true iff every bit of dst is set in src. #[test] - fn cmp_bc_tests_src_bits_within_dst() { - assert!(cmp_picks_true(0x01, 0b1110, 0b0110)); - assert!(!cmp_picks_true(0x01, 0b0100, 0b0110)); - assert!(!cmp_picks_true(0x01, 0b0110, 0b1110)); + fn cmp_bc_tests_dst_bits_within_src() { + assert!(!cmp_picks_true(0x01, 0b1110, 0b0110)); + assert!(cmp_picks_true(0x01, 0b0100, 0b0110)); + assert!(cmp_picks_true(0x01, 0b0110, 0b1110)); } #[test] diff --git a/src/clpi.rs b/src/clpi.rs index 3a185782..45755092 100644 --- a/src/clpi.rs +++ b/src/clpi.rs @@ -293,6 +293,20 @@ mod tests { assert_eq!(clip.streams[0].language, "eng"); } + /// An audio or TextST sci shorter than 5 bytes has no language field: the stream is + /// kept, the language stays empty, and nothing indexes past the sci. + #[test] + fn program_info_short_audio_and_textst_sci_keep_the_stream_without_a_language() { + let pi = build_program_info(&[ + (0x1100, vec![0x83u8, 0x61, b'e', b'n']), + (0x1200, vec![0x92u8, 0x01, b'e', b'n']), + ]); + let data = build_clpi_with_proginfo(100, &pi, None); + let clip = parse(&data).expect("should parse"); + assert_eq!(clip.streams.len(), 2); + assert!(clip.streams.iter().all(|s| s.language.is_empty())); + } + /// ProgramInfo PG (0x90)/IG (0x91): layout is coding_type(1)+lang(3), /// so language is at sci[1..4] (NOT sci[2..5] like audio). Verify the /// PG arm reads from the right offset. diff --git a/src/consts.rs b/src/consts.rs index c134e8b4..90392782 100644 --- a/src/consts.rs +++ b/src/consts.rs @@ -105,7 +105,7 @@ pub mod coding_type { /// 13818-1 Table 2-22). Shared by the program-stream demuxer and the TS/M2TS /// muxers, so defined here once. Each is `u8` (matches the byte on the wire). pub mod pes_stream_id { - /// Video stream (`110x xxxx`; freemkv emits the base id `0xE0`). + /// Video stream (`1110 xxxx`; freemkv emits the base id `0xE0`). pub const VIDEO: u8 = 0xE0; /// MPEG audio stream (`110x xxxx`; 11172-3/13818-3/13818-7, freemkv emits `0xC0`). pub const MPEG_AUDIO: u8 = 0xC0; @@ -122,7 +122,7 @@ pub mod pes_stream_id { /// private_stream_2 — DVD navigation (PCI/DSI); carries no muxable ES. pub const PRIVATE_STREAM_2: u8 = 0xBF; - /// Highest video stream_id — the `110x xxxx` video range tops out at 0xEF. + /// Highest video stream_id — the `1110 xxxx` video range tops out at 0xEF. pub const VIDEO_MAX: u8 = 0xEF; /// Inclusive range of every PES `stream_id` that carries demuxable payload: diff --git a/src/css/mod.rs b/src/css/mod.rs index 768e6241..2e4864a6 100644 --- a/src/css/mod.rs +++ b/src/css/mod.rs @@ -66,7 +66,7 @@ impl std::fmt::Debug for CssState { /// Recover the CSS title key with no keys, by scanning scrambled sectors and /// running the known-plaintext attack (see the [`keyless`] module). /// -/// Scans up to 50000 sectors across `extents` and returns the first sector +/// Scans a bounded sector budget across `extents` and returns the first sector /// that yields a key — no player keys, no disc-key crack. Works on a live /// drive and on disc images alike. /// @@ -269,6 +269,8 @@ fn crack_key_scan( let mut consecutive_locked = 0u32; // Sectors actually inspected: zero means no verdict (never Unencrypted). let mut inspected = 0u32; + // Sectors lost to non-locked read failures: more lost than inspected is no clear verdict. + let mut failed = 0u32; let mut first_err: Option = None; 'outer: for (extent_idx, ext) in extents.iter().enumerate() { @@ -359,6 +361,7 @@ fn crack_key_scan( } } else { consecutive_locked = 0; + failed += n; first_err.get_or_insert(e); } } @@ -367,12 +370,12 @@ fn crack_key_scan( } } - // ENCRYPTED-but-uncracked if a scrambled sector was seen or reads were - // CSS-locked. Nothing inspected is no verdict: fail closed with the read error - // (else as uncracked). Only a scan that READ sectors and saw no scramble is clear. + // ENCRYPTED-but-uncracked if a scrambled sector was seen or reads were CSS-locked. Nothing + // inspected, or more unreadable than inspected, is no verdict: fail closed with the read + // error (else as uncracked). if saw_scrambled || saw_locked { CrackOutcome::ScrambledUncracked - } else if inspected == 0 && tried > 0 { + } else if tried > 0 && (inspected == 0 || failed > inspected) { first_err.map_or(CrackOutcome::ScrambledUncracked, CrackOutcome::Unreadable) } else { CrackOutcome::Unencrypted @@ -1859,4 +1862,31 @@ mod tests { }]; let _ = crack_key_outcome(&mut AlwaysFails, &ext, 1, None); } + + // A first batch with no scrambled PES, then a scrambled region that will not read: the + // unreadable sectors outnumber the inspected ones, so the scan must not call it clear. + #[test] + fn a_clear_head_then_unreadable_tail_is_not_unencrypted() { + struct ClearThenDamaged; + impl SectorSource for ClearThenDamaged { + fn read_sectors(&mut self, lba: u32, c: u16, b: &mut [u8], _r: bool) -> Result { + if lba >= 2 { + return Err(Error::DiscRead { + sector: lba as u64, + status: None, + sense: None, + }); + } + let n = c as usize * 2048; + b[..n].fill(0); + Ok(n) + } + } + let ext = [Extent { + start_lba: 0, + sector_count: 200, + }]; + let out = crack_key_outcome(&mut ClearThenDamaged, &ext, 2, None); + assert!(matches!(out, CrackOutcome::Unreadable(_)), "got {out:?}"); + } } diff --git a/src/css/tables.rs b/src/css/tables.rs index d13b1e04..a1eb4a1e 100644 --- a/src/css/tables.rs +++ b/src/css/tables.rs @@ -171,6 +171,20 @@ mod tests { assert_eq!(TAB1[0x33], 0x00, "TAB1[0x33] is the published 0x00"); } + // FNV-1a over a whole table: a swapped pair keeps a permutation and the anchors, so only + // a full-content pin catches a mistyped or regenerated table. + fn fnv1a(table: &[u8]) -> u64 { + table.iter().fold(0xcbf2_9ce4_8422_2325u64, |h, &b| { + (h ^ b as u64).wrapping_mul(0x0000_0100_0000_01b3) + }) + } + + #[test] + fn tab1_and_tab2_contents_are_pinned() { + assert_eq!(fnv1a(&TAB1), 0xcdb8_7228_1c08_c265, "TAB1 content changed"); + assert_eq!(fnv1a(&TAB2), 0x79e0_69d7_0f3d_2f25, "TAB2 content changed"); + } + // TAB2 is a permutation of 0..256 (LFSR1 high-byte feedback substitution); a non-bijective // table would bias the LFSR1 keystream. #[test] diff --git a/src/ctx.rs b/src/ctx.rs index 646823ca..dde092a6 100644 --- a/src/ctx.rs +++ b/src/ctx.rs @@ -159,4 +159,26 @@ mod tests { } ); } + + // Each switch follows its own variable; no other test reads either one. + #[test] + fn diag_from_env_reads_each_switch_from_its_own_variable() { + let set = |k: &str, on: bool| unsafe { + if on { + std::env::set_var(k, "1") + } else { + std::env::remove_var(k) + } + }; + set("FREEMKV_SKIP_PARSE", true); + set("FREEMKV_PROFILE", false); + let d = Diag::from_env(); + assert!(d.skip_parse && !d.profile); + set("FREEMKV_SKIP_PARSE", false); + set("FREEMKV_PROFILE", true); + let d = Diag::from_env(); + assert!(!d.skip_parse && d.profile); + set("FREEMKV_PROFILE", false); + assert_eq!(Diag::from_env(), Diag::default()); + } } diff --git a/src/decrypt.rs b/src/decrypt.rs index b4fcd020..18d5d274 100644 --- a/src/decrypt.rs +++ b/src/decrypt.rs @@ -386,11 +386,9 @@ impl AacsKeyMap { } } -/// Decrypt `buf` with a resolved AACS key map. Thin wrapper over -/// [`decrypt_span`] — the map is the AACS scheme's input, not a second -/// orchestrator. -// Content-less convenience form used only by tests (production reads always go -// through `decrypt_sectors_mapped_in_content`, which honours the content extents). +// Decrypt `buf` with a resolved AACS key map: the content-less convenience form of +// `decrypt_sectors_mapped_in_content`, used only by tests (production reads always go +// through that one, which honours the content extents). #[cfg(test)] pub(crate) fn decrypt_sectors_mapped( buf: &mut [u8], @@ -2078,6 +2076,76 @@ mod tests { assert!(buf[2 * ul..3 * ul].iter().all(|&b| b == 0)); } + /// Two damaged forensic units beside verifying ones are damage: blanked and counted. Only + /// a key with NO verifying unit is a wrong key. + #[test] + fn mapped_phase_two_damaged_units_beside_verified_ones_are_blanked() { + use crate::disc::ContentFormat; + let ul = aacs::content::ALIGNED_UNIT_LEN; + let usz = (ul / 2048) as u32; + let key = [0xAAu8; 16]; + // Even-phase units sit at even indices 0, 2, 4, 6; the odd ones are left alone. + let mut buf = vec![0u8; 7 * ul]; + for i in 0..7 { + let mut u = clear_ts_unit(); + if i % 2 == 0 { + aacs_encrypt_unit_for_test(&mut u, &key); + if i < 4 { + crate::test_util::damage_unit_seed(&mut u); + u[4] = 0x47; + } + } + buf[i * ul..(i + 1) * ul].copy_from_slice(&u); + } + let keys = DecryptKeys::Aacs { + unit_keys: vec![(0, key)], + format: ContentFormat::BdTs, + }; + let map = AacsKeyMap::from_ranges_phased(vec![(0, 7 * usz, 0, Phase::Even)]); + let blanked = decrypt_sectors_mapped_in_content(&mut buf, &keys, 0, &map, None) + .expect("damage beside verified units is not E7013"); + assert_eq!(blanked, 2); + } + + /// One failing forensic unit that ANOTHER held key opens is a wrong key assignment, not + /// damage: E7013 even though it is the read's only failure. + #[test] + fn mapped_phase_a_unit_another_key_opens_is_a_wrong_key_not_damage() { + use crate::disc::ContentFormat; + let ul = aacs::content::ALIGNED_UNIT_LEN; + let usz = (ul / 2048) as u32; + let (key_a, key_b) = ([0xAAu8; 16], [0xBBu8; 16]); + let mut buf = clear_ts_unit(); + aacs_encrypt_unit_for_test(&mut buf, &key_b); + let keys = DecryptKeys::Aacs { + unit_keys: vec![(0, key_a), (1, key_b)], + format: ContentFormat::BdTs, + }; + let map = AacsKeyMap::from_ranges_phased(vec![(0, usz, 0, Phase::Even)]); + assert!(matches!( + decrypt_sectors_mapped_in_content(&mut buf, &keys, 0, &map, None), + Err(crate::error::Error::DecryptFailed) + )); + } + + /// HD DVD (`MpegPs`) units carry no TS sync at byte 4: none is damage, none is touched. + #[test] + fn blank_damaged_units_leaves_mpegps_untouched() { + use crate::disc::ContentFormat; + let ul = aacs::content::ALIGNED_UNIT_LEN; + // Unflagged, no sync at byte 4, not clean PS: damage if judged as BD-TS. + let mut buf: Vec = (0..2 * ul + 100) + .map(|i| (i as u8).wrapping_mul(31) | 1) + .collect(); + for u in 0..2 { + buf[u * ul + 20] = 0; + } + let before = buf.clone(); + let n = blank_damaged_units(&mut buf, 0, ContentFormat::MpegPs, &|_| true, true); + assert_eq!(n, 0); + assert_eq!(buf, before); + } + /// Option A (E7013): a garbage seed with its CPI bits clear lost the TS sync every BD-TS /// unit carries at byte 4 (KS-2): damage, blanked and counted; clear and zero units are not. #[test] diff --git a/src/diag.rs b/src/diag.rs index 0a23264a..0ea3e5a4 100644 --- a/src/diag.rs +++ b/src/diag.rs @@ -774,6 +774,9 @@ mod tests { assert_eq!(tv_system_str(FrameRate::F25), "PAL"); assert_eq!(fps_str(FrameRate::F29_97), "29.97"); assert_eq!(tv_system_str(FrameRate::F29_97), "NTSC"); + assert_eq!(tv_system_str(FrameRate::F50), "PAL"); + assert_eq!(tv_system_str(FrameRate::F23_976), "NTSC"); + assert_eq!(tv_system_str(FrameRate::F59_94), "NTSC"); } #[test] @@ -893,5 +896,10 @@ mod tests { assert!(row.contains("block_mode=2"), "{row}"); assert!(row.contains("block_type=1"), "{row}"); assert!(row.contains("DROP(leading-secondary-block-piece)"), "{row}"); + + // The same secondary piece past the leading run is kept, as feature body. + let row = dvd_cell_row(3, &sec, false); + assert!(row.contains("keep(feature-body)"), "{row}"); + assert!(!row.contains("keep(plain-feature)"), "{row}"); } } diff --git a/src/dirimage/layout.rs b/src/dirimage/layout.rs index a54f9a2f..26e36fcb 100644 --- a/src/dirimage/layout.rs +++ b/src/dirimage/layout.rs @@ -216,7 +216,14 @@ fn walk(dir: &Path, disc_path: &str, depth: u32, entries: &mut usize) -> Result< // A broken symlink or a file gone between readdir and stat: skip it // rather than plan an unreadable extent. Anything else (e.g. perm // denied) is NOT a missing file, so skipping it would drop it silently. - Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + tracing::warn!( + target: "freemkv::dirimage", + path = %child_path.escape_debug(), + "entry is a dangling link or vanished mid-walk; omitted from the image" + ); + continue; + } Err(e) => return Err(Error::from(e)), }; if !meta.is_file() { @@ -676,7 +683,7 @@ pub(super) fn flatten<'a>(dir: &'a DirNode, out: &mut Vec<&'a FileNode>) { } } -/// Metadata footprint in blocks, for diagnostics. +/// Metadata footprint in blocks; `plan` bounds it against `MAX_META_BYTES`. pub(super) fn metadata_block_count(root: &DirNode) -> u64 { 2 + metadata_blocks(root) } @@ -907,6 +914,26 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + // `walk` counts every entry against `MAX_ENTRIES`: the entry that would pass it is refused. + #[test] + fn the_entry_cap_refuses_the_entry_past_max_entries() { + let dir = std::env::temp_dir().join(format!("fmkv-entries-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write(dir.join("a.bin"), b"x").unwrap(); + let mut at_cap = MAX_ENTRIES - 1; + assert!( + walk(&dir, "/", 0, &mut at_cap).is_ok(), + "the last allowed entry" + ); + let mut past = MAX_ENTRIES; + assert!(matches!( + walk(&dir, "/", 0, &mut past), + Err(Error::DirImageTooLarge) + )); + let _ = std::fs::remove_dir_all(&dir); + } + // The nesting cap (`MAX_DEPTH`) refuses a tree deeper than the planner // represents, before it recurses without bound. Executes `walk`'s depth // guard on a real directory chain, not the constant. diff --git a/src/dirimage/tests.rs b/src/dirimage/tests.rs index d8e93a8c..a1c953b8 100644 --- a/src/dirimage/tests.rs +++ b/src/dirimage/tests.rs @@ -201,6 +201,59 @@ fn a_file_past_the_ad_ceiling_reads_back_as_multiple_extents() { ); } +// Past the 30-bit AD ceiling, each extent reads its own bytes of the file: a read at an +// extent's start, one sector in, and a batch crossing the extent boundary. +#[test] +fn a_multi_extent_file_reads_each_extents_own_bytes() { + use std::io::Seek; + let s = Scratch::new("bigdata"); + s.file("BDMV/index.bdmv", &pattern(1, 16)); + s.dir("BDMV/STREAM"); + let big = s.path().join("BDMV/STREAM/00000.m2ts"); + let ad = super::layout::MAX_AD_BYTES; + let size = ad + 3 * SECTOR as u64; + let mut f = std::fs::File::create(&big).unwrap(); + f.set_len(size).unwrap(); + for (at, b) in [ + (0, 0xA1u8), + (ad - SECTOR as u64, 0xB2), + (ad, 0xC3), + (ad + SECTOR as u64, 0xD4), + ] { + f.seek(SeekFrom::Start(at)).unwrap(); + f.write_all(&[b]).unwrap(); + } + drop(f); + + let mut img = DirImage::open(s.path()).unwrap(); + let fs = udf::read_filesystem(&mut img).unwrap(); + let exts = fs + .file_extents(&mut img, "/BDMV/STREAM/00000.m2ts") + .unwrap(); + assert_eq!(exts.len(), 2); + let (first, second) = (exts[0].0, exts[1].0); + let head = |img: &mut DirImage, lba: u32, n: u16| { + let mut buf = vec![0u8; n as usize * SECTOR]; + img.read_sectors(lba, n, &mut buf, false).unwrap(); + (0..n as usize) + .map(|i| buf[i * SECTOR]) + .collect::>() + }; + assert_eq!(head(&mut img, first, 1), [0xA1]); + assert_eq!(head(&mut img, second, 1), [0xC3]); + assert_eq!( + head(&mut img, second + 1, 1), + [0xD4], + "one sector into the extent" + ); + let last = first + exts[0].1 - 1; + assert_eq!( + head(&mut img, last, 3), + [0xB2, 0xC3, 0xD4], + "a batch crossing the extent boundary" + ); +} + /// An IN-RANGE gap (a sector below capacity that no extent covers) reads as /// zeros, not an error — a real image has unrecorded sectors too, and /// `read_filesystem` probes fixed LBAs (256, the VDS window) before it knows @@ -1197,6 +1250,47 @@ fn a_skipped_symlink_does_not_collide_with_a_same_name_file() { DirImage::open(s.path()).expect("dangling symlink is omitted, not a collision"); } +/// A tree whose metadata exceeds the in-memory ceiling is refused at plan time. +#[test] +fn an_oversized_metadata_footprint_is_refused_at_plan_time() { + let (s, _, _) = bdmv_scratch(); + for dir in ["BDMV/STREAM", "BDMV/CLIPINF"] { + for i in 0..17_000 { + s.file(&format!("{dir}/{i:05}"), b""); + } + } + let err = DirImage::open(s.path()).unwrap_err(); + assert_eq!(err.code(), crate::error::E_DIR_IMAGE_TOO_LARGE, "got {err}"); +} + +/// A symlink to a directory is omitted, and does not take part in the collision check. +#[cfg(unix)] +#[test] +fn a_directory_symlink_is_omitted_from_the_image() { + let (s, _, _) = bdmv_scratch(); + s.file("BDMV/STREAM/A.M2TS", b"x"); + s.dir("elsewhere"); + std::os::unix::fs::symlink( + s.path().join("elsewhere"), + s.path().join("BDMV/STREAM/ A.M2TS"), + ) + .unwrap(); + let mut img = DirImage::open(s.path()).expect("a directory link is omitted, not a collision"); + let fs = udf::read_filesystem(&mut img).unwrap(); + let names: Vec<_> = fs + .find_dir("/BDMV/STREAM") + .unwrap() + .entries + .iter() + .map(|e| e.name.clone()) + .collect(); + assert_eq!( + names.iter().filter(|n| n.trim() == "A.M2TS").count(), + 1, + "{names:?}" + ); +} + /// A directory whose FID list exceeds the UDF reader's cap is refused at plan time, by name. #[test] fn an_over_wide_directory_is_refused_at_plan_time() { diff --git a/src/disc/bluray.rs b/src/disc/bluray.rs index 96c20036..650f16bc 100644 --- a/src/disc/bluray.rs +++ b/src/disc/bluray.rs @@ -190,7 +190,10 @@ impl Disc { // WHOLE-clip size/extents on purpose (no EP-map seek): out-of-mark // bytes are dropped downstream by PTS at the SeamPlan — see // `SeamPlan::place` (src/mux/timeline.rs) `raw_ns >= in_ns && <= out_ns`. - total_size += pkt_count as u64 * crate::consts::BD_SOURCE_PACKET_BYTES as u64; + let clpi_bytes = pkt_count as u64 * crate::consts::BD_SOURCE_PACKET_BYTES as u64; + // The .ssif interleaves the dependent view, so the CLPI count (base + // view only) under-sizes what the extents deliver. + let mut from_ssif = false; // Get stream file extents from UDF allocation descriptors (dual- // layer discs split files across layers). Normally `.m2ts`; AACS 2.1 @@ -211,6 +214,7 @@ impl Disc { let file_exts = match udf_fs.file_extents(reader, &ssif) { Ok(exts) => { is_3d = true; + from_ssif = true; Some(exts) } Err(e) => { @@ -282,11 +286,13 @@ impl Disc { push_extents(file_exts, &mut extents, &mut feed_pos); if feed_pos > span_start { spans.insert(play_item.clip_id.clone(), (span_start, feed_pos)); - // A damaged CLPI reports no packet count: size the clip by its extents. - if pkt_count == 0 { - total_size += feed_pos - span_start; - } } + // A damaged CLPI reports no packet count: size the clip by its extents. + total_size += if pkt_count == 0 || from_ssif { + feed_pos - span_start + } else { + clpi_bytes + }; } } @@ -341,8 +347,8 @@ impl Disc { measured_cicp: None, })), 2 | 5 => { - // Guard: if coding_type is a subtitle codec (PGS 0x90/0x91), - // this is a misaligned stream -- treat as subtitle, not audio + // Guard: if coding_type is the PGS subtitle codec (0x90), this + // is a misaligned stream -- treat as subtitle, not audio if matches!(codec, Codec::Pgs) { Some(Stream::Subtitle(SubtitleStream { pid: s.pid, @@ -501,7 +507,13 @@ impl Disc { if let Some(start) = xml.find("") { let s = start + "".len(); if let Some(end) = xml[s..].find("") { - let title = xml_text_decode(xml[s..s + end].trim()); + // Disc-authored text reaches terminals and file names: + // control characters (decoded from `` too) are dropped. + let title: String = xml_text_decode(xml[s..s + end].trim()) + .chars() + .filter(|c| !c.is_control()) + .collect(); + let title = title.trim().to_string(); if !title.is_empty() && title != "Blu-ray" { return Some(title); } @@ -515,8 +527,6 @@ impl Disc { } } -// Decode the five predefined XML entities and numeric char refs in element text, and unwrap a -// CDATA section; unknown or malformed references are kept literally. /// Append each usable `(lba, sectors)` to `extents`, advancing `feed_pos` by its bytes. An extent /// with no sectors or at LBA 0 occupies no readable bytes and is skipped without moving `feed_pos`. fn push_extents(file_exts: Vec<(u32, u32)>, extents: &mut Vec, feed_pos: &mut u64) { @@ -532,6 +542,8 @@ fn push_extents(file_exts: Vec<(u32, u32)>, extents: &mut Vec, feed_pos: } } +// Decode the five predefined XML entities and numeric char refs in element text, and unwrap a +// CDATA section; unknown or malformed references are kept literally. fn xml_text_decode(raw: &str) -> String { if let Some(inner) = raw .strip_prefix(" Vec { + let mut out = vec![3u8, 0x01]; + out.extend_from_slice(&pid.to_be_bytes()); + out.push(attrs.len() as u8); + out.extend_from_slice(attrs); + out + } + + /// The MPLS format/rate nibbles reach the right `Stream` fields, and the + /// secondary-video (6) and Dolby Vision EL (7) categories are kept as video. + #[test] + fn parse_playlist_maps_format_rate_nibbles_and_secondary_video() { + let mut disc = MemDisc::new(); + let udf = make_bdmv_fs(&mut disc, &[("00001", 100, 400, 5000)]); + let mpls = build_mpls( + &[PiSpec { + clip_id: *b"00001", + in_time: 0, + out_time: 60 * 45000, + }], + (1, 1, 0, 0, 0, 1, 0, 1), + &[ + se_raw(0x1011, &[0x1B, 0x63]), // 1080p, 25 fps + se_raw(0x1100, &[0x81, 0x64, b'e', b'n', b'g']), // 5.1, 96 kHz + // Secondary video carries audio-ref and PG-ref counts after the entry. + [se_raw(0x1B00, &[0x1B, 0x63]), vec![0u8; 4]].concat(), + se_raw(0x1015, &[0x24, 0x63]), + ], + &[], + ); + let t = Disc::parse_playlist(&mut disc, &udf, "00001.mpls", &mpls) + .expect("scan") + .expect("title"); + let videos: Vec<_> = t.video_streams().collect(); + let audio = t.audio_streams().next().expect("audio"); + assert_eq!(videos.len(), 3, "primary, secondary and EL video kept"); + assert_eq!(videos[0].resolution, Resolution::R1080p); + assert_eq!(videos[0].frame_rate, FrameRate::F25); + assert_eq!(audio.channels, AudioChannels::Surround51); + assert_eq!(audio.sample_rate, SampleRate::S96); + } + + /// One unreadable playlist is skipped with a warning; the healthy one survives. + #[test] + fn scan_bluray_titles_skips_an_unreadable_playlist() { + struct FailingReader<'a> { + inner: &'a mut MemDisc, + fail_lba: u32, + } + impl SectorSource for FailingReader<'_> { + fn read_sectors( + &mut self, + lba: u32, + count: u16, + buf: &mut [u8], + recovery: bool, + ) -> Result { + if lba == self.fail_lba { + return Err(Error::DiscRead { + sector: lba as u64, + status: Some(0x02), + sense: None, + }); + } + self.inner.read_sectors(lba, count, buf, recovery) + } + } + let mut disc = MemDisc::new(); + let udf = two_playlist_bd_fs(&mut disc); + let mut reader = FailingReader { + inner: &mut disc, + fail_lba: PART_START + 30000, // 00800.mpls's data extent + }; + let titles = Disc::scan_bluray_titles(&mut reader, &udf, None).expect("scan survives"); + assert_eq!(titles.len(), 1); + assert_eq!(titles[0].playlist, "00801.mpls"); + } + + /// A Stop that lands during the final playlist's reads is `Halted`, not a + /// silently truncated list. + #[test] + fn scan_bluray_titles_polls_halt_after_the_last_playlist() { + struct CancellingReader<'a> { + inner: &'a mut MemDisc, + halt: crate::halt::Halt, + cancel_lba: u32, + } + impl SectorSource for CancellingReader<'_> { + fn read_sectors( + &mut self, + lba: u32, + count: u16, + buf: &mut [u8], + recovery: bool, + ) -> Result { + if lba == self.cancel_lba { + self.halt.cancel(); + } + self.inner.read_sectors(lba, count, buf, recovery) + } + } + let mut disc = MemDisc::new(); + let udf = two_playlist_bd_fs(&mut disc); + let halt = crate::halt::Halt::new(); + let mut reader = CancellingReader { + inner: &mut disc, + halt: halt.clone(), + cancel_lba: PART_START + 40000, // 00801.mpls, the last playlist + }; + let res = Disc::scan_bluray_titles(&mut reader, &udf, Some(&halt)); + assert!(matches!(res, Err(Error::Halted)), "got {res:?}"); + } + + /// Control characters in the disc-authored title never reach a front end, + /// whether raw or written as a character reference. + #[test] + fn read_meta_title_drops_control_characters() { + let mut disc = MemDisc::new(); + let xml = b"AB\x07C".to_vec(); + let dl = DirSpec { + name: "DL".to_string(), + icb_lba: 30, + dir_data_lba: 31, + files: vec![file_with("bdmt_eng.xml", 104, 50000, xml, false)], + subdirs: vec![], + }; + let meta = DirSpec { + name: "META".to_string(), + icb_lba: 28, + dir_data_lba: 29, + files: Vec::new(), + subdirs: vec![dl], + }; + let bdmv = DirSpec { + name: "BDMV".to_string(), + icb_lba: 20, + dir_data_lba: 21, + files: Vec::new(), + subdirs: vec![meta], + }; + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![bdmv], + }; + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = udf::read_filesystem(&mut disc).expect("fs"); + assert_eq!( + Disc::read_meta_title(&mut disc, &udf), + Some("A[2JBC".to_string()) + ); + } + // --------------------------------------------------------------- // Tests: chapters // --------------------------------------------------------------- diff --git a/src/disc/dvd.rs b/src/disc/dvd.rs index 8e523415..a57afa8e 100644 --- a/src/disc/dvd.rs +++ b/src/disc/dvd.rs @@ -91,7 +91,7 @@ impl Disc { // TODO(spec): measured colour/TFF await a CodecParser->title channel. measured_cicp: None, }); - // TODO(spec): DefaultDuration/cadence deferred. + // The frame rate is the standard's; an MKV corrects it from the measured cadence. for (vts_title_idx, dvd_title) in ts.titles.iter().enumerate() { title_number += 1; @@ -256,10 +256,51 @@ impl Disc { if halt.is_some_and(|h| h.is_cancelled()) { return Err(Error::Halted); } + warn_multi_pgc_titles(&vmg_bytes, &dvd_info); Ok((titles, nav_feature)) } } +// One warning per disc per process for titles split across PGCs (only the first is read): +// a rip scans its disc more than once, and each scan would repeat it. Later scans log at +// debug. The disc is told apart by its VIDEO_TS.IFO. +fn warn_multi_pgc_titles(vmg: &[u8], info: &ifo::DvdInfo) { + use std::hash::{Hash, Hasher}; + static WARNED: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); + let split: Vec = info + .title_sets + .iter() + .flat_map(|ts| { + ts.titles.iter().filter(|t| t.pgcs > 1).map(move |t| { + format!( + "VTS {} title {} ({} PGCs)", + ts.vts_number, t.vts_title_num, t.pgcs + ) + }) + }) + .collect(); + if split.is_empty() { + return; + } + let mut h = std::hash::DefaultHasher::new(); + vmg.hash(&mut h); + let key = h.finish(); + let first = { + let mut warned = WARNED.lock().unwrap_or_else(|e| e.into_inner()); + let first = !warned.contains(&key); + if first { + warned.push(key); + } + first + }; + let titles = split.join(", "); + if first { + tracing::warn!(target: "freemkv::scan", titles = %titles, "titles span several PGCs; only the first PGC of each is read, the rest of those titles is missing"); + } else { + tracing::debug!(target: "freemkv::scan", titles = %titles, "titles span several PGCs; only the first PGC of each is read"); + } +} + // This title's audio tracks: logical stream i plays the physical stream its PGC_AST_CTL // names (libdvdnav semantics), absent streams get no track, a repeated physical id keeps // the first. A PGC marking nothing present keeps the positional ids rather than go silent. @@ -1658,6 +1699,11 @@ mod tests { sub.codec_data.is_some(), "non-zero palette must yield codec_data" ); + let idx = String::from_utf8(sub.codec_data.clone().expect("codec_data")).expect("utf8"); + assert!( + idx.starts_with("size: 720x480\npalette: "), + "the .idx carries the NTSC coded frame size, then the palette: {idx:?}" + ); } /// Multiple titles in one VTS each become their own DiscTitle with a @@ -1774,6 +1820,109 @@ mod tests { assert_eq!(titles[1].playlist_id, 2); } + // The nav target joins on the VTS number AND the in-set title number: VTS 2 also has an + // in-set title 1, which must not take the promotion from VTS 1's. + #[test] + fn scan_dvd_titles_nav_target_is_matched_by_vts_number() { + let mut disc = MemDisc::new(); + let mut vmg = build_vmg(&[(1, 1, 1), (1, 2, 1)]); + stamp_first_play_jumptt(&mut vmg, 1); + let vts1 = build_vts(100, 0x00, &[], &[], &[(0, 9)], false); + let vts2 = build_vts(200, 0x00, &[], &[], &[(0, 19)], false); + let udf = build_video_ts_fs( + &mut disc, + &[ + FileSpec { + name: "VIDEO_TS.IFO".into(), + icb_lba: 60, + data_lba: 5000, + contents: vmg, + }, + FileSpec { + name: "VTS_01_0.IFO".into(), + icb_lba: 62, + data_lba: 6000, + contents: vts1, + }, + FileSpec { + name: "VTS_02_0.IFO".into(), + icb_lba: 64, + data_lba: 7000, + contents: vts2, + }, + ], + ); + let (titles, nav_feature) = Disc::scan_dvd_titles(&mut disc, &udf, None).expect("scan"); + assert_eq!(titles.len(), 2); + assert_eq!(nav_feature, Some(1)); + } + + fn one_title_dvd(disc: &mut MemDisc) -> udf::UdfFs { + build_video_ts_fs( + disc, + &[ + FileSpec { + name: "VIDEO_TS.IFO".into(), + icb_lba: 60, + data_lba: 5000, + contents: build_vmg(&[(1, 1, 1)]), + }, + FileSpec { + name: "VTS_01_0.IFO".into(), + icb_lba: 62, + data_lba: 6000, + contents: build_vts(0, 0x00, &[], &[], &[(0, 9)], false), + }, + ], + ) + } + + // A Stop raised before the scan, or during its last reads, is `Halted`, never a list. + #[test] + fn scan_dvd_titles_honours_the_halt_token() { + struct CancellingReader<'a> { + inner: &'a mut MemDisc, + halt: crate::halt::Halt, + reads: u32, + } + impl SectorSource for CancellingReader<'_> { + fn read_sectors( + &mut self, + lba: u32, + count: u16, + buf: &mut [u8], + recovery: bool, + ) -> crate::error::Result { + self.reads += 1; + if lba >= PART_START + 6000 { + self.halt.cancel(); + } + self.inner.read_sectors(lba, count, buf, recovery) + } + } + let mut disc = MemDisc::new(); + let udf = one_title_dvd(&mut disc); + let halt = crate::halt::Halt::new(); + halt.cancel(); + let mut reader = CancellingReader { + inner: &mut disc, + halt: halt.clone(), + reads: 0, + }; + let res = Disc::scan_dvd_titles(&mut reader, &udf, Some(&halt)); + assert!(matches!(res, Err(crate::error::Error::Halted)), "{res:?}"); + assert_eq!(reader.reads, 0, "a cancelled scan reads nothing"); + + let halt = crate::halt::Halt::new(); + let mut reader = CancellingReader { + inner: &mut disc, + halt: halt.clone(), + reads: 0, + }; + let res = Disc::scan_dvd_titles(&mut reader, &udf, Some(&halt)); + assert!(matches!(res, Err(crate::error::Error::Halted)), "{res:?}"); + } + /// chapter_times from the IFO become Chapter entries with ordinal /// names (dvd.rs maps chapter_times → Chapter{time_secs, chapter_name}). #[test] @@ -2076,4 +2225,65 @@ mod tests { fn scan_dvd_titles_bad_ifo_and_bup_is_empty() { assert_eq!(scan_ifo_bup(false, false).expect("no error").len(), 0); } + + fn title(vts_title_num: u8, pgcs: usize) -> ifo::DvdTitle { + ifo::DvdTitle { + chapters: 1, + duration_secs: 1.0, + cells: Vec::new(), + chapter_times: Vec::new(), + palette: None, + ast_ctl: [0; 8], + spst_ctl: [0; 32], + vts_title_num, + pgcs, + } + } + + // A rip scans its disc more than once: the split-title warning is one line per disc, + // naming every split title, and a rescan of the same disc logs it at debug only. + #[test] + fn multi_pgc_titles_warn_once_per_disc() { + let set = |n: u8, titles: Vec| ifo::DvdTitleSet { + vts_number: n, + vob_start_sector: 0, + video: ifo::DvdVideoAttr { + codec: Codec::Mpeg2, + resolution: Resolution::R480i, + aspect: ifo::DvdAspect::R16x9, + standard: ifo::TvSystem::Ntsc, + }, + audio_streams: Vec::new(), + subtitle_streams: Vec::new(), + titles, + }; + let info = ifo::DvdInfo { + title_sets: vec![ + set(3, vec![title(1, 8)]), + set(4, vec![title(1, 5), title(2, 1)]), + set(11, vec![title(1, 1)]), + ], + }; + let vmg = b"multi_pgc_titles_warn_once_per_disc".to_vec(); + let ((), ev) = crate::testlog::capture(|| { + warn_multi_pgc_titles(&vmg, &info); + warn_multi_pgc_titles(&vmg, &info); + warn_multi_pgc_titles(&vmg, &info); + }); + let warns: Vec<_> = ev + .iter() + .filter(|e| e.level == tracing::Level::WARN) + .collect(); + assert_eq!(warns.len(), 1, "{ev:?}"); + assert_eq!( + warns[0].field("titles"), + Some("VTS 3 title 1 (8 PGCs), VTS 4 title 1 (5 PGCs)") + ); + // A disc with no split title logs nothing. + let clear = ifo::DvdInfo { + title_sets: vec![set(1, vec![title(1, 1)])], + }; + let ((), ev) = crate::testlog::capture(|| warn_multi_pgc_titles(b"no split", &clear)); + assert!(ev.is_empty(), "{ev:?}"); + } } diff --git a/src/disc/dvd_audio_probe.rs b/src/disc/dvd_audio_probe.rs index e0ef1977..baaff38f 100644 --- a/src/disc/dvd_audio_probe.rs +++ b/src/disc/dvd_audio_probe.rs @@ -354,5 +354,6 @@ mod tests { panic!("audio") }; assert_eq!(a.pid, 0xBD81); + assert_eq!(a.channels, AudioChannels::Surround51); } } diff --git a/src/disc/encrypt.rs b/src/disc/encrypt.rs index 1e897d21..10e55e60 100644 --- a/src/disc/encrypt.rs +++ b/src/disc/encrypt.rs @@ -193,7 +193,16 @@ pub(super) fn capture( ) { Ok(raw) => Some(raw), Err(Error::Halted) => return Err(Error::Halted), - Err(_) => None, + Err(Error::AacsNoKeys) => None, + Err(e) => { + tracing::warn!( + target: "freemkv::scan", + phase = "aacs_capture", + error_code = e.code(), + "content certificate unreadable; assuming the V20/UHD Unit_Key_RO stride" + ); + None + } }; let cc = cc_raw.as_deref().and_then(aacs::inf::parse_content_cert); // No-cert default = UHD (V20 stride), matching `read_aacs_version`. diff --git a/src/disc/extract.rs b/src/disc/extract.rs index fbd6a9c8..037a4b83 100644 --- a/src/disc/extract.rs +++ b/src/disc/extract.rs @@ -102,8 +102,8 @@ struct PlannedFile { /// ECMA-167 4/14.14.1.1 type-1 extent is allocated but not recorded: it /// occupies the file's byte space and its contents are zeros). extents: Vec, - /// A bus-encrypted stream file the reader's bus map could not locate: it - /// cannot be de-bussed, so it is never read or written, only counted lost. + /// A file whose data cannot be located (an unreadable File Entry, or a bus-encrypted + /// stream file the reader's bus map could not map): never read or written, only counted lost. unmapped: bool, } @@ -441,26 +441,58 @@ fn plan_tree( unmapped: true, }); } else { - let inline = fs.inline_data_at(reader, entry.meta_lba)?; - let extents = if inline.is_some() { - Vec::new() - } else { - fs.extents_abs_at(reader, entry.meta_lba)? - }; - files.push(PlannedFile { - host_rel: child_rel, - disc_name: entry.name.clone(), - size: entry.size, - inline, - extents, - unmapped: false, - }); + let located = fs + .inline_data_at(reader, entry.meta_lba) + .and_then(|inline| { + let extents = if inline.is_some() { + Vec::new() + } else { + fs.extents_abs_at(reader, entry.meta_lba)? + }; + Ok((inline, extents)) + }); + match located { + Ok((inline, extents)) => files.push(PlannedFile { + host_rel: child_rel, + disc_name: entry.name.clone(), + size: entry.size, + inline, + extents, + unmapped: false, + }), + // A Stop, a dead bus or a gone source ends the run; it is not a damaged entry. + Err(e) + if matches!(e, Error::Halted) + || e.is_scsi_transport_failure() + || e.is_source_terminated() => + { + return Err(e); + } + // A damaged File Entry loses that one file, not the tree around it. + Err(e) => { + tracing::warn!( + target: "freemkv::extract", + file = %child_rel.display(), + code = e.code(), + "file entry unreadable; file lost whole" + ); + files.push(PlannedFile { + host_rel: child_rel, + disc_name: entry.name.clone(), + size: entry.size, + inline: None, + extents: Vec::new(), + unmapped: true, + }); + } + } } } Ok(()) } -// Accounts an unmapped stream file as lost whole: nothing is written for it on the host. +// Accounts a file with no locatable data (an unreadable File Entry, or a bus-encrypted stream +// file that cannot be mapped to de-bus) as lost whole: nothing is written for it on the host. fn unmapped_file( pf: &PlannedFile, total_bytes: u64, @@ -471,7 +503,7 @@ fn unmapped_file( tracing::warn!( target: "freemkv::extract", file = %pf.host_rel.display(), - "bus-encrypted stream file cannot be located to de-bus; not extracted" + "file cannot be located on the disc; not extracted" ); *done_bytes = done_bytes.saturating_add(pf.size); *done_unreadable = done_unreadable.saturating_add(pf.size); @@ -741,6 +773,9 @@ fn read_tries( Err(Error::Halted) => return Err(Error::Halted), // The key set's loud stop (KU §2.4): never a hole, the run stops `.partial`. Err(e @ (Error::WholeDiscKeyMissing | Error::NoDiscKey { .. })) => return Err(e), + // A dead bus or gone source is not a hole: zero-filling the rest of the tree + // would finalize every remaining file as complete (the Read stage's rule too). + Err(e) if e.is_scsi_transport_failure() || e.is_source_terminated() => return Err(e), Err(Error::DecryptFailed) => return Ok(Tried::Undecryptable), Err(_) if last => return Ok(Tried::Media), Err(_) => {} @@ -2512,6 +2547,184 @@ mod tests { assert!(!res.complete && !res.halted); } + // A reader that serves one sector once (the tree listing reads each File Entry for its + // size), then fails it with `status`. + struct FailAt(MemDisc, u32, Option, u32); + impl SectorSource for FailAt { + fn read_sectors(&mut self, lba: u32, n: u16, buf: &mut [u8], r: bool) -> Result { + if lba == self.1 { + self.3 += 1; + } + if lba == self.1 && self.3 > 1 { + return Err(Error::DiscRead { + sector: lba as u64, + status: self.2, + sense: None, + }); + } + self.0.read_sectors(lba, n, buf, r) + } + } + + fn two_clip_disc() -> MemDisc { + two_clip_disc_with_first_unmapped().0 + } + + // One damaged File Entry loses that file only; the rest of the tree extracts. + #[test] + fn an_unreadable_file_entry_loses_that_file_not_the_tree() { + let mut disc = FailAt(two_clip_disc(), PART_START + 24, Some(0x02), 0); + let out = TmpDir::new("bad_fe"); + let res = clear_disc() + .extract_tree( + &mut disc, + out.path(), + &ExtractOptions::default(), + &crate::ctx::Ctx::default(), + ) + .expect("extract"); + assert_eq!( + read_out(out.path(), "BDMV/STREAM/00002.m2ts"), + Some(vec![0x22; 3 * 2048]) + ); + assert!(read_out(out.path(), "BDMV/STREAM/00001.m2ts").is_none()); + assert_eq!(res.bytes_unreadable, 3 * 2048); + assert!(!res.complete && !res.halted); + } + + // A dead bus while locating a file is not a damaged entry: the run fails. + #[test] + fn a_transport_failure_locating_a_file_entry_fails_the_run() { + let status = Some(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE); + let mut disc = FailAt(two_clip_disc(), PART_START + 24, status, 0); + let out = TmpDir::new("dead_bus_fe"); + let r = clear_disc().extract_tree( + &mut disc, + out.path(), + &ExtractOptions::default(), + &crate::ctx::Ctx::default(), + ); + assert!(r.is_err_and(|e| e.is_scsi_transport_failure())); + } + + // A holed file feeds the run's loss counters and the live progress split: the + // unreadable bytes are not counted as good. + #[test] + fn a_hole_reaches_the_loss_stats_and_the_progress_split() { + let seen = std::sync::Arc::new(std::sync::Mutex::new((0u64, 0u64))); + let tap = seen.clone(); + let ctx = crate::ctx::Ctx::new(crate::halt::Halt::new()).with_events(std::sync::Arc::new( + move |e: &crate::event::Event<'_>| { + if let crate::event::Event::Pass(p) = e { + *tap.lock().unwrap() = (p.bytes_good_total, p.bytes_unreadable_total); + } + }, + )); + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "BDMV".to_string(), + icb_lba: 20, + dir_data_lba: 21, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "STREAM".to_string(), + icb_lba: 22, + dir_data_lba: 23, + files: vec![ + file("00001.m2ts", 24, 5000, vec![0x11; 3 * 2048], true), + file("00002.m2ts", 25, 6000, vec![0x22; 6 * 2048], true), + ], + subdirs: vec![], + }], + }], + }; + let mut disc = build_disc(root); + for i in 0..3u32 { + disc.bad.insert(PART_START + 5000 + i); // the first clip's whole extent + } + let out = TmpDir::new("hole_stats"); + clear_disc() + .extract_tree(&mut disc, out.path(), &ExtractOptions::default(), &ctx) + .expect("extract"); + let loss = ctx.stats.snapshot(); + assert_eq!(loss.bytes_lost, 3 * 2048); + assert!(loss.read_skips >= 1); + assert_eq!(*seen.lock().unwrap(), (6 * 2048, 3 * 2048)); + } + + // A damaged AACS unit the reader blanks is unreadable bytes, not good ones. + #[test] + fn a_blanked_aacs_unit_is_counted_unreadable() { + const DATA: u32 = 5000; + let key = [0u8; 16]; + let mut content = encrypt_aacs_unit(&key, 0xA1); + let mut damaged = vec![0x55u8; crate::aacs::content::ALIGNED_UNIT_LEN]; + damaged[0] |= 0xC0; // flagged encrypted, but no key opens it + content.extend_from_slice(&damaged); + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "BDMV".to_string(), + icb_lba: 20, + dir_data_lba: 21, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "STREAM".to_string(), + icb_lba: 22, + dir_data_lba: 23, + files: vec![file("00001.m2ts", 24, DATA, content, true)], + subdirs: vec![], + }], + }], + }; + let mut disc = build_disc(root); + let d = aacs_disc(); + let a = PART_START + DATA; + let set = crate::keys::KeyRing::keyed_for_test(&d, key, &[(a, a + 6)]); + let opts = ExtractOptions { + keys: Some(&set), + ..Default::default() + }; + let out = TmpDir::new("blanked_unit"); + let res = d + .extract_tree(&mut disc, out.path(), &opts, &crate::ctx::Ctx::default()) + .expect("extract"); + let unit = crate::aacs::content::ALIGNED_UNIT_LEN as u64; + assert_eq!(res.bytes_unreadable, unit); + assert_eq!(res.files[0].bytes_good, unit); + assert!(!res.complete); + } + + // The key set's loud stop is never a hole: the run ends with it. + #[test] + fn a_missing_disc_key_stops_the_run_instead_of_holing_the_unit() { + struct KeyMissingAt(MemDisc, u32); + impl SectorSource for KeyMissingAt { + fn read_sectors(&mut self, lba: u32, n: u16, b: &mut [u8], r: bool) -> Result { + if lba == self.1 { + return Err(Error::WholeDiscKeyMissing); + } + self.0.read_sectors(lba, n, b, r) + } + } + let mut disc = KeyMissingAt(two_clip_disc(), PART_START + 5000); + let out = TmpDir::new("key_missing"); + let r = clear_disc().extract_tree( + &mut disc, + out.path(), + &ExtractOptions::default(), + &crate::ctx::Ctx::default(), + ); + assert!(matches!(r, Err(Error::WholeDiscKeyMissing)), "{r:?}"); + } + // A Stop at the lost file's report ends the run there, like any other file. #[test] fn progress_stop_on_an_unmapped_stream_file_halts_the_run() { @@ -2763,6 +2976,53 @@ mod tests { assert!(read_out(out.path(), "a.m2ts.partial").is_some()); } + // A dead bus is not a hole: the file must stay `.partial` and the error surface, + // not be zero-filled and finalized as complete. + #[test] + fn a_transport_failure_stops_the_file_instead_of_zero_filling() { + struct DeadBus; + impl SectorSource for DeadBus { + fn read_sectors(&mut self, lba: u32, _: u16, _: &mut [u8], _: bool) -> Result { + Err(Error::DiscRead { + sector: lba as u64, + status: Some(crate::scsi::SCSI_STATUS_TRANSPORT_FAILURE), + sense: None, + }) + } + } + let len = 4 * SECTOR_BYTES as u32; + let pf = PlannedFile { + host_rel: PathBuf::from("a.m2ts"), + disc_name: "a.m2ts".into(), + size: len as u64, + inline: None, + extents: vec![crate::udf::AbsExtent { + lba: 100, + len, + recorded: true, + }], + unmapped: false, + }; + let out = TmpDir::new("dead_bus"); + std::fs::create_dir_all(out.path()).unwrap(); + let mut dec = DecryptingSectorSource::new(DeadBus, DecryptKeys::None); + let (mut done, mut bad) = (0u64, 0u64); + let r = extract_one_file( + &mut dec, + &TreeSink::create(out.path(), true).unwrap(), + &pf, + len as u64, + &mut done, + &mut bad, + &crate::ctx::Ctx::default(), + ); + assert!(r.is_err_and(|e| e.is_scsi_transport_failure())); + assert!( + read_out(out.path(), "a.m2ts").is_none(), + "must stay .partial" + ); + } + // A crafted extent near the top of the LBA space must not overflow: a batch // whose START or END passes u32::MAX is a hole, never a panic or a wrapped read. #[test] diff --git a/src/disc/hddvd.rs b/src/disc/hddvd.rs index a5b788ff..5d3f4774 100644 --- a/src/disc/hddvd.rs +++ b/src/disc/hddvd.rs @@ -361,6 +361,7 @@ const MAX_XPL_TITLES: usize = 512; // disc-supplied XML. const MAX_XPL_BYTES: usize = 1024 * 1024; +// Max `.evo` clips resolved per scan (each costs an ICB read and a probe). const MAX_HDDVD_CLIPS: usize = 512; const MAX_XPL_CLIPS_PER_TITLE: usize = 256; @@ -2606,6 +2607,134 @@ mod tests { ); } + // A Stop is never a probe failure: it surfaces, whether the drive reports it or the + // token is already raised. + #[test] + fn probe_evo_streams_surfaces_a_stop() { + let ext = Extent { + start_lba: 777, + sector_count: 4, + }; + let mut halted = FailReader(|| crate::error::Error::Halted); + assert!(matches!( + probe_evo_streams(&mut halted, std::slice::from_ref(&ext), None), + Err(crate::error::Error::Halted) + )); + let halt = crate::halt::Halt::new(); + halt.cancel(); + let mut counter = ProbeCounter::new(MemDisc::new(), vec![777]); + assert!(matches!( + probe_evo_streams(&mut counter, std::slice::from_ref(&ext), Some(&halt)), + Err(crate::error::Error::Halted) + )); + assert_eq!(counter.sectors_read, 0, "a raised token reads nothing"); + } + + // A cancelled probe is not memoised as "no streams": the next title sharing the + // extents probes again. + #[test] + fn probe_cache_does_not_memoise_a_stop() { + let ext = [Extent { + start_lba: 777, + sector_count: 4, + }]; + let mut cache = EvoProbeCache::default(); + let mut halted = FailReader(|| crate::error::Error::Halted); + assert!(matches!( + cache.streams(&mut halted, &ext, None), + Err(crate::error::Error::Halted) + )); + let mut counter = ProbeCounter::new(MemDisc::new(), vec![777]); + cache.streams(&mut counter, &ext, None).expect("probe"); + assert_eq!(counter.hits[0], 1, "the extents are probed again"); + } + + // A Stop while resolving a clip's extents ends the scan; it is not "clip unreadable". + #[test] + fn scan_hddvd_titles_surfaces_a_stop_resolving_extents() { + struct HaltAt(MemDisc, u32); + impl SectorSource for HaltAt { + fn read_sectors( + &mut self, + lba: u32, + count: u16, + buf: &mut [u8], + recovery: bool, + ) -> crate::error::Result { + if lba == self.1 { + return Err(crate::error::Error::Halted); + } + self.0.read_sectors(lba, count, buf, recovery) + } + } + let mut disc = MemDisc::new(); + let udf = make_hddvd_fs_with_evo(&mut disc, &synthetic_evo()); + let mut reader = HaltAt(disc, PART_START + 100); // FEATURE.EVO's ICB + let res = Disc::scan_hddvd_titles(&mut reader, &udf, None); + assert!(matches!(res, Err(crate::error::Error::Halted)), "{res:?}"); + + let halt = crate::halt::Halt::new(); + halt.cancel(); + let mut counter = ProbeCounter::new(reader.0, vec![]); + let res = Disc::scan_hddvd_titles(&mut counter, &udf, Some(&halt)); + assert!(matches!(res, Err(crate::error::Error::Halted)), "{res:?}"); + assert_eq!(counter.sectors_read, 0, "a raised token reads nothing"); + } + + // A title whose end precedes its begin never reports a negative duration. + #[test] + fn compose_xpl_titles_clamps_a_reversed_clip_duration_and_names_by_number() { + let clip_extents: BTreeMap)> = [( + "a.evo".to_string(), + ( + "A.EVO".to_string(), + 1000u64, + vec![Extent { + start_lba: 1, + sector_count: 1, + }], + ), + )] + .into_iter() + .collect(); + let xpl_titles = vec![XplTitle { + number: 7, + name: String::new(), + duration_secs: 10.0, + clips: vec![XplClip { + evo: "a.evo".to_string(), + begin_secs: 9.0, + end_secs: 5.0, + }], + chapters: vec![], + }]; + let mut disc = MemDisc::new(); + let titles = compose_xpl_titles( + &mut disc, + &xpl_titles, + &clip_extents, + &std::collections::HashSet::new(), + None, + ) + .expect("compose"); + assert_eq!(titles[0].clips[0].duration_secs, 0.0); + assert_eq!( + titles[0].playlist, "TITLE_7", + "an unnamed title is named by number" + ); + } + + // A title with no displayName is named by its id attribute. + #[test] + fn parse_xpl_titles_falls_back_to_the_id_attribute_for_the_name() { + let xpl = r#" + + <PrimaryAudioVideoClip titleTimeBegin="00:00:00:00" titleTimeEnd="00:00:10:00" src="file:///x/B.MAP"/> + "#; + let titles = parse_xpl_titles(xpl.as_bytes()); + assert_eq!(titles[0].name, "Bonus3"); + } + // ── read_adv_obj_xpl: prefix AND suffix are both required ────────────── /// A file matching the `vplst` prefix but NOT the `.xpl` suffix must not diff --git a/src/disc/mod.rs b/src/disc/mod.rs index 8db6c95b..8d6e33b3 100644 --- a/src/disc/mod.rs +++ b/src/disc/mod.rs @@ -49,9 +49,10 @@ pub struct Disc { pub capacity_bytes: u64, /// Number of layers (1 = single, 2 = dual) pub layers: u8, - /// Titles sorted by duration (longest first), then playlist name + /// Titles in main-feature order: `titles[0]` is the selected main feature (nav or + /// authoring pick first, else largest physical size), not necessarily the longest pub titles: Vec, - /// Disc region + /// Disc region. Not decoded yet: scans always report [`DiscRegion::Free`]. pub region: DiscRegion, /// AACS state -- None if disc is unencrypted or keys unavailable pub aacs: Option, @@ -713,8 +714,8 @@ fn bus_map(files: StreamFiles, titles: &[DiscTitle]) -> crate::sector::bus_remov crate::sector::bus_removal::BusMap::new(files, &unknown) } -// Corrects a title's TrueHD channels/sample-rate/Atmos by probing the first decrypted major -// sync (`reader` must yield DECRYPTED sectors: mux time, not scan). +// Test-only: corrects a title's TrueHD channels/sample-rate/Atmos by probing the first decrypted +// major sync. The production path is `correct_truehd_stream`, driven from mux/driver.rs. #[cfg(test)] pub(crate) fn correct_truehd_channels(reader: &mut dyn SectorSource, title: &mut DiscTitle) { let pids: Vec = title @@ -2277,7 +2278,10 @@ impl Disc { // whole content is captured; otherwise records may run past the // prefix — grow and retry, bounded by MAX_BYTES. if (n > 0 && n < buf.len()) || buf.len() < want || want >= MAX_BYTES { - return Ok(crate::aacs::mkb::trim_mkb(buf)); + // The prefix buffer is sized for the padded file; release the unused capacity. + let mut mkb = crate::aacs::mkb::trim_mkb(buf); + mkb.shrink_to_fit(); + return Ok(mkb); } want = (want * 2).min(MAX_BYTES); } @@ -2532,15 +2536,14 @@ impl Disc { crate::labels::fill_defaults(&mut titles); // 5. Format (AACS MKB generation → BD/UHD/FMTS; tree → HD-DVD/DVD) and - // layers. Region coding is not decoded yet — every disc reports - // Region-free (correct for UHD; a stub until BD/DVD region detection). + // layers. Region coding is not decoded yet: every disc reports Region-free. let format = Self::detect_disc_format(reader, &udf_fs, &titles); let layers = Self::layers_for(format, capacity); let region = DiscRegion::Free; - // 6. CSS detection for DVDs is deferred to `Disc::scan`'s drive-auth path - // (has `&mut Drive`), run after this returns — the reader-based crack path - // is NOT run here: on a CSS disc it would fail ~50,000 sectors one-by-one. + // 6. CSS: `scan_image` cracks the title key after this returns and `scan_live` only + // runs bus-auth. The reader-based crack is NOT run here: on a CSS disc it would + // fail ~50,000 sectors one-by-one. let css = None; tracing::info!( @@ -2833,15 +2836,20 @@ impl Disc { if udf_fs.find_dir("/BDMV").is_some() { // Only the Type-and-Version record (first record) is needed. let mkb_paths = crate::aacs::role_paths(udf_fs, crate::aacs::AacsRole::Mkb); - if let Ok(mkb) = - crate::aacs::read_first(&mkb_paths, |p| udf_fs.read_file_prefix(reader, p, 64)) - { - match mkb_type(&mkb).map(|t| t.generation()) { + match crate::aacs::read_first(&mkb_paths, |p| udf_fs.read_file_prefix(reader, p, 64)) { + Ok(mkb) => match mkb_type(&mkb).map(|t| t.generation()) { Some(AacsVersion::V21) => return DiscFormat::Fmts, Some(AacsVersion::V20) => return DiscFormat::Uhd, Some(AacsVersion::V10) => return DiscFormat::BluRay, None => {} - } + }, + // No MKB at all is an unencrypted disc, not a fault. + Err(Error::AacsNoKeys) => {} + Err(e) => tracing::warn!( + target: "freemkv::scan", + error_code = e.code(), + "MKB prefix unreadable; classifying the disc by resolution" + ), } // Unencrypted/unreadable MKB: refine by resolution, but a BD-tree disc // is never below Blu-ray — `detect_format` can return Dvd for an SD @@ -3190,13 +3198,17 @@ pub(crate) fn mapfile_path_for(iso_path: &std::path::Path) -> std::path::PathBuf std::path::PathBuf::from(s) } +// Disc-authored names can be arbitrarily long; the sanitised name stays well under NAME_MAX. +const NULL_MAPFILE_NAME_MAX_CHARS: usize = 128; + impl Disc { /// Path to the mapfile for a given output path. /// /// For the null device ([`crate::io::null_device`]), returns `{dir}/{volume_id_or_title}.mapfile` /// where `{dir}` is a directory this process created under the temp dir /// (owner-only on Unix, unpredictable name, stable for the process). For - /// regular files, returns `{path}.mapfile`. + /// regular files, returns `{path}.mapfile`. The null-device directory and the mapfile in it + /// outlive the process; removing them is the caller's job. pub fn mapfile_for(&self, path: &std::path::Path) -> std::path::PathBuf { if crate::io::is_null_device(path) { let name: String = self @@ -3204,6 +3216,7 @@ impl Disc { .as_deref() .unwrap_or(&self.volume_id) .chars() + .take(NULL_MAPFILE_NAME_MAX_CHARS) .map(|c| { if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c @@ -3342,10 +3355,6 @@ pub fn detect_max_batch_sectors(device_path: &str) -> u16 { } } -// ─── Format helpers ──────────────────────────────────────────────────────── - -// Old format_* functions replaced by Resolution/FrameRate/AudioChannels/SampleRate enums - #[cfg(test)] mod tests { use super::*; @@ -3537,6 +3546,122 @@ mod tests { ); } + // A one-title DVD image whose one VOB sector is `sector`. + fn dvd_image_with(sector: &[u8]) -> crate::udf::fixture::MemDisc { + use crate::udf::fixture::*; + let mut disc = MemDisc::new(); + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "VIDEO_TS".into(), + icb_lba: 50, + dir_data_lba: 51, + files: vec![ + file_with("VIDEO_TS.IFO", 60, 5000, dvd_vmg_bytes(), true), + file_with("VTS_01_0.IFO", 62, 6000, dvd_vts_bytes(1000, 10, 10), true), + ], + subdirs: vec![], + }], + }; + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + disc.put_bytes(9010, sector); + disc + } + + // A key source that counts the times it is asked and holds no key. + struct Counting(std::sync::Arc); + + impl crate::keysource::KeySource for Counting { + fn get_unit_keys( + &self, + _: &dyn crate::keysource::ResolveCtx, + ) -> Result> { + self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Ok(Vec::new()) + } + } + + // `img` scanned, its main title's keys acquired over `asked`, and the decrypt gate. + fn dvd_decision( + img: &mut crate::udf::fixture::MemDisc, + asked: &std::sync::Arc, + ) -> (Disc, Result<()>) { + use crate::keys::{AcquireOptions, KeyRing, KeyScope, check_decryptable}; + let disc = Disc::scan_image(img, 500_000, &ScanOptions::default()).unwrap(); + let a = asked.clone(); + let sources: crate::session::KeySourceFactory = std::sync::Arc::new(move || { + vec![Box::new(Counting(a.clone())) as Box] + }); + let ctx = crate::ctx::Ctx::new(crate::halt::Halt::new()); + let scope = KeyScope::Titles(vec![0]); + let ring = KeyRing::acquire_for_disc( + &disc, + img, + scope.clone(), + &sources, + AcquireOptions::default(), + &ctx, + ) + .unwrap() + .keys; + let gate = check_decryptable(&disc, false, Some(&ring), &scope); + (disc, gate) + } + + /// DVD takes the one encryption decision on CSS's own scramble flag: clear content passes + /// with no key source asked and nothing to descramble; scrambled content with no key + /// recovered refuses with E7027; a recovered key descrambles. + #[test] + fn dvd_takes_the_one_encryption_decision() { + let asked = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let title_key = [0x42, 0x13, 0x37, 0xBE, 0xEF]; + + let mut clear = [0u8; 2048]; + crate::css::dvd_pack_header(&mut clear, 0xE0); + for (i, b) in clear.iter_mut().enumerate().skip(0x80) { + *b = (i % 251) as u8; + } + assert!( + crate::css::scrambled_at(&clear).is_none(), + "fixture is clear" + ); + let (disc, gate) = dvd_decision(&mut dvd_image_with(&clear), &asked); + assert!(disc.css.is_none() && disc.css_error.is_none() && !disc.encrypted); + assert!(gate.is_ok(), "{gate:?}"); + + let mut locked = clear; + locked[0x14] |= 0x10; + crate::css::lfsr::scramble_sector(&title_key, &mut locked); + let (disc, gate) = dvd_decision(&mut dvd_image_with(&locked), &asked); + assert!(disc.encrypted && disc.css.is_none()); + assert!(matches!(gate, Err(Error::CssNoDiscKey)), "{gate:?}"); + + let crackable = crackable_css_sector(&title_key); + let (disc, gate) = dvd_decision(&mut dvd_image_with(&crackable), &asked); + assert!(gate.is_ok(), "{gate:?}"); + let state = disc.css.expect("the key is recovered"); + let mut plain = crackable; + crate::css::descramble_sector(&state, &mut plain); + let mut want = crackable; + crate::css::lfsr::descramble_sector(&title_key, &mut want); + assert_eq!( + plain[0x80..], + want[0x80..], + "descrambled under the disc's key" + ); + assert_ne!(plain[0x80..], crackable[0x80..]); + + assert_eq!( + asked.load(std::sync::atomic::Ordering::SeqCst), + 0, + "CSS asks no AACS key source" + ); + } + // An operator Stop during scan_image's CSS crack must end the scan as Halted — // the token in `opts.halt` has to reach the crack, not a hardcoded `None`. #[test] @@ -8705,6 +8830,196 @@ mod tests { assert_eq!(AudioChannels::from_audio_format(12), AudioChannels::Unknown); } + // The BD-ROM STN attribute nibbles decode through these tables into the fields + // that feed muxer timing and metadata. + #[test] + fn mpls_attribute_nibbles_decode_to_the_spec_values() { + use Resolution::*; + for (vf, want) in [ + (1, R480i), + (2, R576i), + (3, R480p), + (4, R1080i), + (5, R720p), + (6, R1080p), + (7, R576p), + (8, R2160p), + (0, Unknown), + ] { + assert_eq!(Resolution::from_video_format(vf), want, "video_format {vf}"); + } + for (vr, want, frac) in [ + (1, FrameRate::F23_976, (24000, 1001)), + (2, FrameRate::F24, (24, 1)), + (3, FrameRate::F25, (25, 1)), + (4, FrameRate::F29_97, (30000, 1001)), + (5, FrameRate::F30, (30, 1)), + (6, FrameRate::F50, (50, 1)), + (7, FrameRate::F59_94, (60000, 1001)), + (8, FrameRate::F60, (60, 1)), + ] { + assert_eq!(FrameRate::from_video_rate(vr), want, "video_rate {vr}"); + assert_eq!(want.as_fraction(), frac, "{want:?}"); + } + assert_eq!(FrameRate::from_video_rate(0), FrameRate::Unknown); + for (af, want) in [ + (1, AudioChannels::Mono), + (3, AudioChannels::Stereo), + (6, AudioChannels::Surround51), + ] { + assert_eq!( + AudioChannels::from_audio_format(af), + want, + "audio_format {af}" + ); + } + for (ar, want) in [ + (1, SampleRate::S48), + (4, SampleRate::S96), + (5, SampleRate::S192), + (12, SampleRate::S48_192), + (14, SampleRate::S48_96), + (0, SampleRate::Unknown), + ] { + assert_eq!(SampleRate::from_audio_rate(ar), want, "audio_rate {ar}"); + } + } + + #[test] + fn coding_type_maps_every_known_stream_to_its_codec() { + use crate::consts::coding_type as c; + for (ct, want) in [ + (c::HEVC, Codec::Hevc), + (c::H264, Codec::H264), + (c::H264_MVC, Codec::H264), + (c::VC1, Codec::Vc1), + (c::MPEG2_VIDEO, Codec::Mpeg2), + (c::TRUEHD, Codec::TrueHd), + (c::DTS_HD_MA, Codec::DtsHdMa), + (c::DTS_HD_HR, Codec::DtsHdHr), + (c::DTS, Codec::Dts), + (c::AC3, Codec::Ac3), + (c::AC3_PLUS, Codec::Ac3Plus), + (c::AC3_PLUS_SECONDARY, Codec::Ac3Plus), + (c::LPCM, Codec::Lpcm), + (c::DTS_HD_SECONDARY, Codec::DtsHdHr), + (c::PG, Codec::Pgs), + ] { + assert_eq!(Codec::from_coding_type(ct), want, "coding_type {ct:#04x}"); + } + } + + #[test] + fn every_hd_resolution_is_hd_and_detects_as_bluray() { + for r in [ + Resolution::R720p, + Resolution::R1080i, + Resolution::R1080p, + Resolution::R2160p, + ] { + assert!(r.is_hd(), "{r:?}"); + } + for r in [ + Resolution::R480i, + Resolution::R480p, + Resolution::R576i, + Resolution::R576p, + Resolution::Unknown, + ] { + assert!(!r.is_hd(), "{r:?}"); + } + for r in [Resolution::R720p, Resolution::R1080i] { + let titles = vec![title_with_video(Codec::H264, r)]; + assert_eq!(Disc::detect_format(&titles), DiscFormat::BluRay, "{r:?}"); + } + } + + // Key sampling skips a size-inflated streamless decoy, and falls back to the + // largest title when none has video. + #[test] + fn main_title_prefers_the_largest_title_with_video() { + let mut disc = make_test_disc(1_000, "BDROM"); + let mut decoy = DiscTitle::empty(); + decoy.playlist = "decoy".into(); + decoy.size_bytes = 9_000_000_000; + let mut feature = title_with_video(Codec::H264, Resolution::R1080p); + feature.playlist = "feature".into(); + feature.size_bytes = 1_000; + disc.titles = vec![decoy.clone(), feature]; + assert_eq!( + disc.main_title().map(|t| t.playlist.as_str()), + Some("feature") + ); + let mut small = decoy.clone(); + small.playlist = "small".into(); + small.size_bytes = 5; + disc.titles = vec![small, decoy]; + assert_eq!( + disc.main_title().map(|t| t.playlist.as_str()), + Some("decoy") + ); + } + + // The key-service label is the trimmed volume id, else the BDMV title; the AACS + // inputs come from the captured state. + #[test] + fn inputs_label_is_the_trimmed_volume_id_else_the_meta_title() { + let mut disc = make_test_disc(1_000, " TITLE_2024 "); + let mut st = aacs_empty(); + st.uk_ro = vec![1, 2, 3]; + st.mkb = vec![4, 5]; + disc.aacs = Some(st); + let i = disc.inputs().expect("inputs"); + assert_eq!(i.volume_label.as_deref(), Some("TITLE_2024")); + assert_eq!(i.unit_key_ro, vec![1, 2, 3]); + assert_eq!(i.mkb, vec![4, 5]); + disc.volume_id = " ".into(); + disc.meta_title = Some("Meta".into()); + assert_eq!( + disc.inputs().expect("inputs").volume_label.as_deref(), + Some("Meta") + ); + } + + #[test] + fn mapfile_for_dev_null_caps_a_long_disc_name() { + let disc = make_test_disc(1_000, &"x".repeat(1_000)); + let p = disc.mapfile_for(std::path::Path::new("/dev/null")); + let name = p.file_name().and_then(|n| n.to_str()).unwrap_or(""); + assert!(name.len() < 255, "{} bytes", name.len()); + } + + // A padded MKB is trimmed to its record stream, and the buffer does not keep the + // capacity of the padded prefix it was read into. + #[test] + fn read_mkb_content_releases_the_padding_capacity() { + use crate::udf::fixture::*; + let mut mkb = vec![0x04, 0x00, 0x10, 0x00]; + mkb.resize(0x1000, 0xAA); + mkb.extend_from_slice(&[0, 0, 0, 0]); + mkb.resize(17 * 1024 * 1024, 0); + let mut disc = MemDisc::new(); + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "AACS".into(), + icb_lba: 12, + dir_data_lba: 13, + files: vec![file_with("MKB_RO.inf", 14, 1000, mkb, true)], + subdirs: vec![], + }], + }; + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = crate::udf::read_filesystem(&mut disc).expect("fs"); + let got = Disc::read_mkb_content(&mut disc, &udf).expect("mkb"); + assert_eq!(got.len(), 0x1000); + assert!(got.capacity() < 1024 * 1024, "capacity {}", got.capacity()); + } + #[test] fn layers_follow_the_format_capacity() { for (fmt, sectors, want) in [ diff --git a/src/disc/pgs_forced_probe.rs b/src/disc/pgs_forced_probe.rs index 58503058..0b805cb2 100644 --- a/src/disc/pgs_forced_probe.rs +++ b/src/disc/pgs_forced_probe.rs @@ -896,6 +896,19 @@ mod tests { let es_off = p + 14; // ES data follows the 14-byte PES header let n = es.len().min(192 - es_off); pkt[es_off..es_off + n].copy_from_slice(&es[..n]); + // Stuff through the adaptation field so the PES ends exactly at its last + // ES byte rather than carrying zero padding after it. + let len = es_off + n - p; + if len < 184 { + let start = 192 - len; + pkt.copy_within(p..p + len, start); + pkt[7] |= 0x20; + pkt[8] = (start - 9) as u8; + if start > 9 { + pkt[9] = 0; + pkt[10..start].fill(0xFF); + } + } pkt } @@ -1675,7 +1688,12 @@ mod tests { // Slot per track so two tracks sharing a sector do not // overwrite each other; both stay on the 192-byte grid. let off = (at - u64::from(lba)) as usize * SECTOR_BYTES + idx * 192; - let pkt = bd_pes_packet(t.pid, (i % 16) as u8, &pcs_display(t.forced)); + // PCS + END: a display set cut off before its END is not + // emitted when a stream gap follows it. + let mut set = pcs_display(t.forced); + set[2] = (set.len() - 3) as u8; + set.extend_from_slice(&[0x80, 0, 0]); + let pkt = bd_pes_packet(t.pid, (i % 16) as u8, &set); if off + pkt.len() <= want { buf[off..off + pkt.len()].copy_from_slice(&pkt); } @@ -2279,6 +2297,44 @@ mod tests { ); } + // A window that runs past the top of the 32-bit LBA space is skipped, never wrapped + // onto an unrelated low region. + #[test] + fn a_window_past_the_top_of_the_lba_space_is_not_wrapped() { + struct LbaLog(Vec); + impl SectorSource for LbaLog { + fn read_sectors( + &mut self, + lba: u32, + _count: u16, + buf: &mut [u8], + _recovery: bool, + ) -> crate::error::Result { + self.0.push(lba); + buf.fill(0); + Ok(buf.len()) + } + } + let start = u32::MAX - 10; + let mut title = pgs_title(0x1200, false); + title.extents = vec![Extent { + start_lba: start, + sector_count: 2_000_000, + }]; + let mut reader = LbaLog(Vec::new()); + probe_and_set_forced(&mut reader, &mut title, &mut ForcedProbeCache::new(), None); + assert!( + reader.0.iter().all(|&l| l >= start), + "read below the extent: {:?}", + reader + .0 + .iter() + .filter(|&&l| l < start) + .take(3) + .collect::>() + ); + } + // ...but a COMPLETE read has no unread gap to hide a non-forced set in, so a genuine // single-sign forced track is still promoted. #[test] diff --git a/src/disc/profile.rs b/src/disc/profile.rs index 093a2ae8..29242d29 100644 --- a/src/disc/profile.rs +++ b/src/disc/profile.rs @@ -496,6 +496,26 @@ mod tests { assert_eq!(dvd.video[0].codec, "mpeg2"); } + // Forced is the probe/STN flag OR the label qualifier: either alone is enough. + #[test] + fn subtitle_forced_is_the_flag_or_the_qualifier() { + for (flag, qualifier, want) in [ + (false, LabelQualifier::None, false), + (true, LabelQualifier::None, true), + (false, LabelQualifier::Forced, true), + (true, LabelQualifier::Forced, true), + ] { + let Stream::Subtitle(s) = subtitle("eng", flag, qualifier) else { + unreachable!() + }; + assert_eq!( + SubtitleTrack::from_stream(&s).forced, + want, + "{flag} {qualifier:?}" + ); + } + } + #[test] fn from_disc_hoists_main_and_is_main() { let mut disc = test_disc(); diff --git a/src/disc/scan_order_tests.rs b/src/disc/scan_order_tests.rs index d55d47c4..1d8de8f6 100644 --- a/src/disc/scan_order_tests.rs +++ b/src/disc/scan_order_tests.rs @@ -409,6 +409,40 @@ fn live_unreadable_uk_ro_is_recorded_scan_continues() { } } +#[test] +fn live_unreadable_mkb_is_not_fatal() { + let mut rig = Rig::new(bd_disc(Some(false)), |t| { + t.fail_read = Some(Box::new(|lba, n, _| { + (lba..lba + n as u32) + .contains(&MKB_LBA) + .then(|| check(3, 0x11, 0, 0x28)) + })); + }); + let d = + Disc::scan(&mut rig.drive, &with_hc()).expect("a medium error in the MKB: scan goes on"); + assert!(!d.titles.is_empty()); +} + +#[test] +fn live_unreadable_cert_is_logged_and_not_fatal() { + let mut rig = Rig::new(bd_disc(Some(false)), |t| { + t.fail_read = Some(Box::new(|lba, n, _| { + (lba..lba + n as u32) + .contains(&CERT_LBA) + .then(|| check(3, 0x11, 0, 0x28)) + })); + }); + let (r, ev) = crate::testlog::capture(|| Disc::scan(&mut rig.drive, &with_hc())); + let d = r.expect("a medium error in the cert: scan goes on"); + assert!(!d.titles.is_empty()); + assert!( + aacs_warns(&ev) + .iter() + .any(|e| e.field("phase") == Some("aacs_capture")), + "the cert read failure must be visible: {ev:?}" + ); +} + #[test] fn failed_handshake_cannot_poison_uk_ro_read() { let mut rig = Rig::new(bd_disc(Some(false)), |t| { diff --git a/src/drive/allow.rs b/src/drive/allow.rs index 106945f3..7b225729 100644 --- a/src/drive/allow.rs +++ b/src/drive/allow.rs @@ -117,6 +117,9 @@ mod tests { assert!(allowed_after_cancel(&EJECT, none, FinishEject)); assert!(!allowed_after_cancel(&EJECT, locked, Plain)); assert!(!allowed_after_cancel(&START, none, FinishEject), "LoEj=0"); + // LoEj=1 with Start=1 loads the tray: not an eject. + let load = [0x1B, 0, 0, 0, 0x03, 0]; + assert!(!allowed_after_cancel(&load, none, FinishEject), "Start=1"); // Row 3: 0x3F iff that AGID's bit is set. assert!(allowed_after_cancel(&release(2), agid2, Plain)); assert!(!allowed_after_cancel(&release(1), agid2, Plain)); diff --git a/src/drive/mod.rs b/src/drive/mod.rs index 8e508ba8..8c78c140 100644 --- a/src/drive/mod.rs +++ b/src/drive/mod.rs @@ -492,7 +492,7 @@ impl Drive { buf: &mut [u8], timeout_ms: u32, ) -> Result { - crate::halt::diag::assert_may_block("Drive::exec", Duration::MAX); + crate::halt::diag::assert_may_block("Drive::exec_uncancellable", Duration::MAX); self.dispatch(cdb, dir, buf, timeout_ms) } @@ -507,7 +507,7 @@ impl Drive { timeout_ms: u32, ctx: CleanupCtx, ) -> Result { - crate::halt::diag::assert_may_block("Drive::exec", Duration::MAX); + crate::halt::diag::assert_may_block("Drive::exec_cleanup", Duration::MAX); if self.is_halted() && !allow::allowed_after_cancel(cdb, self.ledger(), ctx) { return Err(Error::Halted); } @@ -849,9 +849,10 @@ impl Drive { } } - /// True when the mounted disc is a DVD (profile family `0x0010..=0x001F`). + /// True when the mounted disc is a DVD (profile family `0x0010..=0x001F`, + /// plus DVD+RW DL `0x002A` and DVD+R DL `0x002B`). pub(crate) fn disc_is_dvd(&mut self) -> bool { - matches!(self.current_profile(), Some(p) if (0x0010..=0x001F).contains(&p)) + matches!(self.current_profile(), Some(p) if (0x0010..=0x001F).contains(&p) || p == 0x002A || p == 0x002B) } /// Initialize drive — drive-prep unlock + init. @@ -2282,7 +2283,8 @@ mod command_tests { )); } - // disc_is_dvd() must match ONLY the DVD profile family (0x0010..=0x001F). + // disc_is_dvd() must match ONLY the DVD profile family (0x0010..=0x001F plus + // the DVD+ DL profiles 0x002A / 0x002B). #[test] fn disc_is_dvd_matches_only_dvd_profile_family() { let probe = |profile: u16| { @@ -2294,7 +2296,9 @@ mod command_tests { // DVD family → DVD (skip drive unlock, run stock for CSS). assert!(probe(0x0010), "DVD-ROM"); assert!(probe(0x0011), "DVD-R"); - assert!(probe(0x001B), "DVD+R DL"); + assert!(probe(0x001B), "DVD+R"); + assert!(probe(0x002A), "DVD+RW DL"); + assert!(probe(0x002B), "DVD+R DL"); // BD/UHD family → NOT DVD (must keep today's unlock path). assert!(!probe(0x0040), "BD-ROM (UHD) must NOT be classed as DVD"); assert!(!probe(0x0041), "BD-R"); @@ -3876,6 +3880,38 @@ mod command_tests { ); } + // A Stop that lands DURING the spin-up settle (the second pause) must wake it too. + #[test] + fn spin_cycle_wakes_from_its_spin_up_settle_when_stopped() { + let RecordingHarness { + drive: mut d, + cdb, + timeouts: _to, + } = recording(TransportOutcome::Ok(0)); + let flag = d.halt_flag(); + let stopper = std::thread::spawn(move || { + // Wait for the START (spin-up) CDB, then stop inside the settle pause. + while cdb.lock().unwrap().get(4) != Some(&0x01) { + std::thread::sleep(std::time::Duration::from_millis(20)); + } + std::thread::sleep(std::time::Duration::from_millis(200)); + flag.store(true, Ordering::Relaxed); + }); + let t0 = std::time::Instant::now(); + let r = d.spin_cycle(); + let took = t0.elapsed(); + stopper.join().expect("stopper thread"); + assert!( + matches!(r, Err(Error::Halted)), + "a Stop during the spin-up settle must surface as Halted: {r:?}" + ); + assert!( + took < std::time::Duration::from_secs(SPIN_DOWN_IDLE_SECS + 3), + "the settle must be halt-aware, not a blind {SPIN_UP_SETTLE_SECS}s \ + thread::sleep; took {took:?}" + ); + } + // A READ(10) with GOOD status but a residual underrun must be refused AND logged. #[test] fn read_logs_a_good_status_short_transfer() { diff --git a/src/drive/stop_tests.rs b/src/drive/stop_tests.rs index 97e12fae..9fa4f82f 100644 --- a/src/drive/stop_tests.rs +++ b/src/drive/stop_tests.rs @@ -122,6 +122,7 @@ fn every_drive_cdb_goes_through_exec() { assert!(d.mode_sense_page(0x2A).is_none()); assert!(d.read_buffer(2, 0xF1, 16).is_none()); assert!(d.report_key_rpc_state().is_none()); + assert!(!d.disc_is_dvd()); assert!(!d.enable_recovered_error_reporting()); assert!(matches!(d.eject(), Err(Error::Halted))); assert!(matches!( @@ -198,6 +199,18 @@ fn read_postcheck_discards_after_cancel() { ); } +/// LD3: every READ-class opcode is checked again on completion, not only READ(10). +#[test] +fn read_class_covers_every_read_opcode() { + for op in [crate::scsi::SCSI_READ_10, 0xA8, 0x88, 0xBE, 0xB9] { + assert!(is_read_class(&[op, 0, 0]), "{op:#04x}"); + } + for op in [SCSI_TEST_UNIT_READY, 0x1E, 0x46, 0x25] { + assert!(!is_read_class(&[op, 0, 0]), "{op:#04x}"); + } + assert!(!is_read_class(&[])); +} + /// LD4 (Drive half): `exec_cleanup` admits the ALLOW for a tray this Drive locked /// and refuses anything else after a cancel. #[test] diff --git a/src/dvdnav/nav.rs b/src/dvdnav/nav.rs index 164bb70b..f411891f 100644 --- a/src/dvdnav/nav.rs +++ b/src/dvdnav/nav.rs @@ -345,7 +345,7 @@ fn run_first_play(cmds: &[[u8; 8]]) -> Option { } => { if tainted { vm.taint_gprm(reg); - if op == 2 { + if op == SET_SWAP { vm.taint_gprm(src); // swap also writes its source } } else { @@ -611,6 +611,49 @@ mod tests { assert_eq!(resolve_from_vmg(&vmgi), None); } + // SetGPRMMD counter mode makes a GPRM time-dependent: a compare on it must abstain, + // including when the mode was set by a line whose own predicate was false. + #[test] + fn counter_mode_gprm_makes_a_later_compare_abstain() { + // 53 .. imm 5, b5 = 0x83: counter bit + g3. Then `if g3 == g3 -> JumpTT 1`. + let cmp = h("3022000000010303"); + for set in ["5300000500830000", "5330000500830000"] { + let vmgi = build_vmgi(&[h(set), cmp], 1, &[(2, 1)]); + assert_eq!(resolve_from_vmg(&vmgi), None, "{set}"); + } + // Counter bit clear: the compare is decidable. + let vmgi = build_vmgi(&[h("5300000500030000"), cmp], 1, &[(2, 1)]); + assert!(resolve_from_vmg(&vmgi).is_some()); + } + + // SetGPRMMD stores like a mov: the immediate (bytes 2-3) and source register (byte 3) + // land in the register named by byte 5. + #[test] + fn setgprmmd_stores_its_immediate_and_source_register() { + // g1 = 9; SetGPRMMD g2 = imm 9 / g2 = g1; then `if g2 == g1 -> JumpTT 1`. + let jump = h("3022000000010201"); + for md in ["5300000900020000", "4300000100020000"] { + let vmgi = build_vmgi(&[h("7100000100090000"), h(md), jump], 1, &[(2, 1)]); + assert!(resolve_from_vmg(&vmgi).is_some(), "{md}"); + } + } + + // A random value is never decidable, and a swap under an undecidable guard leaves + // its source register unknown too. + #[test] + fn rnd_and_guarded_swap_taint_their_registers() { + let mut vm = Vm::new(); + vm.set(0, SET_RND, true, 1, 0); + assert!(vm.reg_tainted(0)); + // line 0: swap g0,g1 guarded by an SPRM compare; line 1: `if g1 == g1 -> JumpTT 1`. + let vmgi = build_vmgi( + &[h("6220800000010000"), h("3022000000010101")], + 1, + &[(2, 1)], + ); + assert_eq!(resolve_from_vmg(&vmgi), None); + } + // A SetSystem (SPRM write, type 2) in the First-Play list is a no-op FOR // RESOLUTION: it must keep executing, not abort, so a following JumpTT still // resolves. `40..` = type 2, zero compare nibble → unconditional. diff --git a/src/dvdnav/vmcmd.rs b/src/dvdnav/vmcmd.rs index b7a9dec3..167902cb 100644 --- a/src/dvdnav/vmcmd.rs +++ b/src/dvdnav/vmcmd.rs @@ -337,6 +337,22 @@ mod tests { assert!(!decode(&h("7101000000010011")).link, "sub-op 0x11 is not"); } + // Every link code from LinkPGCN through LinkCN leaves the pre list; neighbours do not. + #[test] + fn set_link_flag_covers_linkpgcn_through_linkcn() { + for (cmd, link) in [ + (3, false), + (4, true), + (5, true), + (6, true), + (7, true), + (8, false), + ] { + let c = decode(&h(&format!("710{cmd:x}000000000001"))); + assert_eq!(c.link, link, "cmd {cmd}"); + } + } + // Special sub-command 3 (SetTmpPML) sets SPRM13 then gotos byte7 when cond holds. #[test] fn set_tmp_pml_decodes_as_goto() { diff --git a/src/error.rs b/src/error.rs index bfab7c94..ece6ccfc 100644 --- a/src/error.rs +++ b/src/error.rs @@ -1253,8 +1253,10 @@ impl std::fmt::Display for Error { (None, None) => write!(f, "E{}: {}", self.code(), sector), }, Error::Halted => write!(f, "E{}", self.code()), - Error::UdfNotFound { path } => write!(f, "E{}: {}", self.code(), path), - Error::UdfUnrecordedExtent { path } => write!(f, "E{}: {}", self.code(), path), + // Disc- or backup-folder-derived names: escape control characters. + Error::UdfNotFound { path } | Error::UdfUnrecordedExtent { path } => { + write!(f, "E{}: {}", self.code(), path.escape_debug()) + } Error::SeamPlanDroppedMost { dropped, written } => { write!(f, "E{} {dropped}/{written}", self.code()) } @@ -1265,7 +1267,7 @@ impl std::fmt::Display for Error { | Error::DirImageFileChanged { path } | Error::DirNameTooLong { path } | Error::DirImageFanout { path } => { - write!(f, "E{}: {}", self.code(), path) + write!(f, "E{}: {}", self.code(), path.escape_debug()) } Error::DiscTitleRange { index, count } => { write!(f, "E{}: {}/{}", self.code(), index, count) @@ -1331,8 +1333,10 @@ impl std::fmt::Display for Error { Error::TimedOut { op } | Error::WorkerLost { op } => { write!(f, "E{}: {op}", self.code()) } - Error::BusStreamUnmapped { files } => write!(f, "E{}: {files}", self.code()), - Error::ImageScoped { path } => write!(f, "E{}: {path}", self.code()), + Error::BusStreamUnmapped { files } => { + write!(f, "E{}: {}", self.code(), files.escape_debug()) + } + Error::ImageScoped { path } => write!(f, "E{}: {}", self.code(), path.escape_debug()), Error::RemuxVerifyFailed { kind, path } => match kind { RemuxVerifyKind::RuntimeMismatch { have_secs, @@ -1409,21 +1413,21 @@ impl From for std::io::Error { 9000..=9001 => std::io::ErrorKind::Unsupported, 9002..=9008 => std::io::ErrorKind::InvalidInput, // 9010 HevcParamParse: malformed hvcC payload. - 9010 => std::io::ErrorKind::InvalidData, + E_HEVC_PARAM_PARSE => std::io::ErrorKind::InvalidData, // 9011 MuxTrackRange: caller passed a bad track index. - 9011 => std::io::ErrorKind::InvalidInput, + E_MUX_TRACK_RANGE => std::io::ErrorKind::InvalidInput, // 9012 Fmp4Unimplemented: sink can't emit media yet. - 9012 => std::io::ErrorKind::Unsupported, + E_FMP4_UNIMPLEMENTED => std::io::ErrorKind::Unsupported, // 9014 PipelineJoinTimeout: consumer drain exceeded deadline. E_PIPELINE_JOIN_TIMEOUT => std::io::ErrorKind::TimedOut, // 9017 PesTrackTooLarge: out-of-range track index on serialize. - 9017 => std::io::ErrorKind::InvalidInput, + E_PES_TRACK_TOO_LARGE => std::io::ErrorKind::InvalidInput, // 9020 DiscCapacityOverflow: disc reported a capacity sentinel // we can't represent — treat as bad/invalid device data. - 9020 => std::io::ErrorKind::InvalidData, + E_DISC_CAPACITY_OVERFLOW => std::io::ErrorKind::InvalidData, // 9021 M2tsPacketMalformed: a muxer invariant break produced // a non-188-byte packet — treat as invalid data. - 9021 => std::io::ErrorKind::InvalidData, + E_M2TS_PACKET_MALFORMED => std::io::ErrorKind::InvalidData, // 9022 NetworkAddrBlocked: the output host resolved only to // invalid (unspecified/multicast/broadcast/reserved) addresses. E_NETWORK_ADDR_BLOCKED => std::io::ErrorKind::PermissionDenied, @@ -1440,7 +1444,7 @@ impl From for std::io::Error { // truncated — invalid input data. E_MKV_SOURCE_INVALID => std::io::ErrorKind::InvalidData, // 9054 MkvUnencodable: the writer was asked for an element size EBML - // cannot represent. An output-side limit, not bad input. + // cannot represent; the data it was given cannot be encoded. E_MKV_UNENCODABLE => std::io::ErrorKind::InvalidData, // mp4:// demux errors: a malformed/truncated source file // (E_MP4_INVALID), or a source whose tracks the mux can't use — no @@ -1457,10 +1461,10 @@ impl From for std::io::Error { E_SYNC_TIMEOUT | E_SYNC_WORKER_LOST => std::io::ErrorKind::TimedOut, // 9030 ExtentNotUnitAligned: a malformed/non-AACS-aligned // extent was handed to the prefetch producer. - 9030 => std::io::ErrorKind::InvalidInput, + E_EXTENT_NOT_UNIT_ALIGNED => std::io::ErrorKind::InvalidInput, // 9047 DiscCapacityMalformed: the drive returned an unusable // READ CAPACITY response (short transfer / overflow). - 9047 => std::io::ErrorKind::InvalidData, + E_DISC_CAPACITY_MALFORMED => std::io::ErrorKind::InvalidData, // dir:// usage / footgun gates (9019, 9024–9026, 9028): the caller // gave an invalid flag/source/name combination — InvalidInput. E_DIR_RAW_REJECTED @@ -1787,6 +1791,36 @@ mod tests { assert!(shown.contains("A\\u{1b}[2JB\\n"), "{shown}"); } + #[test] + fn disc_derived_names_cannot_inject_terminal_escapes_via_display() { + let n = || "A\x1b[2JB".to_string(); + let errors = [ + Error::UdfNotFound { path: n() }, + Error::UdfUnrecordedExtent { path: n() }, + Error::DirImagePlacement { path: n() }, + Error::DirImageFileChanged { path: n() }, + Error::DirNameTooLong { path: n() }, + Error::DirImageFanout { path: n() }, + Error::BusStreamUnmapped { files: n() }, + Error::ImageScoped { path: n() }, + ]; + for e in errors { + let shown = e.to_string(); + assert!(!shown.chars().any(char::is_control), "{shown:?}"); + assert_eq!(shown, format!("E{}: A\\u{{1b}}[2JB", e.code())); + } + } + + #[test] + fn image_ends_before_read_display_is_lba_then_have_slash_want() { + let e = Error::ImageEndsBeforeRead { + lba: 111, + have: 222, + want: 333, + }; + assert_eq!(e.to_string(), format!("E{}: 111 222/333", e.code())); + } + #[test] fn new_variants_have_distinct_codes() { let codes = [ diff --git a/src/identity.rs b/src/identity.rs index b5849d84..3e5bc912 100644 --- a/src/identity.rs +++ b/src/identity.rs @@ -42,7 +42,8 @@ pub struct DriveId { /// Drive serial number — GET CONFIGURATION Feature 0108h pub serial_number: String, - /// Raw 96-byte INQUIRY response for additional parsing if needed. + /// Raw INQUIRY response for additional parsing if needed: what the drive + /// sent, 36 to 96 bytes. pub raw_inquiry: Vec, /// Raw GET CONFIGURATION Feature 010Ch response bytes. @@ -663,6 +664,18 @@ mod tests { assert_eq!(id.raw_gc_010c.len(), 28, "header + 20-byte descriptor"); } + // MMC-6 §5.3.10: the date is CCYYMMDDHHMI, 12 characters; longer vendor + // payload in the descriptor is not part of it. + #[test] + fn from_drive_firmware_date_is_capped_at_12_characters() { + let mut t = GcReplyTransport { + firmware: gc_reply(0x010C, b"201604250000EXTRA"), + serial: gc_reply(0x0108, b"ABCD1234"), + }; + let id = DriveId::from_drive(&mut t).unwrap(); + assert_eq!(id.firmware_date, "201604250000"); + } + // A drive lacking the feature answers RT=10b with only the 8-byte header. #[test] fn from_drive_absent_or_foreign_gc_feature_yields_empty_fields() { diff --git a/src/ifo.rs b/src/ifo.rs index d3b7e46f..699fe21e 100644 --- a/src/ifo.rs +++ b/src/ifo.rs @@ -60,6 +60,9 @@ pub struct DvdTitle { /// resolution joins on the REAL title number rather than the position in the /// titles vec, which desyncs when a sibling PGC is dropped as unparseable. pub vts_title_num: u8, + /// Distinct PGCs this title's part-of-title entries point at. Only the first is read, + /// so a title with more is missing the rest. + pub pgcs: usize, } /// A cell — contiguous sector range within a VOB. @@ -519,7 +522,8 @@ pub(crate) fn parse_tt_srpt( // ── VTS parser ────────────────────────────────────────────────────────────── -/// Parse VTS_XX_0.IFO for one title set, falling back to its VTS_XX_0.BUP copy. +/// Parse VTS_XX_0.IFO for one title set, falling back to its VTS_XX_0.BUP copy when the +/// IFO fails or keeps fewer titles than `titles_info` declares. /// /// `titles_info` is a list of (chapter_count, vts_title_number) from TT_SRPT. fn parse_vts( @@ -529,19 +533,30 @@ fn parse_vts( titles_info: &[(u16, u8)], ) -> Result { let path = format!("/VIDEO_TS/VTS_{vts_number:02}_0.IFO"); - let err = match parse_vts_file(reader, udf, &path, &path, vts_number, titles_info) { - Ok(ts) => return Ok(ts), + let mut partial = None; + let mut err = None; + match parse_vts_file(reader, udf, &path, &path, vts_number, titles_info) { + Ok(ts) if ts.titles.len() >= titles_info.len() => return Ok(ts), + Ok(ts) => partial = Some(ts), Err(Error::Halted) => return Err(Error::Halted), - Err(e) => e, - }; + Err(e) => err = Some(e), + } let bup = format!("/VIDEO_TS/VTS_{vts_number:02}_0.BUP"); match parse_vts_file(reader, udf, &bup, &path, vts_number, titles_info) { - Ok(ts) => { - tracing::warn!(target: "freemkv::scan", vts = vts_number, code = err.code(), "VTS IFO bad; using BUP"); + Ok(ts) + if partial + .as_ref() + .is_none_or(|p| ts.titles.len() > p.titles.len()) => + { + tracing::warn!(target: "freemkv::scan", vts = vts_number, "VTS IFO bad; using BUP"); Ok(ts) } Err(Error::Halted) => Err(Error::Halted), - Err(_) => Err(err), + _ => match (partial, err) { + (Some(ts), _) => Ok(ts), + (None, Some(e)) => Err(e), + (None, None) => Err(Error::IfoParse), + }, } } @@ -815,6 +830,43 @@ fn ptt_srpt_pgc_index(data: &[u8], ptt_offset: usize, ttn: u8) -> Option pgcn.checked_sub(1) } +// Distinct PGCs title `ttn`'s part-of-title entries point at. A title split across +// PGCs (one per chapter) is read from its first PGC only; the caller warns on > 1. +fn ptt_srpt_pgc_count(data: &[u8], ptt_offset: usize, ttn: u8) -> usize { + let entries = || -> Option> { + let count = be_u16(data, ptt_offset).ok()? as usize; + let last_byte = be_u32(data, ptt_offset + 4).ok()? as usize; + if ttn as usize > count { + return None; + } + let at = |t: usize| { + be_u32(data, ptt_offset + 8 + (t - 1) * 4) + .ok() + .map(|r| r as usize) + }; + let start = at(ttn as usize)?; + let end = if (ttn as usize) < count { + at(ttn as usize + 1)? + } else { + last_byte.checked_add(1)? + }; + let mut pgcns = Vec::new(); + let mut rel = start; + while rel.checked_add(4)? <= end { + pgcns.push(be_u16(data, ptt_offset.checked_add(rel)?).ok()?); + rel += 4; + } + Some(pgcns) + }; + if ptt_offset == 0 || ttn == 0 { + return 0; + } + let mut pgcns = entries().unwrap_or_default(); + pgcns.sort_unstable(); + pgcns.dedup(); + pgcns.len() +} + /// Parse VTS_PGCIT (Program Chain Information Table) to extract titles. /// `ptt_offset` is the VTS_PTT_SRPT byte offset (0 = absent). pub(crate) fn parse_pgcit( @@ -855,6 +907,9 @@ pub(crate) fn parse_pgcit( continue; } + // Warned once per disc by the DVD scan, which sees every title set. + let pgcs = ptt_srpt_pgc_count(data, ptt_offset, vts_title_num); + let entry_offset = entries_start + pgc_index * 8; if entry_offset + 8 > data.len() { skipped += 1; @@ -879,6 +934,7 @@ pub(crate) fn parse_pgcit( // Stamp the REAL title number so downstream nav joins survive a // dropped sibling PGC (position in `titles` is not vts_title_num). title.vts_title_num = vts_title_num; + title.pgcs = pgcs; titles.push(title); } Err(e) => { @@ -983,7 +1039,8 @@ pub(crate) fn parse_pgc(data: &[u8], pgc_offset: usize, chapters: u16) -> Result let cell_base = pgc_offset + cell_playback_offset; for i in 0..num_cells { let co = cell_base + i * 24; - if co + 8 > data.len() { + // Same bound as the cell loop above: a cell that is not kept adds nothing. + if co + 24 > data.len() { cell_frames.push((0, 0.0)); continue; } @@ -1058,6 +1115,7 @@ pub(crate) fn parse_pgc(data: &[u8], pgc_offset: usize, chapters: u16) -> Result spst_ctl, // Set by the caller (parse_pgcit) which knows the TT_SRPT title number. vts_title_num: 0, + pgcs: 0, }) } @@ -2011,6 +2069,7 @@ mod tests { ast_ctl: [0; 8], spst_ctl: [0; 32], vts_title_num: 0, + pgcs: 1, }; assert_eq!(t.feature_start_cell(), 0); assert_eq!(t.feature_cells().len(), 3); @@ -2035,6 +2094,7 @@ mod tests { ast_ctl: [0; 8], spst_ctl: [0; 32], vts_title_num: 0, + pgcs: 1, }; assert_eq!(t.feature_start_cell(), 2); let fc = t.feature_cells(); @@ -2055,6 +2115,7 @@ mod tests { ast_ctl: [0; 8], spst_ctl: [0; 32], vts_title_num: 0, + pgcs: 1, }; assert_eq!(t.feature_start_cell(), 0); assert_eq!(t.feature_cells().len(), 2); @@ -2072,6 +2133,7 @@ mod tests { ast_ctl: [0; 8], spst_ctl: [0; 32], vts_title_num: 0, + pgcs: 1, }; assert_eq!(t.feature_start_cell(), 0); assert!(t.feature_cells().is_empty()); @@ -2401,6 +2463,25 @@ mod tests { assert_eq!(title.chapter_times, vec![0.0, 10.01]); } + // A cell cut off mid-record is not in `cells`, so it adds no time to a chapter mark. + #[test] + fn pgc_chapter_times_ignore_a_cell_cut_off_mid_record() { + let mut pgc = build_pgc( + bcd_secs(59), + &[ + (0x00, bcd_secs(10), 0, 9), + (0x00, bcd_secs(20), 10, 19), + (0x00, bcd_secs(31), 20, 29), + ], + &[1, 3], + None, + ); + pgc.truncate(0xEA + 2 + 24 + 12); + let title = parse_pgc(&pgc, 0, 2).unwrap(); + assert_eq!(title.cells.len(), 1); + assert_eq!(title.chapter_times, vec![0.0, 10.01]); + } + // A program map running past the end of the data must stop at the buffer // end. Fixture map begins at PGC+0xEA, buffer holds 50 bytes from there, // so a declared 255 programs must yield exactly 50 chapter times. @@ -2705,6 +2786,56 @@ mod tests { d } + // The title number and its entry offset are both bounded by the table: bytes beyond + // it are never read as a PGCN. The table sits at byte 8 (offset 0 means absent). + #[test] + fn ptt_srpt_pgc_index_stays_inside_the_table() { + let put = + |d: &mut Vec, at: usize, v: &[u8]| d[8 + at..8 + at + v.len()].copy_from_slice(v); + let mut d = vec![0u8; 8 + 24]; + put(&mut d, 0, &1u16.to_be_bytes()); + put(&mut d, 4, &23u32.to_be_bytes()); + put(&mut d, 8, &16u32.to_be_bytes()); + put(&mut d, 12, &20u32.to_be_bytes()); + put(&mut d, 16, &3u16.to_be_bytes()); + put(&mut d, 20, &9u16.to_be_bytes()); + assert_eq!(ptt_srpt_pgc_index(&d, 8, 1), Some(2)); + assert_eq!( + ptt_srpt_pgc_index(&d, 8, 2), + None, + "title 2 is past the count" + ); + // The declared table ends before the entry. + put(&mut d, 4, &15u32.to_be_bytes()); + assert_eq!( + ptt_srpt_pgc_index(&d, 8, 1), + None, + "entry outside the table" + ); + } + + #[test] + fn ptt_srpt_counts_the_pgcs_a_title_spans() { + // Title 1: one PTT per PGC (1, 2, 3); title 2: two PTTs in PGC 4. + let entries: [(u16, u16); 5] = [(1, 1), (2, 1), (3, 1), (4, 1), (4, 2)]; + let total = 8 + 2 * 4 + entries.len() * 4; + let mut d = vec![0u8; total]; + d[0..2].copy_from_slice(&2u16.to_be_bytes()); + d[4..8].copy_from_slice(&((total - 1) as u32).to_be_bytes()); + d[8..12].copy_from_slice(&16u32.to_be_bytes()); + d[12..16].copy_from_slice(&28u32.to_be_bytes()); + for (i, (pgcn, pgn)) in entries.iter().enumerate() { + d[16 + i * 4..18 + i * 4].copy_from_slice(&pgcn.to_be_bytes()); + d[18 + i * 4..20 + i * 4].copy_from_slice(&pgn.to_be_bytes()); + } + assert_eq!(ptt_srpt_pgc_count(&d, 0, 1), 0); + let mut buf = vec![0u8; 4]; + buf.extend_from_slice(&d); + assert_eq!(ptt_srpt_pgc_count(&buf, 4, 1), 3); + assert_eq!(ptt_srpt_pgc_count(&buf, 4, 2), 1); + assert_eq!(ptt_srpt_pgc_count(&buf, 4, 3), 0); + } + /// TTN -> PGC goes through VTS_PTT_SRPT, not "TTN N is PGCIT entry N"; the PGCIT /// sits at a NON-ZERO offset so the pgcit base term is exercised, and the real TTN /// is stamped on each title (disc/dvd.rs joins on it). @@ -2890,6 +3021,71 @@ mod tests { assert_eq!(ts.titles[0].duration_secs, 22.022, "TTN 1 -> PGC 2 via PTT"); } + fn bup_fixture() -> (Vec, Vec) { + let two = vec![ + build_pgc(bcd_secs(11), &[(0x00, bcd_secs(11), 0, 9)], &[], None), + build_pgc(bcd_secs(22), &[(0x00, bcd_secs(22), 50, 59)], &[], None), + ]; + let good = build_vts_ifo(0, 0, &two, None); + let mut bad = good.clone(); + bad[0] = b'X'; + (bad, good) + } + + /// A bad IFO falls back to the BUP, but the title VOBS stay anchored on the IFO's LBA. + #[test] + fn parse_vts_falls_back_to_the_bup_and_anchors_on_the_ifo() { + let (bad, good) = bup_fixture(); + let (mut disc, udf) = video_ts_disc(vec![("VTS_01_0.IFO", bad), ("VTS_01_0.BUP", good)]); + let ts = parse_vts(&mut disc, &udf, 1, &[(1, 1), (1, 2)]).expect("BUP used"); + assert_eq!(ts.titles.len(), 2); + let lba = udf + .file_start_lba(&mut disc, "/VIDEO_TS/VTS_01_0.IFO") + .unwrap(); + assert_eq!(ts.vob_start_sector, lba + 7); + } + + /// An IFO that parses but drops a damaged PGC defers to a BUP that keeps it. + #[test] + fn parse_vts_prefers_a_bup_that_keeps_more_titles() { + let (_, good) = bup_fixture(); + let mut damaged = good.clone(); + // Second PGC entry's offset: point it past the end of the table. + damaged[2 * 2048 + 8 + 8 + 4..2 * 2048 + 8 + 8 + 8] + .copy_from_slice(&0x00ff_ffffu32.to_be_bytes()); + let (mut disc, udf) = video_ts_disc(vec![ + ("VTS_01_0.IFO", damaged.clone()), + ("VTS_01_0.BUP", good), + ]); + let ts = parse_vts(&mut disc, &udf, 1, &[(1, 1), (1, 2)]).expect("vts"); + assert_eq!(ts.titles.len(), 2, "BUP kept both titles"); + // With no usable BUP the IFO's partial result stands. + let (mut disc, udf) = video_ts_disc(vec![("VTS_01_0.IFO", damaged)]); + let ts = parse_vts(&mut disc, &udf, 1, &[(1, 1), (1, 2)]).expect("vts"); + assert_eq!(ts.titles.len(), 1); + } + + /// A stop during the BUP attempt is a stop, not a reason to report the IFO's error. + #[test] + fn parse_vts_propagates_a_halt_during_the_bup_attempt() { + struct HaltAt(MemDisc, u32); + impl SectorSource for HaltAt { + fn read_sectors(&mut self, lba: u32, n: u16, buf: &mut [u8], r: bool) -> Result { + if lba == self.1 { + return Err(Error::Halted); + } + self.0.read_sectors(lba, n, buf, r) + } + } + let (bad, good) = bup_fixture(); + let (mut disc, udf) = video_ts_disc(vec![("VTS_01_0.IFO", bad), ("VTS_01_0.BUP", good)]); + let bup_lba = udf + .file_start_lba(&mut disc, "/VIDEO_TS/VTS_01_0.BUP") + .unwrap(); + let r = parse_vts(&mut HaltAt(disc, bup_lba), &udf, 1, &[(1, 1)]); + assert!(matches!(r, Err(Error::Halted)), "{r:?}"); + } + #[test] fn parse_vts_rejects_bad_magic_and_short_files() { let mut bad_magic = build_vts_ifo(0, 0, &one_pgc(), None); diff --git a/src/io/block_sink.rs b/src/io/block_sink.rs index 4e7865c6..76ac362b 100644 --- a/src/io/block_sink.rs +++ b/src/io/block_sink.rs @@ -128,9 +128,8 @@ impl BlockSink for NullBlockSink { } /// Open the block output `url`: `iso://` (an image, created now) or `null://`. -/// `dir://` is the folder output of a whole-disc extraction ([`crate::Disc::extract_tree`]); -/// a PES output scheme is refused with [`Error::StreamWriteOnly`]'s sibling -/// [`Error::StreamUrlInvalid`]. +/// `dir://` is the folder output of a whole-disc extraction ([`crate::Disc::extract_tree`]) +/// and is refused here, as is a PES output scheme, with [`Error::StreamUrlInvalid`]. pub fn open_block_sink(url: &str) -> Result> { match crate::mux::parse_url(url) { crate::mux::StreamUrl::Iso { path } => { diff --git a/src/io/file_sector_source/macos.rs b/src/io/file_sector_source/macos.rs index 54df7c74..a7176f1e 100644 --- a/src/io/file_sector_source/macos.rs +++ b/src/io/file_sector_source/macos.rs @@ -34,6 +34,7 @@ pub(crate) fn prefetch(file: &File, offset: u64, len: u64) { ra_count: bytes as libc::c_int, }; // Best-effort — kernel hint only. + // SAFETY: `file` is a live borrowed fd and `ra` a live stack radvisory for the call. unsafe { libc::fcntl(file.as_raw_fd(), libc::F_RDADVISE, &mut ra); } diff --git a/src/io/file_sector_source/mod.rs b/src/io/file_sector_source/mod.rs index 61772624..1a0ae225 100644 --- a/src/io/file_sector_source/mod.rs +++ b/src/io/file_sector_source/mod.rs @@ -36,8 +36,8 @@ use crate::sector::SectorSource; use crate::consts::{SECTOR_BYTES, SECTOR_BYTES_U64}; -// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop; mirrors WRITEBACK_CHUNK_BYTES. 32 -// MiB is empirically tuned (7200rpm HDD/SATA). +// Bytes-read threshold per `posix_fadvise(DONTNEED)` drop; mirrors WRITEBACK_CHUNK_BYTES_DEFAULT. +// 32 MiB is empirically tuned (7200rpm HDD/SATA). const READ_DROP_CHUNK_BYTES_DEFAULT: u64 = 32 * 1024 * 1024; // Max MiB from FREEMKV_READ_DROP_CHUNK_MIB: 64 GiB, and small enough n*1024*1024 can't overflow @@ -661,4 +661,52 @@ mod tests { ); let _ = std::fs::remove_file(&path); } + + // A padded read of one sector from a longer file takes only that sector, and an lba past + // the file's end reads as zeros. + #[test] + fn open_padded_clamps_each_read_to_its_own_sectors() { + let td = tempfile::tempdir().unwrap(); + let path = td.path().join("p.bin"); + std::fs::write(&path, vec![7u8; 3 * 2048 + 100]).unwrap(); + let mut s = FileSectorSource::open_padded(&path).unwrap(); + let mut buf = vec![0xAAu8; 2048]; + assert_eq!(s.read_sectors(1, 1, &mut buf, false).unwrap(), 2048); + assert!(buf.iter().all(|&b| b == 7)); + assert_eq!(s.read_sectors(9, 1, &mut buf, false).unwrap(), 2048); + assert!(buf.iter().all(|&b| b == 0), "past EOF is the synthetic pad"); + } + + // The 32-bit LBA ceiling: u32::MAX sectors opens, one more is IsoTooLarge, for both + // open paths. Sparse files; skipped where the filesystem refuses the length. + #[test] + fn open_refuses_a_file_past_the_lba_space() { + let td = tempfile::tempdir().unwrap(); + let path = td.path().join("huge.bin"); + let f = std::fs::File::create(&path).unwrap(); + let max = u32::MAX as u64 * SECTOR_BYTES_U64; + if f.set_len(max).is_err() { + return; + } + assert_eq!( + FileSectorSource::open(&path).unwrap().capacity_sectors(), + u32::MAX + ); + assert_eq!( + FileSectorSource::open_padded(&path) + .unwrap() + .capacity_sectors(), + u32::MAX + ); + f.set_len(max + SECTOR_BYTES_U64).unwrap(); + assert!(matches!( + FileSectorSource::open(&path), + Err(Error::IsoTooLarge { .. }) + )); + f.set_len(max + 1).unwrap(); + assert!(matches!( + FileSectorSource::open_padded(&path), + Err(Error::IsoTooLarge { .. }) + )); + } } diff --git a/src/io/flush/tests.rs b/src/io/flush/tests.rs index 6f2acc4b..cf93d385 100644 --- a/src/io/flush/tests.rs +++ b/src/io/flush/tests.rs @@ -705,6 +705,107 @@ fn durable_sync_file_stall_and_stop() { assert!(t.elapsed() < SLACK, "{:?}", t.elapsed()); } +// A flush whose primitives fail or whose counter jumps by `step` per sample. +struct ScriptedOps { + range_err: bool, + finish_err: bool, + finish_sleep: Duration, + step: AtomicU64, +} + +impl ScriptedOps { + fn new() -> Self { + Self { + range_err: false, + finish_err: false, + finish_sleep: Duration::ZERO, + step: AtomicU64::new(0), + } + } +} + +impl FlushOps for ScriptedOps { + fn chunk(&self, _: &File) -> io::Result<()> { + Ok(()) + } + fn range(&self, _: &File, _: u64, _: u64) -> Option> { + self.range_err.then(|| Err(io::Error::from_raw_os_error(5))) + } + fn finish(&self, _: &File) -> io::Result<()> { + std::thread::sleep(self.finish_sleep); + if self.finish_err { + return Err(io::Error::from_raw_os_error(5)); + } + Ok(()) + } + fn sample(&self) -> Option { + (self.step.load(Ordering::SeqCst) > 0) + .then(|| self.step.fetch_add(1 << 40, Ordering::SeqCst)) + } +} + +/// A failed range sync or final flush is the call's error, never a quiet success. +#[test] +fn durable_sync_file_surfaces_range_and_finish_errors() { + let (_d, f) = sized_file(8 * K); + let t = durable_timing(Duration::from_secs(30)); + let range = ScriptedOps { + range_err: true, + ..ScriptedOps::new() + }; + let e = durable_sync_file_with(&f, None, |_, _| {}, Arc::new(range), t) + .expect_err("a failed range sync fails"); + assert_eq!(e.raw_os_error(), Some(5), "{e}"); + let finish = ScriptedOps { + finish_err: true, + ..ScriptedOps::new() + }; + let e = durable_sync_file_with(&f, None, |_, _| {}, Arc::new(finish), t) + .expect_err("a failed final flush fails"); + assert_eq!(e.raw_os_error(), Some(5), "{e}"); +} + +/// A sampled counter past the file length (other writers on the mount) never reports over +/// 100% and still ends with the final `(len, len)` report. +#[test] +fn durable_sync_file_clamps_sampled_progress_below_the_length() { + let ops = ScriptedOps { + finish_sleep: W * 3, + step: AtomicU64::new(1), + ..ScriptedOps::new() + }; + let (_d, f) = sized_file(8 * K); + let mut seen = Vec::new(); + durable_sync_file_with( + &f, + None, + |d, n| seen.push((d, n)), + Arc::new(ops), + durable_timing(W), + ) + .unwrap(); + assert!(seen.len() >= 2, "sampled while blocked: {seen:?}"); + assert!(seen.iter().all(|(d, n)| d <= n), "over 100%: {seen:?}"); + assert_eq!(seen.last(), Some(&(8 * K, 8 * K))); +} + +/// An empty file still gets its one completion report. +#[test] +fn durable_sync_file_reports_an_empty_file_once() { + let (_d, f) = sized_file(0); + let mut seen = Vec::new(); + let t = durable_timing(W); + durable_sync_file_with( + &f, + None, + |d, n| seen.push((d, n)), + Arc::new(ScriptedOps::new()), + t, + ) + .unwrap(); + assert_eq!(seen, [(0, 0)]); +} + const MOUNTSTATS: &str = "\ device rootfs mounted on / with fstype rootfs device nas:/export mounted on /mnt/nas with fstype nfs4 statvers=1.1 diff --git a/src/io/image_writer.rs b/src/io/image_writer.rs index 6fedbbd9..280ec6da 100644 --- a/src/io/image_writer.rs +++ b/src/io/image_writer.rs @@ -339,6 +339,7 @@ mod tests { let err = write_image(&mut src, &dest, 100_000, &Ctx::new(halt.clone())).expect_err("must halt"); assert!(matches!(err, Error::Halted), "got {err:?}"); + assert!(dest.exists(), "a halt must keep the partial image"); let _ = std::fs::remove_file(&dest); } diff --git a/src/io/pipeline.rs b/src/io/pipeline.rs index f38ec21a..083ae07e 100644 --- a/src/io/pipeline.rs +++ b/src/io/pipeline.rs @@ -40,8 +40,8 @@ impl Default for JoinTiming { } } -// Cached FREEMKV_DEBUG=1 lookup — called per item on the mux hot loop, -// so the env lock is paid once, not per call. +// Cached FREEMKV_DEBUG lookup ("1", "true" or "yes" enables per-item debug tracing) — called +// per item on the mux hot loop, so the env lock is paid once, not per call. pub fn debug_enabled() -> bool { static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); *ENABLED.get_or_init(|| { diff --git a/src/io/pipeline/stop_tests.rs b/src/io/pipeline/stop_tests.rs index a7080955..5dbe337c 100644 --- a/src/io/pipeline/stop_tests.rs +++ b/src/io/pipeline/stop_tests.rs @@ -155,23 +155,23 @@ fn closing_before_cancel_is_done() { ); } -/// LP4a (T7, stall pair a): one item applied per 0.5 × window for 4 windows is +/// LP4a (T7, stall pair a): one item applied per 0.25 × window for 4 windows is /// progress, so the join waits and returns `Ok` — 600 s is a stall window, not a total. #[test] fn join_rearms_on_consumer_progress() { let count = Arc::new(AtomicUsize::new(0)); let sink = SlowSinkFor { - delay: WINDOW / 2, + delay: WINDOW / 4, count: count.clone(), }; let pipe = Pipeline::spawn(16, sink).unwrap(); - for _ in 0..8 { + for _ in 0..16 { pipe.send(()).unwrap(); } let t = Instant::now(); let r = pipe.finish_with_halt_timing(None, timing()); assert!( - matches!(r, Ok(8)), + matches!(r, Ok(16)), "a progressing consumer is not a stall: {r:?}" ); assert!( diff --git a/src/io/tree_sink.rs b/src/io/tree_sink.rs index 88cd4b3a..cf7a8d35 100644 --- a/src/io/tree_sink.rs +++ b/src/io/tree_sink.rs @@ -7,7 +7,7 @@ use crate::error::{Error, Result}; use std::collections::HashMap; -use std::io::Write; +use std::io::{self, Write}; use std::path::{Path, PathBuf}; /// Attempts to delete the case-probe marker before giving up. @@ -30,8 +30,14 @@ impl TreeSink { std::fs::create_dir_all(dest).map_err(|e| Error::DirWriteFailed { errno: e.raw_os_error(), })?; - if !force && dir_is_non_empty(dest) { - return Err(Error::DirNotEmpty); + if !force { + // An unlistable folder is not known to be empty: refuse rather than mix trees. + let non_empty = dir_is_non_empty(dest).map_err(|e| Error::DirWriteFailed { + errno: e.raw_os_error(), + })?; + if non_empty { + return Err(Error::DirNotEmpty); + } } Ok(TreeSink { dest: dest.to_path_buf(), @@ -389,11 +395,9 @@ pub(crate) fn available_space(_dir: &Path) -> Option { None } -/// Whether a directory exists and contains any entry. -pub(crate) fn dir_is_non_empty(dir: &Path) -> bool { - std::fs::read_dir(dir) - .map(|mut it| it.next().is_some()) - .unwrap_or(false) +/// Whether a directory contains any entry; an unlistable one is an error. +pub(crate) fn dir_is_non_empty(dir: &Path) -> io::Result { + Ok(std::fs::read_dir(dir)?.next().is_some()) } #[cfg(test)] @@ -421,6 +425,23 @@ mod tests { assert!(!dir.path().join("a.bin.partial").exists()); } + #[cfg(unix)] + #[test] + fn an_unlistable_target_is_refused_not_treated_as_empty() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let out = dir.path().join("tree"); + std::fs::create_dir(&out).unwrap(); + std::fs::write(out.join("old.bin"), b"x").unwrap(); + std::fs::set_permissions(&out, std::fs::Permissions::from_mode(0o300)).unwrap(); + let listable = std::fs::read_dir(&out).is_ok(); + let r = TreeSink::create(&out, false); + std::fs::set_permissions(&out, std::fs::Permissions::from_mode(0o700)).unwrap(); + if !listable { + assert!(matches!(r, Err(Error::DirWriteFailed { .. }))); + } + } + #[test] fn the_aacs_directories_are_not_part_of_the_tree() { assert!(!TreeSink::keeps_top_level("AACS", false)); diff --git a/src/io/writeback/linux.rs b/src/io/writeback/linux.rs index e738a9d8..fd1f0715 100644 --- a/src/io/writeback/linux.rs +++ b/src/io/writeback/linux.rs @@ -351,11 +351,10 @@ impl WritebackPipeline { } } - /// Caller is about to seek away from the current write region. - /// Drain any in-flight chunk and reset tracking. - // MKV seeks every cluster, so never wait here: kick the tail and let the - // pending chunk wait at the next boundary. An un-waited range keeps its error - // for the final fsync. + /// Caller is about to seek away from the current write region. Kicks off + /// the tail and resets tracking; never waits (MKV seeks every cluster), so + /// the pending chunk is waited at the next boundary or `finalize`, and an + /// un-waited range keeps its error for the final fsync. pub(crate) fn handle_seek(&mut self, new_pos: u64) { let tail_len = self.pos.saturating_sub(self.last_flush_pos); if tail_len > 0 && self.waitable { @@ -745,6 +744,50 @@ mod tests { assert_eq!(p.error().and_then(|e| e.raw_os_error()), Some(libc::EIO)); } + static ERRNO_WAITS: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + + fn eio_then_erofs_wait(_fd: RawFd, _off: u64, _len: u64) -> i32 { + match ERRNO_WAITS.fetch_add(1, Ordering::SeqCst) { + 0 => libc::EIO, + _ => libc::EROFS, + } + } + + // The first data-loss errno is the cause; a later one never replaces it. + #[test] + fn the_first_latched_errno_is_kept() { + let (_f, mut p) = local_pipeline(CHUNK_BYTES_MIN); + p.wait_op = eio_then_erofs_wait; + p.note_progress(CHUNK_BYTES_MIN); + p.note_progress(2 * CHUNK_BYTES_MIN); + p.note_progress(3 * CHUNK_BYTES_MIN); + p.finalize(); + assert!(ERRNO_WAITS.load(Ordering::SeqCst) >= 2, "a second wait ran"); + assert_eq!(p.error().and_then(|e| e.raw_os_error()), Some(libc::EIO)); + } + + fn ok_wait(_fd: RawFd, _off: u64, _len: u64) -> i32 { + 0 + } + + // Each completed WAIT_AFTER credits exactly the waited chunk's length as durable, at a + // boundary and at finalize. + #[test] + fn completed_waits_credit_the_waited_length() { + let (_f, mut p) = local_pipeline(CHUNK_BYTES_MIN); + let flush = crate::io::flush::FlushProgress::default(); + flush.note_total(10 * CHUNK_BYTES_MIN); + p.set_flush_progress(flush.clone()); + p.wait_op = ok_wait; + p.note_progress(CHUNK_BYTES_MIN); + assert_eq!(flush.bytes_durable(), 0, "nothing waited yet"); + // A second chunk twice as long: the wait on the first credits only its length. + p.note_progress(3 * CHUNK_BYTES_MIN); + assert_eq!(flush.bytes_durable(), CHUNK_BYTES_MIN); + p.finalize(); + assert_eq!(flush.bytes_durable(), 3 * CHUNK_BYTES_MIN); + } + // Only data-loss errnos latch: ESPIPE/EINVAL mean the call was unsupported. #[test] fn non_writeback_errno_is_not_latched() { diff --git a/src/io/writeback_file/flusher.rs b/src/io/writeback_file/flusher.rs index f3e2e6eb..d2973f35 100644 --- a/src/io/writeback_file/flusher.rs +++ b/src/io/writeback_file/flusher.rs @@ -1,6 +1,6 @@ //! The in-write flusher (stop design §2.10 items 1–4): one worker per file makes written -//! bytes durable in chunks of `C` while the writer runs. `C` halves (to a floor) when one -//! chunk flush is slow; the writer waits, halt-aware, while more than `2 × C` is unflushed; +//! bytes durable in chunks of `C` while the writer runs. `C` starts at a floor, halves when one +//! chunk flush is slow and doubles (to a ceiling) when one is fast; the writer waits, halt-aware, while more than `2 × C` is unflushed; //! a flusher with no progress for the stall window latches `SyncTimeout` (sticky). use std::fs::File; @@ -329,4 +329,50 @@ mod tests { writer.join().unwrap().unwrap(); assert!(f.error().is_none()); } + + struct FailingChunk; + impl FlushOps for FailingChunk { + fn chunk(&self, _: &File) -> io::Result<()> { + Err(io::Error::from_raw_os_error(28)) + } + fn range(&self, _: &File, _: u64, _: u64) -> Option> { + None + } + fn finish(&self, _: &File) -> io::Result<()> { + Ok(()) + } + fn sample(&self) -> Option { + None + } + } + + // A failed chunk flush latches its own errno: the waiter gets the real error at once, + // not `SyncTimeout` after the stall window, and it stays latched. + #[test] + fn a_chunk_error_is_latched_and_reported_as_itself() { + let file = tempfile::tempfile().unwrap(); + let timing = FlushTiming { + stall: Duration::from_millis(400), + slow_chunk: Duration::from_secs(10), + chunk_min: 100, + chunk_max: 100, + sample_every: Duration::from_millis(50), + }; + let f = Flusher::spawn( + &file, + Arc::new(FailingChunk), + timing, + FlushProgress::default(), + 0, + ) + .unwrap(); + let start = Instant::now(); + let e = f.drain(100, None, None).expect_err("the chunk failed"); + assert_eq!(e.raw_os_error(), Some(28), "{e}"); + assert!( + start.elapsed() < Duration::from_millis(300), + "waited a stall" + ); + assert_eq!(f.error().and_then(|e| e.raw_os_error()), Some(28)); + } } diff --git a/src/io/writeback_file/mod.rs b/src/io/writeback_file/mod.rs index 0c47104a..8aa4f002 100644 --- a/src/io/writeback_file/mod.rs +++ b/src/io/writeback_file/mod.rs @@ -695,6 +695,54 @@ mod tests { assert!(w.hinted, "reservation released before a successful sync"); } + // Rebinding the flush counters carries the length so far, makes the new counters the + // file's own, and moves a running flusher's durable credit onto them. + #[test] + fn set_flush_progress_rebinds_every_holder() { + struct OkOps; + impl FlushOps for OkOps { + fn chunk(&self, _: &File) -> io::Result<()> { + Ok(()) + } + fn range(&self, _: &File, _: u64, _: u64) -> Option> { + None + } + fn finish(&self, _: &File) -> io::Result<()> { + Ok(()) + } + fn sample(&self) -> Option { + None + } + } + let mut file = tempfile::tempfile().unwrap(); + file.write_all(b"12345").unwrap(); + let timing = FlushTiming { + chunk_min: 100, + chunk_max: 100, + ..FlushTiming::default() + }; + let mut w = WritebackFile::with_flush_ops(file, Arc::new(OkOps), timing).unwrap(); + w.write_all(&[0u8; 100]).unwrap(); + let fresh = FlushProgress::default(); + w.set_flush_progress(fresh.clone()); + assert_eq!( + fresh.bytes_total(), + 105, + "the length so far is carried over" + ); + w.write_all(&[0u8; 300]).unwrap(); + assert_eq!(w.flush_progress().bytes_total(), 405); + assert_eq!(fresh.bytes_total(), 405, "writes count on the new counters"); + let start = std::time::Instant::now(); + while fresh.bytes_durable() == 0 && start.elapsed() < std::time::Duration::from_secs(2) { + std::thread::sleep(std::time::Duration::from_millis(5)); + } + assert!( + fresh.bytes_durable() > 0, + "the flusher credits the new counters" + ); + } + // `sync_all` is idempotent: calling it twice, then Drop (also // finalizes), must not corrupt data or panic. #[test] diff --git a/src/keys/evidence.rs b/src/keys/evidence.rs index ceb8cfa2..12793fbd 100644 --- a/src/keys/evidence.rs +++ b/src/keys/evidence.rs @@ -55,10 +55,24 @@ impl Disc { pub enum Detector { /// BD/UHD CPI flags: keys are proven on ciphertext. Verified, - /// HD DVD: the test is unverified, so a single declared key is applied without proof. + /// HD DVD: the flag is unverified, so a unit reads clear only when it is also whole + /// content; encrypted content gets a single declared key applied without proof. Unverified, } +impl Detector { + // The format's own test of one aligned unit, feeding the one encryption decision + // (`resolve::acquire`). Verified: the CPI flag. Unverified: encrypted unless the unit is + // unflagged and structurally whole content (an AACS unit's ciphertext is not). + pub(crate) fn unit_encrypted(self, unit: &[u8], format: ContentFormat) -> bool { + use crate::aacs::content::{aacs_unit_encrypted, is_clean}; + match self { + Detector::Verified => aacs_unit_encrypted(unit, format), + Detector::Unverified => aacs_unit_encrypted(unit, format) || !is_clean(unit, format), + } + } +} + /// One unit of key proof (design §2.2): a stream file, a title extent no file covers, or a /// whole loose file. Spans are sectors `(start, count, unit anchor)`, sorted pieces by /// first sector. @@ -224,10 +238,8 @@ impl KeyEvidence { }); let whole = ev.scope == KeyScope::WholeDisc; if ev.detector == Detector::Unverified { - // Acquisition refuses a multi-key HD DVD before it lists any file. - if n_declared != Some(1) { - return Ok(ev); - } + // Listed whatever the declared count: acquisition judges a copy in the clear from + // its pieces before it refuses a multi-key HD DVD. let files = match crate::whole_disc::content_files(reader) { Ok(f) => f, Err(e) if whole => return Err(e), @@ -395,37 +407,22 @@ impl<'a> Sampler<'a> { read_unit(self.reader, lba) } - // Up to 32 units on `p`'s grid (KU §2.3 step 7), skipping FMTS segment units. + // Up to 32 units on `p`'s grid (KU §2.3 step 7), skipping FMTS segment units, each + // judged by `detector`. pub(crate) fn probe( &mut self, p: &Piece, segments: &[(u32, u32)], format: ContentFormat, + detector: Detector, halt: &Halt, ) -> Result<(u64, Vec)> { - let grid = p.grid(); - let units: u64 = grid.iter().map(|g| g.1).sum(); + let (units, lbas) = probe_lbas(p, segments); let mut out = Vec::new(); - for idx in probe_units(units) { + for lba in lbas { halt.check()?; - let mut k = idx; - let Some(&(head, _)) = grid.iter().find(|g| { - let hit = k < g.1; - if !hit { - k -= g.1; - } - hit - }) else { - continue; - }; - let Ok(lba) = u32::try_from(head + k * UNIT) else { - continue; - }; - if in_segment(segments, lba) { - continue; - } out.push(match self.unit(lba)? { - Some(u) if crate::aacs::content::aacs_unit_encrypted(&u, format) => Sample::Enc(u), + Some(u) if detector.unit_encrypted(&u, format) => Sample::Enc(u), Some(_) => Sample::Clear, None => Sample::Fault, }); @@ -440,6 +437,33 @@ impl<'a> Sampler<'a> { } } +// The units `probe` samples on `p`'s grid: (the grid's unit count, their LBAs), FMTS +// segment units skipped. +fn probe_lbas(p: &Piece, segments: &[(u32, u32)]) -> (u64, Vec) { + let grid = p.grid(); + let units: u64 = grid.iter().map(|g| g.1).sum(); + let mut out = Vec::new(); + for idx in probe_units(units) { + let mut k = idx; + let Some(&(head, _)) = grid.iter().find(|g| { + let hit = k < g.1; + if !hit { + k -= g.1; + } + hit + }) else { + continue; + }; + let Ok(lba) = u32::try_from(head + k * UNIT) else { + continue; + }; + if !in_segment(segments, lba) { + out.push(lba); + } + } + (units, out) +} + pub(crate) fn read_unit(reader: &mut dyn SectorSource, lba: u32) -> Result>> { let mut buf = vec![0u8; crate::aacs::content::ALIGNED_UNIT_LEN]; match reader.read_sectors(lba, UNIT as u16, &mut buf, false) { diff --git a/src/keys/fmts.rs b/src/keys/fmts.rs index 81eaf5a1..a78d970e 100644 --- a/src/keys/fmts.rs +++ b/src/keys/fmts.rs @@ -110,6 +110,28 @@ fn read_unit( super::evidence::read_unit(reader, lba).ok().flatten() } +/// Whether the forensic segments read in the clear: the first unit of each of up to +/// [`MAX_ANCHOR_ATTEMPTS`] segments is readable and unflagged. `false` when a segment has no +/// sector range or no unit was read. +pub(crate) fn segments_clear( + reader: &mut dyn SectorSource, + layout: &Layout, + format: ContentFormat, + halt: &Halt, +) -> Result { + if layout.unresolved || layout.segments.is_empty() { + return Ok(false); + } + for seg in layout.segments.iter().take(MAX_ANCHOR_ATTEMPTS) { + halt.check()?; + match read_unit(reader, &layout.clip, seg, 0) { + Some(u) if !crate::aacs::content::aacs_unit_encrypted(&u, format) => {} + _ => return Ok(false), + } + } + Ok(true) +} + /// Sends one anchor batch to the sources: `Ok(None)` for an empty answer. pub(crate) type AskFn<'a> = dyn FnMut(&[Vec]) -> Result>> + 'a; @@ -314,8 +336,7 @@ pub(crate) fn forensic_clip_extents( } // Keep only forensic segments addressable within the FORENSIC CLIP's extents; stale/foreign -// records past the clip's end are dropped. Extracted from resolve_fmts_key_map for direct -// testing. +// records past the clip's end are dropped. Split out of `layout` for direct testing. pub(crate) fn filter_addressable_segments( segments: Vec, extents: &[crate::disc::Extent], @@ -323,7 +344,8 @@ pub(crate) fn filter_addressable_segments( segments .into_iter() .filter(|s| { - crate::aacs::segment::clip_byte_to_lba(extents, s.start_spn as u64 * 192).is_some() + crate::aacs::segment::clip_byte_to_lba(extents, s.start_spn as u64 * SPN_BYTES) + .is_some() }) .collect() } @@ -569,7 +591,7 @@ mod probe_tests { mod fmts_helper_tests { use super::*; - // ── resolve_fmts_key_map decision helpers (behaviors flagged by audit) ── + // ── layout decision helpers (behaviors flagged by audit) ── // BEHAVIOR 1 — segment filter: a segment mapping inside the title's extents is kept, // past-clip dropped, all-outside -> empty. @@ -595,7 +617,7 @@ mod fmts_helper_tests { }; let kept = super::filter_addressable_segments(vec![inside, outside], &extents); assert_eq!(kept, vec![inside], "only the in-title segment survives"); - // All-outside → empty; `resolve_fmts_key_map` maps this to Ok(None). + // All-outside → empty; `layout` maps this to Ok(None). assert!( super::filter_addressable_segments(vec![outside], &extents).is_empty(), "no addressable segment → empty (→ resolver Ok(None))" diff --git a/src/keys/mod.rs b/src/keys/mod.rs index f25ba551..e5144ba8 100644 --- a/src/keys/mod.rs +++ b/src/keys/mod.rs @@ -331,10 +331,10 @@ fn sha256(tag: &[u8], bytes: &[u8]) -> [u8; 32] { } impl KeyRing { - /// Acquire the keys `ev.scope()` needs, once, before any output (KU §2.3). The only - /// code in the product that asks a [`crate::KeySource`] for keys; `sources` is called - /// once and every source it builds is dropped before this returns (LK7). `sampler` - /// reads the evidence's pieces from the same raw source; `ctx.halt` stops it. + /// Acquire the keys `ev.scope()` needs, once, before any output (KU §2.3): the only code + /// that asks a [`crate::KeySource`]; every source `sources` builds is dropped before this + /// returns (LK7). `sampler` reads the pieces from the same raw source. Content the + /// format's own detector reads in the clear asks no source, whatever key files it declares. /// /// `Err` refuses the rip before any output: E7022 (a title piece is Missing), E7032 (an /// image or folder piece is Missing), E7013 (a single declared key opens none of the @@ -648,11 +648,9 @@ impl KeyRing { pub(crate) fn keyless_for_disc(disc: &Disc, idx: usize) -> Option { let title = disc.titles.get(idx)?; let mut i = Self::keyless_inner(title, disc.content_format); - i.disc_hash = disc - .aacs - .as_ref() - .map(|a| a.disc_hash.clone()) - .unwrap_or_default(); + // The same identity `is_for` compares: a capture without a hash is named by its + // title-key file. + i.disc_hash = disc.media_id().disc_hash; i.format = disc.format; i.capacity = disc.capacity_sectors; Some(Self(Arc::new(i))) diff --git a/src/keys/resolve.rs b/src/keys/resolve.rs index ee2d081d..dee9668e 100644 --- a/src/keys/resolve.rs +++ b/src/keys/resolve.rs @@ -333,9 +333,10 @@ fn probe( p: &mut Probed, segments: &[(u32, u32)], format: ContentFormat, + detector: Detector, halt: &Halt, ) -> Result<()> { - let (units, samples) = sampler.probe(&p.piece, segments, format, halt)?; + let (units, samples) = sampler.probe(&p.piece, segments, format, detector, halt)?; p.units = units; for s in samples { match s { @@ -420,10 +421,7 @@ pub(crate) fn acquire( if let Some(s) = seed { run.add_keys(&s.0.pool, "seed"); } - let result = match ev.detector { - Detector::Unverified => resolve_hddvd(ev, sampler, &disc_hash, n_decl, &mut run), - Detector::Verified => resolve_bd(ev, sampler, &disc_hash, n_decl, seed, &mut run), - }; + let result = decide(ev, sampler, &disc_hash, n_decl, seed, &mut run); // Every source is dropped here: nothing can ask after `acquire` (LK7). let trace = std::mem::take(&mut run.trace); drop(run.sources); @@ -474,8 +472,95 @@ fn aacs_inner(run: &Run) -> Inner { inner } -// KU §2.6: HD DVD is best effort. Not probed (its encrypted flag is unverified, KS-27): -// a single declared key is applied to every piece; several, or none parseable, refuse. +// The pieces in scope with their probes, and the forensic layout when its clip is in scope. +struct Scoped<'e> { + ps: Vec, + layout: Option<&'e super::fmts::Layout>, + clip: Vec<(u32, u32)>, +} + +// KU §2.3 steps 5–7, every AACS format: the evidence's pieces, the forensic layout when its +// clip is in scope, and each piece probed by the format's own detector. +fn probe_scope<'e>(ev: &'e KeyEvidence, sampler: &mut Sampler, halt: &Halt) -> Result> { + let mut ps: Vec = ev.pieces.iter().cloned().map(Probed::new).collect(); + let layout = ev.fmts.as_ref(); + let clip: Vec<(u32, u32)> = layout + .map(|l| { + l.clip + .iter() + .map(|e| (e.start_lba, e.start_lba.saturating_add(e.sector_count))) + .collect() + }) + .unwrap_or_default(); + let layout = layout.filter(|_| { + ps.iter() + .any(|p| p.ranges().any(|(s, e)| overlaps(&clip, s, e))) + }); + let segments: Vec<(u32, u32)> = layout + .map(|l| sorted_ranges(l.ranges.iter().map(|&(s, e, _)| (s, e)).collect())) + .unwrap_or_default(); + for p in &mut ps { + probe(sampler, p, &segments, ev.container, ev.detector, halt)?; + } + Ok(Scoped { ps, layout, clip }) +} + +// The one encryption decision, every AACS format and source (CSS makes it on its scramble +// flag): content read in the clear asks no source, whatever key files the disc declares; +// encrypted content is keyed by the format's rules, refused when no key is found. +fn decide( + ev: &KeyEvidence, + sampler: &mut Sampler, + disc_hash: &str, + n_decl: Option, + seed: Option<&KeyRing>, + run: &mut Run, +) -> Result { + let scoped = probe_scope(ev, sampler, run.halt)?; + if in_clear(&scoped, sampler, ev.container, run.halt)? { + tracing::info!(target: "freemkv::keys", pieces = scoped.ps.len(), "content in the clear: no key source asked"); + let ps: Vec = scoped + .ps + .into_iter() + .map(|p| Probed { + verdict: Verdict::Clear, + ..p + }) + .collect(); + let mut inner = aacs_inner(run); + inner.no_stream_files = ev.no_stream_files; + build(&mut inner, &ps, None, None, run, Vec::new()); + return Ok(inner); + } + match ev.detector { + Detector::Unverified => resolve_hddvd(ev, sampler, disc_hash, n_decl, run), + Detector::Verified => resolve_bd(ev, scoped, sampler, disc_hash, n_decl, seed, run), + } +} + +// Whether every piece's probes read clear (a piece with no whole unit holds no AACS unit) +// and so do the forensic segments in scope. +fn in_clear( + scoped: &Scoped, + sampler: &mut Sampler, + format: ContentFormat, + halt: &Halt, +) -> Result { + let pieces_clear = scoped + .ps + .iter() + .all(|p| p.units == 0 || (p.enc.is_empty() && p.faults == 0)); + if !pieces_clear { + return Ok(false); + } + match scoped.layout { + None => Ok(true), + Some(l) => super::fmts::segments_clear(sampler.source(), l, format, halt), + } +} + +// KU §2.6: HD DVD with encrypted content is best effort (its encrypted flag is unverified, +// KS-27): a single declared key is applied to every piece; several, or none parseable, refuse. fn resolve_hddvd( ev: &KeyEvidence, sampler: &mut Sampler, @@ -514,13 +599,16 @@ fn resolve_hddvd( ranges.extend(p.ranges().map(|(s, e)| (s, e, 0usize))); inner.spans.extend(p.spans.iter().copied()); } + // `span_at` bisects: a fragmented file or interleaved pieces arrive out of LBA order. + inner.spans.sort_unstable_by_key(|s| s.0); inner.map = Arc::new(AacsKeyMap::from_ranges(ranges)); Ok(inner) } -// KU §2.3 steps 5–14 for BD/UHD content. +// KU §2.3 steps 8–14 for BD/UHD content not in the clear, over `probe_scope`'s probes. fn resolve_bd( ev: &KeyEvidence, + scoped: Scoped, sampler: &mut Sampler, disc_hash: &str, n_decl: Option, @@ -529,29 +617,11 @@ fn resolve_bd( ) -> Result { let format = ev.container; let scope = &ev.scope; - // Step 5: the evidence's pieces. - let mut ps: Vec = ev.pieces.iter().cloned().map(Probed::new).collect(); - // Step 6: FMTS, when the forensic clip is in scope. - let layout = ev.fmts.as_ref(); - let clip: Vec<(u32, u32)> = layout - .map(|l| { - l.clip - .iter() - .map(|e| (e.start_lba, e.start_lba.saturating_add(e.sector_count))) - .collect() - }) - .unwrap_or_default(); - let layout = layout.filter(|_| { - ps.iter() - .any(|p| p.ranges().any(|(s, e)| overlaps(&clip, s, e))) - }); - let segments: Vec<(u32, u32)> = layout - .map(|l| sorted_ranges(l.ranges.iter().map(|&(s, e, _)| (s, e)).collect())) - .unwrap_or_default(); - // Step 7: probes. - for p in &mut ps { - probe(sampler, p, &segments, format, run.halt)?; - } + let Scoped { + mut ps, + layout, + clip, + } = scoped; // Step 8: sample-independent sources, once, with the main title's samples. let needs_keys = ps .iter() diff --git a/src/keys/tests.rs b/src/keys/tests.rs index e55cebf5..8f0e8951 100644 --- a/src/keys/tests.rs +++ b/src/keys/tests.rs @@ -978,6 +978,88 @@ fn hddvd_single_key_best_effort_multi_key_refuses() { } } +/// An HD DVD copy already in the clear (whole MPEG-PS packs, no scrambling flag) that kept its +/// AACS directory needs no key, however many keys its title-key file declares: the set is +/// clear, no source is asked, nothing is best effort. +#[test] +fn hddvd_in_the_clear_needs_no_key_whatever_it_declares() { + let mut clip = vec![0u8; 30 * 2048]; + for (i, sector) in clip.chunks_mut(2048).enumerate() { + for (j, b) in sector.iter_mut().enumerate() { + *b = ((i * 31 + j) % 251) as u8 | 1; + } + sector[..4].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]); + sector[20] &= !0x30; + } + let files = [ + BdFile::new("HVDVD_TS/FEATURE.EVO", 30, None).with_clip(clip), + BdFile::new("BDMV/index.bdmv", 1, None), + ]; + for declared in [1, 64] { + let uk_ro = unit_key_ro(AacsVersion::V10, &vec![[0xEE; 16]; declared], &[1]); + let img = encrypted_bd_image(&files, &uk_ro); + let disc = disc_over(&img, &uk_ro, &[&[0]], DiscFormat::HdDvd); + let fx = Fx { img, disc }; + let calls = Calls::default(); + let set = resolve(&fx, KeyScope::Titles(vec![0]), &[Spec::keydb(&[], &calls)]) + .unwrap_or_else(|e| panic!("{declared} declared: a clear HD DVD refused: {e}")); + let s = set.status(); + assert_eq!( + (s.keyed, s.clear, s.best_effort), + (0, 1, false), + "{declared} declared" + ); + assert_eq!(calls.len(), 0, "no source is asked for a clear disc"); + assert!(set.is_aacs(), "the set still answers for the AACS disc"); + } +} + +/// An HD DVD set's unit spans stay in LBA order across fragmented and interleaved pieces: +/// `span_at` bisects them. +#[test] +fn hddvd_set_spans_are_sorted_across_pieces() { + use crate::keys::evidence::Piece; + let files = [ + BdFile::new("HVDVD_TS/FEATURE.EVO", 30, Some(K1)), + BdFile::new("BDMV/index.bdmv", 1, None), + ]; + let uk_ro = unit_key_ro(AacsVersion::V10, &[[0xEE; 16]], &[1]); + let img = encrypted_bd_image(&files, &uk_ro); + let disc = disc_over(&img, &uk_ro, &[&[0]], DiscFormat::HdDvd); + let fx = Fx { img, disc }; + let calls = Calls::default(); + let f = factory(&[Spec::keydb(&[K1], &calls)]); + let ctx = crate::ctx::Ctx::new(Halt::new()); + let mut reader = fx.source(); + let mut ev = + KeyEvidence::from_disc(&fx.disc, &mut reader, KeyScope::Titles(vec![0]), &ctx).unwrap(); + // Piece A is fragmented (descending extents); piece B sits between A's extents. + ev.pieces = vec![ + Piece { + spans: vec![(5000, 30, 5000), (1000, 30, 5030)], + titles: vec![0], + rank: 1, + }, + Piece { + spans: vec![(3000, 30, 3000)], + titles: vec![0], + rank: 1, + }, + ]; + let set = super::resolve::acquire( + &ev, + &mut Sampler::new(&mut reader), + &f, + AcquireOptions::default(), + &ctx, + &FakeClock::default(), + ) + .unwrap() + .keys; + let starts: Vec = set.0.spans.iter().map(|s| s.0).collect(); + assert_eq!(starts, [1000, 3000, 5000]); +} + /// LK17. A set is bound to its disc (hash, capacity, format, VID fingerprint), and its /// `Debug` shows no key or VID bytes. #[test] @@ -2360,6 +2442,17 @@ fn keyless_session_set_is_bound_to_its_disc() { assert!(!bare.is_for(&fx.disc.media_id()), "no identity, no bypass"); } +/// A capture whose AACS `disc_hash` is empty is identified by its title-key file: the keyless +/// set still fits that disc. +#[test] +fn keyless_session_set_fits_a_disc_without_a_captured_hash() { + let mut fx = two_units(); + fx.disc.aacs.as_mut().unwrap().disc_hash = String::new(); + assert!(!fx.disc.aacs.as_ref().unwrap().uk_ro.is_empty()); + let set = KeyRing::keyless_for_disc(&fx.disc, 0).expect("title 0"); + assert!(set.is_for(&fx.disc.media_id())); +} + /// J22 with J21 parity (review r4). KS-14 [BD] §3.9.3: "Num_of_CPS_Unit field (16 bits) /// indicates the number of CPS Units on the disc". Step 10 keys on no less evidence than step /// 9: one opened probe of a piece longer than one unit is not proof, so the piece stays Lazy @@ -3067,3 +3160,209 @@ fn evidence_without_a_hash_is_looked_up_by_its_title_key_file() { ); assert!(ring.is_for(&fx.disc.media_id())); } + +// ── One encryption decision, every AACS format ────────────────────────────── + +#[derive(Clone, Copy, Debug)] +enum Media { + Bd, + Uhd, + HdDvd, +} + +const MEDIA: [Media; 3] = [Media::Bd, Media::Uhd, Media::HdDvd]; + +// Units in the one stream file of a `media_fx` disc. +const MEDIA_UNITS: u32 = 10; + +// Whole MPEG-PS packs (pack start in every sector, no scrambling flag): an HD DVD feature in +// the clear. +fn ps_clip(sectors: usize) -> Vec { + let mut clip = vec![0u8; sectors * 2048]; + for (i, sector) in clip.chunks_mut(2048).enumerate() { + for (j, b) in sector.iter_mut().enumerate() { + *b = ((i * 31 + j) % 251) as u8 | 1; + } + sector[..4].copy_from_slice(&[0x00, 0x00, 0x01, 0xBA]); + sector[20] &= !0x30; + } + clip +} + +// Encrypt every unit of file `f` (MPEG-PS) under `key`, its plain copy kept. A plaintext +// header byte is varied until the ciphertext carries the PES scrambling flag the reader +// gates on (that flag lies past the unit's clear seed). +fn encrypt_ps(img: &mut EncryptedBdImage, f: usize, key: &[u8; 16]) { + let (start, sectors) = img.files[f]; + for u in 0..sectors / 3 { + let at = (start + u * 3) as usize * 2048; + let mut plain = img.plain[at..at + ALIGNED_UNIT_LEN].to_vec(); + let ct = (0..=255u8) + .find_map(|t| { + plain[24] = t; + let mut ct = plain.clone(); + assert!(encrypt_unit(&mut ct, key)); + (ct[20] & 0x30 != 0).then_some(ct) + }) + .expect("a flagged ciphertext"); + img.plain[at..at + ALIGNED_UNIT_LEN].copy_from_slice(&plain); + img.image[at..at + ALIGNED_UNIT_LEN].copy_from_slice(&ct); + } +} + +// A one-title `media` disc whose one stream file holds `MEDIA_UNITS` units, in the clear or +// encrypted under `key`, with a title-key file declaring `declared` keys. +fn media_fx(media: Media, key: Option<[u8; 16]>, declared: usize) -> Fx { + let version = match media { + Media::Uhd => AacsVersion::V20, + _ => AacsVersion::V10, + }; + let uk_ro = unit_key_ro(version, &vec![[0xEE; 16]; declared], &[1]); + let (img, format) = match media { + Media::Bd | Media::Uhd => { + let img = encrypted_bd_image(&[stream(0, MEDIA_UNITS, key)], &uk_ro); + let format = match media { + Media::Uhd => DiscFormat::Uhd, + _ => DiscFormat::BluRay, + }; + (img, format) + } + Media::HdDvd => { + let files = [ + BdFile::new("HVDVD_TS/FEATURE.EVO", MEDIA_UNITS * 3, None) + .with_clip(ps_clip(MEDIA_UNITS as usize * 3)), + BdFile::new("BDMV/index.bdmv", 1, None), + ]; + let mut img = encrypted_bd_image(&files, &uk_ro); + if let Some(k) = &key { + encrypt_ps(&mut img, 0, k); + } + (img, DiscFormat::HdDvd) + } + }; + let mut disc = disc_over(&img, &uk_ro, &[&[0]], format); + if let (Media::Uhd, Some(a)) = (media, disc.aacs.as_mut()) { + a.version = 2; + } + Fx { img, disc } +} + +/// Every AACS format (BD, UHD, HD DVD) in the clear proceeds with no key source asked, +/// whatever its title-key file declares, for a title rip and a whole-disc copy alike, and +/// reads back as it is. +#[test] +fn every_format_in_the_clear_asks_no_source_whatever_it_declares() { + for media in MEDIA { + for declared in [1, 3] { + for scope in [KeyScope::Titles(vec![0]), KeyScope::WholeDisc] { + let fx = media_fx(media, None, declared); + let calls = Calls::default(); + let specs = [Spec::keydb(&[K1], &calls), Spec::online(&[K1], &calls)]; + let set = resolve(&fx, scope.clone(), &specs).unwrap_or_else(|e| { + panic!("{media:?} {declared} declared {scope:?}: clear content refused: {e}") + }); + assert_eq!(calls.len(), 0, "{media:?} {scope:?}: a source was asked"); + let s = set.status(); + assert_eq!( + (s.keyed, s.lazy, s.best_effort), + (0, 0, false), + "{media:?} {scope:?}" + ); + assert!(s.clear >= 1, "{media:?} {scope:?}"); + let mut r = set.title_reader(&fx.disc, 0, fx.source()).unwrap(); + assert_eq!( + read(&mut r, &fx, 0, 0, MEDIA_UNITS).unwrap(), + fx.plain(fx.file(0).0, MEDIA_UNITS), + "{media:?} {scope:?}" + ); + } + } + } +} + +/// Every AACS format with encrypted content asks the configured sources and, when none has +/// the key, refuses before any output: E7022 for a title rip, E7032 for a whole-disc copy. +#[test] +fn every_format_encrypted_with_no_key_refuses_typed() { + for media in MEDIA { + for (scope, want) in [ + (KeyScope::Titles(vec![0]), E7022), + (KeyScope::WholeDisc, E7032), + ] { + let fx = media_fx(media, Some(K1), 1); + let calls = Calls::default(); + let r = resolve(&fx, scope.clone(), &[Spec::keydb(&[], &calls)]); + assert_eq!(code(r), want, "{media:?} {scope:?}"); + assert_eq!(calls.len(), 1, "{media:?} {scope:?}: the source is asked"); + } + } +} + +/// Every AACS format with encrypted content and a source holding its key decrypts it. +#[test] +fn every_format_encrypted_with_a_key_decrypts() { + for media in MEDIA { + let fx = media_fx(media, Some(K1), 1); + assert_ne!( + fx.raw(fx.file(0).0, 1), + fx.img.plain[fx.file(0).0 as usize * 2048..][..ALIGNED_UNIT_LEN], + "{media:?}: the fixture is encrypted" + ); + let calls = Calls::default(); + let set = resolve( + &fx, + KeyScope::Titles(vec![0]), + &[Spec::keydb(&[K1], &calls)], + ) + .unwrap_or_else(|e| panic!("{media:?}: refused with its key held: {e}")); + assert_eq!(calls.len(), 1, "{media:?}"); + assert_eq!(set.status().keyed, 1, "{media:?}"); + let mut r = set.title_reader(&fx.disc, 0, fx.source()).unwrap(); + assert_eq!( + read(&mut r, &fx, 0, 0, MEDIA_UNITS).unwrap(), + fx.plain(fx.file(0).0, MEDIA_UNITS), + "{media:?}: decrypted" + ); + } +} + +/// An FMTS disc takes the same decision: base content and forensic segments in the clear ask +/// no source, forensic keys included; encrypted segments under clear base content are keyed. +#[test] +fn fmts_in_the_clear_asks_no_source_encrypted_segments_are_keyed() { + let mut fx = fmts_fixture(); + for f in [0, 1] { + let (start, n) = fx.file(f); + let span = start as usize * 2048..(start + n) as usize * 2048; + let clear = masked(&fx.img.plain[span.clone()]); + fx.img.image[span.clone()].copy_from_slice(&clear); + fx.img.plain[span].copy_from_slice(&clear); + } + let calls = Calls::default(); + let set = resolve( + &fx, + KeyScope::Titles(vec![0, 1, 2]), + &[Spec::keydb(&[K1, K2], &calls), fmts_online(&calls)], + ) + .unwrap(); + assert_eq!(calls.len(), 0, "no source, forensic included"); + assert_eq!(set.status().forensic, ForensicState::None); + + for (a, b, key) in [(0, 16, F1), (20, 36, F2)] { + for u in a..b { + fx.reencrypt(1, u, if (u - a) % 2 == 0 { &key } else { &ALT }); + } + } + let calls = Calls::default(); + let set = resolve( + &fx, + KeyScope::Titles(vec![0, 1, 2]), + &[Spec::keydb(&[], &calls), fmts_online(&calls)], + ) + .unwrap(); + assert!( + calls.of("online").iter().any(|c| c.forensic), + "encrypted segments ask for the forensic set" + ); + assert_eq!(set.status().forensic, ForensicState::Resolved); +} diff --git a/src/keys/tests/parity.rs b/src/keys/tests/parity.rs index fa3c8d48..a70a75be 100644 --- a/src/keys/tests/parity.rs +++ b/src/keys/tests/parity.rs @@ -425,10 +425,8 @@ fn parity_image_dvd_css_multi_vts() { } // BUG-1 (pipeline design §1.4): a live-scanned DVD has no `disc.css`, so the folder must -// still come out descrambled. Red today (the VOBs are written scrambled); ignored until the -// decrypt stage detects CSS by content (slice 3b). +// still come out descrambled, each VTS under its own cracked key. #[test] -#[ignore = "BUG-1: live DVD -> dir never engages CSS (fixed in slice 3b)"] fn parity_bug1_live_dvd_folder_is_descrambled() { let (fx, clear, _) = dvd_fx(true); let dest = tempfile::tempdir().unwrap(); diff --git a/src/keysource.rs b/src/keysource.rs index 7811c32f..1b85b139 100644 --- a/src/keysource.rs +++ b/src/keysource.rs @@ -196,7 +196,10 @@ impl<'a> DiscInputsCtx<'a> { } else { parse_title_keys(&inputs.unit_key_ro, AacsVersion::from_major(inputs.version)) .map(|f| f.encrypted_keys.into_iter().map(|(_, k)| k).collect()) - .unwrap_or_default() + .unwrap_or_else(|| { + tracing::warn!("unit key file unparseable; no encrypted title keys"); + Vec::new() + }) }; Self { inner: inputs, @@ -317,7 +320,7 @@ pub trait KeySource { /// `i + 1`). The source hands back the COMPLETE set it holds; the caller /// trusts any non-empty result as all of them and never assumes a fixed count. /// Defaults to empty: a source with no forensic material (a plain keydb, the - /// mapfile) opts out, and only an FMTS disc's mux ever calls this. + /// mapfile) opts out, and only an FMTS disc's key acquisition ever calls this. fn get_fmts_indexes(&self, _ctx: &dyn ResolveCtx) -> Result, Error> { Ok(Vec::new()) } @@ -428,6 +431,8 @@ pub(crate) fn encrypted_units_in( Ok(_) => {} // A Stop ends sampling: every remaining probe would only re-hit the drive. Err(crate::error::Error::Halted) => return out, + // A dead bus or gone source is not a bad probe: stop rather than skip on. + Err(e) if e.is_scsi_transport_failure() || e.is_source_terminated() => return out, Err(_) => continue, } for i in 0..units_this as usize { @@ -510,6 +515,22 @@ mod tests { assert!(s.host_certs(Some(68)).is_empty()); } + /// The trait defaults are the contract external sources rely on (retry, per-piece asking, + /// VID use); a flipped default changes every foreign source's behaviour. + #[test] + fn key_source_defaults_are_conservative() { + struct MinimalSource; + impl KeySource for MinimalSource { + fn get_unit_keys(&self, _ctx: &dyn ResolveCtx) -> Result, Error> { + Ok(Vec::new()) + } + } + let s = MinimalSource; + assert!(!s.last_failure_was_transport()); + assert!(s.answer_depends_on_samples()); + assert!(!s.uses_vid()); + } + /// DiscInputsCtx maps DiscInputs faithfully: zero VID → None, non-zero VID → /// Some; title from volume_label; samples truncate to n; enc_title_keys /// parses Unit_Key_RO.inf at the version stride. @@ -1102,4 +1123,64 @@ mod tests { assert_eq!(lbas.len(), n, "no unit may be sampled twice"); assert!(n as u32 <= units); } + + fn sampling_extent(sectors: u32) -> Vec { + vec![crate::disc::Extent { + start_lba: 500, + sector_count: sectors, + }] + } + + /// A gone source ends sampling at the first failed probe: later probes would only hit the + /// dead device, and the empty result must not be mistaken for a clear title by more reads. + #[test] + fn encrypted_units_in_stops_at_a_gone_source() { + use crate::aacs::content::ALIGNED_UNIT_SECTORS; + struct Gone(u32); + impl crate::sector::SectorSource for Gone { + fn read_sectors( + &mut self, + _l: u32, + _c: u16, + _b: &mut [u8], + _r: bool, + ) -> crate::error::Result { + self.0 += 1; + Err(crate::error::Error::SourceTerminated) + } + } + let mut src = Gone(0); + let ext = sampling_extent(400 * ALIGNED_UNIT_SECTORS); + let out = encrypted_units_in(&mut src, &ext, crate::disc::ContentFormat::BdTs, 8); + assert!(out.is_empty()); + assert_eq!(src.0, 1, "no probe after the source is gone"); + } + + /// An HD DVD title is classified by the PS scrambling bits, not the BD CPI byte. + #[test] + fn encrypted_units_in_uses_the_title_format() { + use crate::aacs::content::{ALIGNED_UNIT_LEN, ALIGNED_UNIT_SECTORS}; + struct Fill; + impl crate::sector::SectorSource for Fill { + fn read_sectors( + &mut self, + _l: u32, + c: u16, + b: &mut [u8], + _r: bool, + ) -> crate::error::Result { + let n = c as usize * 2048; + for unit in b[..n].chunks_mut(ALIGNED_UNIT_LEN) { + unit.fill(0); + unit[0] = 0xC0; // BD CPI set, PS bits clear + } + Ok(n) + } + } + let ext = sampling_extent(400 * ALIGNED_UNIT_SECTORS); + let ps = encrypted_units_in(&mut Fill, &ext, crate::disc::ContentFormat::MpegPs, 8); + assert!(ps.is_empty(), "a BD CPI byte is not a PS scramble flag"); + let ts = encrypted_units_in(&mut Fill, &ext, crate::disc::ContentFormat::BdTs, 8); + assert_eq!(ts.len(), 8); + } } diff --git a/src/labels/bdj_feature.rs b/src/labels/bdj_feature.rs index 482cbd34..65bc16a5 100644 --- a/src/labels/bdj_feature.rs +++ b/src/labels/bdj_feature.rs @@ -551,6 +551,11 @@ mod tests { // A playlist: one play item of `secs` seconds with `n_audio` audio streams. fn build_mpls(secs: u32, n_audio: u8) -> Vec { + build_mpls_pg(secs, n_audio, 0) + } + + // As `build_mpls`, plus `n_pg` PG subtitle streams after the audio. + fn build_mpls_pg(secs: u32, n_audio: u8, n_pg: u8) -> Vec { let playlist_start: u32 = 40; let mut buf = Vec::new(); buf.extend_from_slice(b"MPLS0200"); @@ -582,11 +587,17 @@ mod tests { item.extend_from_slice(&[0u8; 2]); // reserved item.push(0); // n_video item.push(n_audio); // n_audio - item.extend_from_slice(&[0u8; 6]); // pg/ig/sec.../dv counts + item.push(n_pg); // n_pg + item.extend_from_slice(&[0u8; 5]); // ig/sec.../dv counts item.extend_from_slice(&[0u8; 4]); // reserved for i in 0..n_audio { item.extend_from_slice(&audio_stream_entry(0x1100 + i as u16)); } + for i in 0..n_pg { + item.extend_from_slice(&[3, 0x01]); + item.extend_from_slice(&(0x1200 + i as u16).to_be_bytes()); + item.extend_from_slice(&[4, 0x90, b'e', b'n', b'g']); + } let stn_len = (item.len() - stn_start - 2) as u16; item[stn_start..stn_start + 2].copy_from_slice(&stn_len.to_be_bytes()); @@ -949,6 +960,83 @@ mod tests { assert_eq!(resolve(&mut disc, &udf, &[]), None); } + // Only AUTOSTART apps' jars are Tier-2 targets: a PRESENT-only decoy app's jar, + // naming a longer playlist, must not join the candidate set. + #[test] + fn tier2_ignores_non_autostart_apps() { + let auto_jar = build_jar(&[("com/studio/MainXlet.class", build_class(&["00800.mpls"]))]); + let decoy_jar = build_jar(&[("com/studio/Decoy.class", build_class(&["00801.mpls"]))]); + let (mut disc, udf) = build_disc( + vec![ + ("00800.mpls", build_mpls(7000, 6)), + ("00801.mpls", build_mpls(9000, 6)), + ], + vec![("00000.jar", auto_jar), ("00009.jar", decoy_jar)], + vec![("00000.bdjo", bdjo_with_apps(&[(1, "00000"), (2, "00009")]))], + ); + let hint = resolve(&mut disc, &udf, &[]).expect("Tier-2 hint"); + assert_eq!(hint.playlist_id, Some(800)); + } + + // A manifest over MAX_MANIFEST_BYTES is skipped, so Tier 2 decides. + #[test] + fn oversized_manifest_is_not_a_tier1_hint() { + let mut props = b"feature.playlist.id=00800\n".to_vec(); + props.resize(MAX_MANIFEST_BYTES + 16, b' '); + let jar = build_jar(&[ + ("app.properties", props), + ("com/studio/MainXlet.class", build_class(&["00801.mpls"])), + ]); + let (mut disc, udf) = build_disc( + vec![ + ("00800.mpls", build_mpls(7000, 6)), + ("00801.mpls", build_mpls(9000, 6)), + ], + vec![("00000.jar", jar)], + vec![], + ); + let hint = resolve(&mut disc, &udf, &[]).expect("Tier-2 hint"); + assert_eq!(hint.playlist_id, Some(801)); + } + + #[test] + fn props_hint_ignores_comment_lines() { + for text in [ + "# feature.playlist.id=00800\n", + "! feature.playlist.id=00800\n", + ] { + assert_eq!(props_hint(text, &|_| true), None, "{text}"); + } + } + + // Only primary audio counts toward the multi-audio gate, not PG subtitles. + #[test] + fn mpls_stats_counts_only_audio_streams() { + let (mut disc, udf) = build_disc( + vec![("00800.mpls", build_mpls_pg(7000, 1, 3))], + vec![], + vec![], + ); + assert_eq!(mpls_stats(&mut disc, &udf, 800), Some((7000, 1))); + } + + // Only `NNNNN.mpls` names (five digits) are playlists. + #[test] + fn real_playlist_ids_requires_five_digit_stems() { + let m = || build_mpls(60, 1); + let (_disc, udf) = build_disc( + vec![ + ("800.mpls", m()), + ("0800a.mpls", m()), + ("+0800.mpls", m()), + ("00801.mpls", m()), + ], + vec![], + vec![], + ); + assert_eq!(real_playlist_ids(&udf), HashSet::from([801])); + } + // A sector source that records every LBA it is asked for. struct Counting<'a>(&'a mut MemDisc, Vec); impl SectorSource for Counting<'_> { @@ -1095,6 +1183,11 @@ mod tests { // Build a minimal single-app BDJO whose AUTOSTART app has the given base_dir. fn bdjo_with_autostart(base_dir: &str) -> Vec { + bdjo_with_apps(&[(1, base_dir)]) + } + + // Build a BDJO with one app per `(control_code, base_dir)`; control code 1 is AUTOSTART. + fn bdjo_with_apps(apps: &[(u8, &str)]) -> Vec { fn app_string(buf: &mut Vec, s: &str) { buf.push(s.len() as u8); buf.extend_from_slice(s.as_bytes()); @@ -1115,26 +1208,28 @@ mod tests { b.extend_from_slice(&[0u8; 4]); // AccessiblePlaylists length b.extend_from_slice(&[0u8; 4]); // num_pl(11)+flags(2)+pad(19) = 0 b.extend_from_slice(&[0u8; 4]); // AMT length - b.push(1); // num_app + b.push(apps.len() as u8); // num_app b.push(0); // padding - // App record - b.push(1); // control_code = AUTOSTART - b.push(0); // type(4)+reserved(4) - b.extend_from_slice(&[0u8; 4]); // org_id - b.extend_from_slice(&[0u8; 2]); // app_id - b.extend_from_slice(&[0u8; 10]); // descriptor tag+length - b.push(0); // num_profile(4)+pad - b.push(0); - b.push(0); // priority - b.push(0); // binding/visibility/reserved - b.extend_from_slice(&[0u8; 2]); // app_name data_length = 0 - app_string(&mut b, ""); // icon_locator - b.extend_from_slice(&[0u8; 2]); // icon_flags - app_string(&mut b, base_dir); // base_dir - app_string(&mut b, ""); // classpath_extension - app_string(&mut b, "com.studio.MainXlet"); // initial_class - b.push(0); // params data_length = 0 - b.push(0); // word-align pad + for &(control_code, base_dir) in apps { + // App record + b.push(control_code); + b.push(0); // type(4)+reserved(4) + b.extend_from_slice(&[0u8; 4]); // org_id + b.extend_from_slice(&[0u8; 2]); // app_id + b.extend_from_slice(&[0u8; 10]); // descriptor tag+length + b.push(0); // num_profile(4)+pad + b.push(0); + b.push(0); // priority + b.push(0); // binding/visibility/reserved + b.extend_from_slice(&[0u8; 2]); // app_name data_length = 0 + app_string(&mut b, ""); // icon_locator + b.extend_from_slice(&[0u8; 2]); // icon_flags + app_string(&mut b, base_dir); // base_dir + app_string(&mut b, ""); // classpath_extension + app_string(&mut b, "com.studio.MainXlet"); // initial_class + b.push(0); // params data_length = 0 + b.push(0); // word-align pad + } b } } diff --git a/src/labels/bdmt.rs b/src/labels/bdmt.rs index 1e72b69d..5141edff 100644 --- a/src/labels/bdmt.rs +++ b/src/labels/bdmt.rs @@ -43,8 +43,8 @@ pub struct DiscMetadata { pub titles: BTreeMap, /// First-line / short description, per lang pub descriptions: BTreeMap, - /// Disc N of M, returned whenever both `` and - /// `` are present and sane (including `(1, 1)` for a + /// Disc N of M, returned whenever both `` (or the + /// `` fallback) and `` are present and sane (including `(1, 1)` for a /// single-disc release); `None` if either is missing or invalid. pub disc_number: Option<(u32, u32)>, } @@ -147,11 +147,8 @@ pub(crate) type BdmtFields = (String, Option, Option<(u32, u32)>); // could be located. pub(crate) fn parse_bdmt_xml(xml_text: &str) -> Option { let title = cap_text(extract_title(xml_text)?); - // xml::text already returns a trimmed string (see xml::text), so the - // description is only filtered for emptiness and XML-fragment noise. let description = xml::text(xml_text, "description") - .filter(|s| !s.is_empty()) - .filter(|s| !looks_like_xml(s)) + .and_then(|s| field_text(&s)) .map(cap_text); let disc_set = extract_disc_set(xml_text); Some((title, description, disc_set)) @@ -174,6 +171,66 @@ fn looks_like_xml(s: &str) -> bool { }) } +/// Turn a raw element body into display text: a whole-value CDATA section is +/// unwrapped, otherwise the predefined and numeric character entities are decoded. +/// `None` for empty values and for values carrying element markup. +fn field_text(raw: &str) -> Option { + let raw = raw.trim(); + let out = match raw + .strip_prefix("")) + { + Some(inner) if !inner.contains("]]>") => inner.trim().to_string(), + Some(_) => return None, + None if looks_like_xml(raw) => return None, + None => decode_entities(raw), + }; + if out.is_empty() || looks_like_xml(&out) { + return None; + } + Some(out) +} + +// Decode `& < > " ' &#N; &#xH;`; anything else is kept verbatim. +fn decode_entities(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut rest = s; + while let Some(i) = rest.find('&') { + out.push_str(&rest[..i]); + rest = &rest[i..]; + let decoded = rest.find(';').filter(|&e| e <= 10).and_then(|e| { + let c = match &rest[1..e] { + "amp" => '&', + "lt" => '<', + "gt" => '>', + "quot" => '"', + "apos" => '\'', + n => { + let n = n.strip_prefix('#')?; + let cp = match n.strip_prefix(['x', 'X']) { + Some(h) => u32::from_str_radix(h, 16).ok()?, + None => n.parse().ok()?, + }; + char::from_u32(cp).filter(|c| !c.is_control())? + } + }; + Some((c, e + 1)) + }); + match decoded { + Some((c, len)) => { + out.push(c); + rest = &rest[len..]; + } + None => { + out.push('&'); + rest = &rest[1..]; + } + } + } + out.push_str(rest); + out +} + /// Try title-bearing element variants in priority order. The `xml` /// helpers are case- and namespace-insensitive, so callers pass the /// bare local name (no `di:` prefix). @@ -182,10 +239,7 @@ fn extract_title(xml_text: &str) -> Option { // tableOfContents/titleName. xml::text trims, so an empty string here // means a genuinely empty element. for tag in ["name", "title"] { - if let Some(s) = xml::text(xml_text, tag) - && !s.is_empty() - && !looks_like_xml(&s) - { + if let Some(s) = xml::text(xml_text, tag).and_then(|s| field_text(&s)) { return Some(s); } } @@ -193,9 +247,7 @@ fn extract_title(xml_text: &str) -> Option { // don't accidentally pick a stray from elsewhere. if let Some((s, e)) = xml::find_element(xml_text, "tableOfContents", 0) { let block = &xml_text[s..e]; - if let Some(t) = xml::text(block, "titleName") - && !t.is_empty() - { + if let Some(t) = xml::text(block, "titleName").and_then(|s| field_text(&s)) { return Some(t); } } @@ -256,6 +308,35 @@ mod tests { assert!(d.expect("desc").len() <= MAX_BDMT_TEXT); } + #[test] + fn cap_text_cuts_on_a_char_boundary() { + let s = format!("aa{}", "\u{4e16}".repeat(400)); + let t = cap_text(s); + assert!(t.len() <= MAX_BDMT_TEXT && t.chars().all(|c| c == 'a' || c == '\u{4e16}')); + } + + #[test] + fn entities_and_cdata_are_decoded() { + let xml = "Tom & Jerry 's A\ + "; + let (t, d, _) = parse_bdmt_xml(xml).expect("parse"); + assert_eq!(t, "Tom & Jerry 's A"); + assert_eq!(d.as_deref(), Some("a < b & c")); + } + + #[test] + fn cdata_title_is_unwrapped() { + let xml = ""; + assert_eq!(parse_bdmt_xml(xml).expect("parse").0, "Real"); + } + + #[test] + fn toc_title_name_with_markup_is_rejected() { + let xml = "Real \ + "; + assert_eq!(parse_bdmt_xml(xml), None); + } + #[test] fn extract_simple_title() { // Minimal Paramount-style document: as the title diff --git a/src/labels/class_reader.rs b/src/labels/class_reader.rs index f7c8ae06..f358c810 100644 --- a/src/labels/class_reader.rs +++ b/src/labels/class_reader.rs @@ -1145,6 +1145,24 @@ mod tests { ); } + #[test] + fn modified_utf8_unpaired_surrogates_degrade_without_panicking() { + // high + high: each is lone. + assert_eq!( + decode_modified_utf8(&[0xED, 0xA0, 0xBD, 0xED, 0xA0, 0xBD]).unwrap(), + "\u{FFFD}\u{FFFD}" + ); + // high + ASCII, and a lone low surrogate. + assert_eq!( + decode_modified_utf8(&[0xED, 0xA0, 0xBD, b'a']).unwrap(), + "\u{FFFD}a" + ); + assert_eq!( + decode_modified_utf8(&[0xED, 0xB8, 0x80]).unwrap(), + "\u{FFFD}" + ); + } + #[test] fn modified_utf8_two_byte() { // U+00E9 'é' in the standard 2-byte modified-UTF-8 encoding diff --git a/src/labels/clpi_audit.rs b/src/labels/clpi_audit.rs index 016b1491..219fd206 100644 --- a/src/labels/clpi_audit.rs +++ b/src/labels/clpi_audit.rs @@ -660,4 +660,73 @@ mod tests { ); assert_eq!(a.class_counts(), (1, 0, 1, 0), "{:?}", a.rows); } + + // Audit one clip `00001` (CLPI `clpi`) against one playlist (`mpls`). + fn audit_one(clpi: Vec, mpls: Vec) -> ClpiVsMplsAudit { + use crate::udf::fixture::*; + let dir = |name: &str, icb: u32, files| DirSpec { + name: name.to_string(), + icb_lba: icb, + dir_data_lba: icb + 1, + files, + subdirs: vec![], + }; + let clipinf = dir( + "CLIPINF", + 24, + vec![file_with("00001.clpi", 26, 8000, clpi, false)], + ); + let playlist = dir( + "PLAYLIST", + 30, + vec![file_with("00800.mpls", 32, 8200, mpls, false)], + ); + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "BDMV".to_string(), + icb_lba: 20, + dir_data_lba: 21, + files: Vec::new(), + subdirs: vec![clipinf, playlist], + }], + }; + let mut disc = MemDisc::new(); + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = crate::udf::read_filesystem(&mut disc).expect("fs"); + audit(&mut disc, &udf) + } + + // A PID the playlist names but the CLPI lacks is a real MplsOnly row. + #[test] + fn playlist_pid_missing_from_clpi_is_mpls_only() { + use crate::consts::coding_type as c; + let build_clpi = super::super::clpi_orphan_tests::build_clpi; + let a = audit_one( + build_clpi(&[(0x1200, c::PG, "eng")]), + build_mpls(b"00001", 0x1100, c::TRUEHD, b"eng"), + ); + assert_eq!(a.class_counts(), (1, 1, 0, 0), "{:?}", a.rows); + } + + // PID 0 never makes a row, and a repeated PID in one clip keeps the first entry. + #[test] + fn pid_zero_is_skipped_and_first_duplicate_pid_wins() { + use crate::consts::coding_type as c; + let build_clpi = super::super::clpi_orphan_tests::build_clpi; + let a = audit_one( + build_clpi(&[ + (0, c::AC3, "eng"), + (0x1100, c::TRUEHD, "eng"), + (0x1100, c::AC3, "fra"), + ]), + build_mpls(b"00001", 0x1100, c::TRUEHD, b"eng"), + ); + assert_eq!(a.rows.len(), 1, "{:?}", a.rows); + assert_eq!(a.class_counts(), (0, 0, 1, 0), "{:?}", a.rows); + } } diff --git a/src/labels/criterion.rs b/src/labels/criterion.rs index a9e837ca..30163261 100644 --- a/src/labels/criterion.rs +++ b/src/labels/criterion.rs @@ -407,6 +407,24 @@ mod tests { assert_eq!(by_lang, [("eng", 2), ("fra", 1)]); } + // Two streams claiming the same number: the second is dropped, not emitted with the + // NO_STN_SLOT sentinel. + #[test] + fn parse_drops_a_stream_with_a_contradictory_number_claim() { + let sp = "a0ENG\ + a1FRA"; + let pc = "2a0\ + 2a1"; + let (mut disc, udf) = jar_disc(&[("streamproperties.xml", sp), ("playbackconfig.xml", pc)]); + let got = parse(&mut disc, &udf).expect("labels"); + let kept: Vec<(&str, u16)> = got + .labels + .iter() + .map(|l| (l.language.as_str(), l.stream_number)) + .collect(); + assert_eq!(kept, [("eng", 2)]); + } + // ── Additional hardening tests ───────────────────────────────────────── /// Spec: audio and subtitle counters are INDEPENDENT — audio fallback counter diff --git a/src/labels/ctrm.rs b/src/labels/ctrm.rs index 98124711..190758ec 100644 --- a/src/labels/ctrm.rs +++ b/src/labels/ctrm.rs @@ -226,10 +226,130 @@ fn parse_language_streams_text(text: &str) -> Vec { labels } -// NOTE: `parse_menu_base` / `parse_menu_base_text` structurally belong above this -// module but are left below (lint allowed) — a ~120-line move is safer as its -// own focused change. -#[allow(clippy::items_after_test_module)] +// ── menu_base.prop parser ────────────────────────────────────────────────── + +fn parse_menu_base(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option> { + let data = super::read_jar_file(reader, udf, "menu_base.prop")?; + let text = std::str::from_utf8(&data).ok()?; + let labels = parse_menu_base_text(text); + if labels.is_empty() { + return None; + } + Some(labels) +} + +// Parses menu_base.prop body into labels. Split from parse_menu_base +// (I/O + UTF-8 decode only) so tests exercise real parsing logic. +// Returns labels sorted by (type, number). +fn parse_menu_base_text(text: &str) -> Vec { + // Parse key=value, group by prefix + // BTreeMap: deterministic prefix order so equal (type, number) ties sort stably. + let mut entries: BTreeMap> = BTreeMap::new(); + + for line in text.lines() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let eq_pos = match line.find('=') { + Some(p) => p, + None => continue, + }; + let full_key = line[..eq_pos].trim(); + let value = line[eq_pos + 1..].trim_start(); + + if let Some(dot_pos) = full_key.rfind('.') { + let prefix = full_key[..dot_pos].to_string(); + let key = full_key[dot_pos + 1..].to_string(); + if !entries.contains_key(&prefix) && entries.len() >= MAX_CTRM_LABELS { + continue; + } + let props = entries.entry(prefix).or_default(); + if props.len() < MAX_PROPS_PER_PREFIX || props.contains_key(&key) { + props.insert(key, value.to_string()); + } + } + } + + let mut labels = Vec::new(); + + for (prefix, props) in &entries { + // Audio: has "streamNumber" or "audioStream" and audio-related class + let is_audio = props + .get("class") + .is_some_and(|c| c.contains("AudioButton")) + || prefix.starts_with("audio_"); + let is_subtitle = props + .get("class") + .is_some_and(|c| c.contains("SubtitleButton")) + || prefix.starts_with("subtitle_"); + + let stream_num_str = props + .get("streamNumber") + .or_else(|| props.get("audioStream")) + .or_else(|| props.get("subtitleStream")); + + let stream_num: u16 = match stream_num_str.and_then(|s| s.parse().ok()) { + Some(n) if n > 0 => n, + _ => continue, + }; + + if !is_audio && !is_subtitle { + continue; + } + + // Resolve the stream type FIRST: when an entry trips both flags + // (e.g. an `audio_` prefix with a class containing + // "SubtitleButton"), audio wins the type. + let stream_type = if is_audio { + StreamLabelType::Audio + } else { + StreamLabelType::Subtitle + }; + + let name = props.get("name").cloned().unwrap_or_default(); + + // Ask vocab first (word-boundary matched, avoiding the "Commenter" + // false positive of the old `name.contains("comment")`), then fall + // back to the structural prefix check (`audio_commentary.foo`-style). + let purpose = match vocab::purpose(&name) { + LabelPurpose::Normal if prefix_is_commentary(prefix) => LabelPurpose::Commentary, + p => p, + }; + + // Qualifier (SDH/Forced) is a subtitle-only concept. Gate on the + // RESOLVED type, not the raw is_subtitle flag, so an entry that + // resolved to Audio never carries a subtitle qualifier. + let qualifier = if stream_type == StreamLabelType::Subtitle { + vocab::qualifier(&name) + } else { + LabelQualifier::None + }; + + // Try to extract language from audioLanguage/subtitleLanguage prop + let language = props + .get("audioLanguage") + .or_else(|| props.get("subtitleLanguage")) + .cloned() + .unwrap_or_default(); + + labels.push(StreamLabel { + stream_id: None, + stream_number: stream_num, + stream_type, + language, + name, + purpose, + qualifier, + codec_hint: String::new(), + variant: String::new(), + }); + } + + labels.sort_by_key(|l| (l.stream_type as u8, l.stream_number)); + labels +} + #[cfg(test)] mod tests { use super::*; @@ -274,6 +394,55 @@ mod tests { assert_eq!(parse_language_streams_text(&ls).len(), MAX_CTRM_LABELS); } + #[test] + fn menu_base_sorts_by_type_then_number_across_prefixes() { + let labels = parse_props( + "a_sub.class=SubtitleButton\na_sub.streamNumber=1\n\ + b_aud.class=AudioButton\nb_aud.streamNumber=1\n\ + c_aud.class=AudioButton\nc_aud.streamNumber=2\n\ + d_aud.class=AudioButton\nd_aud.streamNumber=1\nd_aud.name=Low\n", + ); + let key: Vec<(StreamLabelType, u16)> = labels + .iter() + .map(|l| (l.stream_type, l.stream_number)) + .collect(); + assert_eq!( + key, + vec![ + (StreamLabelType::Audio, 1), + (StreamLabelType::Audio, 1), + (StreamLabelType::Audio, 2), + (StreamLabelType::Subtitle, 1), + ] + ); + } + + #[test] + fn menu_base_caps_properties_per_prefix() { + let mut text = String::from("audio_1.streamNumber=1\n"); + for i in 0..(MAX_PROPS_PER_PREFIX + 50) { + text.push_str(&format!("audio_1.junk{i}=x\n")); + } + // Past the cap: a new key is dropped, an already-retained key is still updated. + text.push_str("audio_1.late=x\naudio_1.name=Late\naudio_1.streamNumber=2\n"); + let labels = parse_props(&text); + assert_eq!(labels.len(), 1); + assert_eq!(labels[0].stream_number, 2); + assert_eq!(labels[0].name, ""); + } + + #[test] + fn menu_base_accepts_alternate_stream_and_language_keys() { + let labels = parse_props( + "audio_1.audioStream=4\n\ + subtitle_1.subtitleStream=5\nsubtitle_1.subtitleLanguage=fra\n", + ); + assert_eq!(labels.len(), 2); + assert_eq!(labels[0].stream_number, 4); + assert_eq!(labels[1].stream_number, 5); + assert_eq!(labels[1].language, "fra"); + } + #[test] fn merge_fills_names_and_appends_only_missing() { let mk = |n: u16, name: &str| StreamLabel { @@ -810,127 +979,3 @@ mod tests { assert!(prefix_is_commentary("comm_track_1")); } } - -// ── menu_base.prop parser ────────────────────────────────────────────────── - -fn parse_menu_base(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option> { - let data = super::read_jar_file(reader, udf, "menu_base.prop")?; - let text = std::str::from_utf8(&data).ok()?; - let labels = parse_menu_base_text(text); - if labels.is_empty() { - return None; - } - Some(labels) -} - -// Parses menu_base.prop body into labels. Split from parse_menu_base -// (I/O + UTF-8 decode only) so tests exercise real parsing logic. -// Returns labels sorted by (type, number). -fn parse_menu_base_text(text: &str) -> Vec { - // Parse key=value, group by prefix - // BTreeMap: deterministic prefix order so equal (type, number) ties sort stably. - let mut entries: BTreeMap> = BTreeMap::new(); - - for line in text.lines() { - let line = line.trim(); - if line.is_empty() || line.starts_with('#') { - continue; - } - let eq_pos = match line.find('=') { - Some(p) => p, - None => continue, - }; - let full_key = line[..eq_pos].trim(); - let value = line[eq_pos + 1..].trim_start(); - - if let Some(dot_pos) = full_key.rfind('.') { - let prefix = full_key[..dot_pos].to_string(); - let key = full_key[dot_pos + 1..].to_string(); - if !entries.contains_key(&prefix) && entries.len() >= MAX_CTRM_LABELS { - continue; - } - let props = entries.entry(prefix).or_default(); - if props.len() < MAX_PROPS_PER_PREFIX || props.contains_key(&key) { - props.insert(key, value.to_string()); - } - } - } - - let mut labels = Vec::new(); - - for (prefix, props) in &entries { - // Audio: has "streamNumber" or "audioStream" and audio-related class - let is_audio = props - .get("class") - .is_some_and(|c| c.contains("AudioButton")) - || prefix.starts_with("audio_"); - let is_subtitle = props - .get("class") - .is_some_and(|c| c.contains("SubtitleButton")) - || prefix.starts_with("subtitle_"); - - let stream_num_str = props - .get("streamNumber") - .or_else(|| props.get("audioStream")) - .or_else(|| props.get("subtitleStream")); - - let stream_num: u16 = match stream_num_str.and_then(|s| s.parse().ok()) { - Some(n) if n > 0 => n, - _ => continue, - }; - - if !is_audio && !is_subtitle { - continue; - } - - // Resolve the stream type FIRST: when an entry trips both flags - // (e.g. an `audio_` prefix with a class containing - // "SubtitleButton"), audio wins the type. - let stream_type = if is_audio { - StreamLabelType::Audio - } else { - StreamLabelType::Subtitle - }; - - let name = props.get("name").cloned().unwrap_or_default(); - - // Ask vocab first (word-boundary matched, avoiding the "Commenter" - // false positive of the old `name.contains("comment")`), then fall - // back to the structural prefix check (`audio_commentary.foo`-style). - let purpose = match vocab::purpose(&name) { - LabelPurpose::Normal if prefix_is_commentary(prefix) => LabelPurpose::Commentary, - p => p, - }; - - // Qualifier (SDH/Forced) is a subtitle-only concept. Gate on the - // RESOLVED type, not the raw is_subtitle flag, so an entry that - // resolved to Audio never carries a subtitle qualifier. - let qualifier = if stream_type == StreamLabelType::Subtitle { - vocab::qualifier(&name) - } else { - LabelQualifier::None - }; - - // Try to extract language from audioLanguage/subtitleLanguage prop - let language = props - .get("audioLanguage") - .or_else(|| props.get("subtitleLanguage")) - .cloned() - .unwrap_or_default(); - - labels.push(StreamLabel { - stream_id: None, - stream_number: stream_num, - stream_type, - language, - name, - purpose, - qualifier, - codec_hint: String::new(), - variant: String::new(), - }); - } - - labels.sort_by_key(|l| (l.stream_type as u8, l.stream_number)); - labels -} diff --git a/src/labels/dbp.rs b/src/labels/dbp.rs index b41bcd1c..38bf7db0 100644 --- a/src/labels/dbp.rs +++ b/src/labels/dbp.rs @@ -454,6 +454,20 @@ mod tests { assert_eq!(subs[&2], "Spanish"); } + #[test] + fn collect_skips_entries_with_an_empty_or_missing_label() { + let mut audios = BTreeMap::new(); + let mut subs = BTreeMap::new(); + for s in [ + "XTextField,Audio1,,rest", + "XTextField,Audio2", + "XTextField,Subtitle1, ,rest", + ] { + collect_textfield(s, &mut audios, &mut subs); + } + assert!(audios.is_empty() && subs.is_empty()); + } + #[test] fn collect_ignores_non_textfield_strings() { let mut audios = BTreeMap::new(); diff --git a/src/labels/deluxe.rs b/src/labels/deluxe.rs index b0a0715b..3459d3e4 100644 --- a/src/labels/deluxe.rs +++ b/src/labels/deluxe.rs @@ -22,6 +22,7 @@ use super::{LabelPurpose, LabelQualifier, ParseResult, StreamLabel, StreamLabelT use crate::sector::SectorSource; use crate::udf::UdfFs; use std::collections::{BTreeMap, HashMap, HashSet}; +use std::rc::Rc; pub fn detect(reader: &mut dyn SectorSource, udf: &UdfFs) -> bool { // Real signal is `com/bydeluxe/` in a jar's central directory: a cheap @@ -31,12 +32,19 @@ pub fn detect(reader: &mut dyn SectorSource, udf: &UdfFs) -> bool { } pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option { + // One inflate budget for every sweep over every jar this parse opens. + parse_with_budget(reader, udf, jar::PARSE_INFLATE_BUDGET) +} + +fn parse_with_budget( + reader: &mut dyn SectorSource, + udf: &UdfFs, + mut budget: u64, +) -> Option { // Per-studio dispatch signal: Deluxe authored the framework for several // studios, each shipping `/BDMV/JAR//config.xml` naming the studio. // Matching below is studio-agnostic, so this is informational only (logged). let studio = detect_studio(reader, udf); - // One inflate budget for every sweep over every jar this parse opens. - let mut budget = jar::PARSE_INFLATE_BUDGET; jar::for_each_jar(reader, udf, |entry_name, archive| { if !jar::has_path_prefix(archive, "com/bydeluxe/") { @@ -363,10 +371,11 @@ fn identify_master_enums( let matches_fp = FINGERPRINTS.iter().any(|fp| fp_matches(fp, &ldcs)); if pool.insert(key, ldcs) { if matches_fp && !pool.insert_fields(key, clinit_enum_field_names(class)) { - tracing::debug!( + tracing::warn!( class = ?key, bytes = pool.bytes, - "deluxe: candidate pool cap hit, enum field names dropped" + "deluxe: candidate pool cap hit, enum field names dropped; \ + getstatic references to this enum may not resolve" ); } } else { @@ -548,6 +557,13 @@ fn find_binding_classes( const MIN_GETSTATIC: usize = 4; let mut candidates: Vec<(String, usize)> = Vec::new(); jar::for_each_class_budgeted(archive, budget, |class_name, class| { + // A master enum's own reads every constant into $VALUES; it is not a binding. + if class + .this_class_name() + .is_some_and(|n| master_enum_classes.contains(n)) + { + return; + } let count = count_master_enum_getstatic(class, master_enum_classes); if count >= MIN_GETSTATIC { candidates.push((class_name.to_string(), count)); @@ -618,10 +634,10 @@ pub(crate) enum StackVal { // Reference to org.bluray.ti.CodingType; field name (e.g. // DOLBY_AC3_AUDIO) is the codec id — NOT a Deluxe-internal enum, read // straight from the binding constructor's getstatic operand. - CodingType(String), + CodingType(Rc), /// An uninitialized `new` object — popped by the matching /// invokespecial. - NewObj(String), + NewObj(Rc), /// Anything we can't model — stack effect tracked but content /// opaque. Lets the walker stay in sync past loads/computed /// values it doesn't understand. @@ -663,7 +679,7 @@ pub(crate) struct Decoded { pub drift: usize, } -/// Phase D entry point: find the binding class in `archive` and run the +/// Phase D entry point: fetch the named binding class from `archive` and run the /// bytecode walker against its ``. fn decode_binding( archive: &mut jar::Jar, @@ -671,14 +687,9 @@ fn decode_binding( master: &MasterEnumTable, budget: &mut u64, ) -> Decoded { - let target_name = binding_class_name.to_string(); - // Short-circuit on the name match: the sweep stops iterating (and - // decompressing/parsing remaining .class entries) once the closure - // returns Some, instead of walking the whole jar past the target. - jar::try_each_class_budgeted(archive, budget, |class_name, class| { - if class_name != target_name { - return None; - } + // Fetched by entry name: the classes ahead of it in the jar are not inflated or + // parsed, so they do not spend the shared budget. + jar::try_class_budgeted(archive, binding_class_name, budget, |class| { Some(decode_binding_class(class, master)) }) .unwrap_or_default() @@ -898,9 +909,9 @@ impl<'a> BindingDecoder<'a> { let class_name = insn .cp_index() .and_then(|i| self.pool.class_name(i)) - .unwrap_or("") - .to_string(); - self.push(StackVal::NewObj(class_name)); + .unwrap_or(""); + // Rc: `dup` copies the handle, not a name of up to 64 KiB. + self.push(StackVal::NewObj(class_name.into())); } // Word-level stack ops (a long/double is one two-word value). POP => drop(self.take_words(insn, 1)), @@ -943,7 +954,7 @@ impl<'a> BindingDecoder<'a> { return; }; let val = if m.class_name == BD_CODING_TYPE_CLASS { - StackVal::CodingType(m.name.to_string()) + StackVal::CodingType(m.name.into()) } else if let Some((kind, ordinal)) = self.master.resolve(m.class_name, m.name) { StackVal::EnumRef { kind, ordinal } } else { @@ -988,12 +999,12 @@ impl<'a> BindingDecoder<'a> { // Underneath the args: the object the constructor // operates on. For our pattern it's NewObj(X). match self.stack.pop() { - Some(StackVal::NewObj(name)) if is_init && name == member.class_name => { + Some(StackVal::NewObj(name)) if is_init && *name == *member.class_name => { // Bounded by MAX_CONSTRUCTIONS: an unbounded push here // is ~1 GiB reachable from a crafted ``. if !is_container && self.constructions.len() < MAX_CONSTRUCTIONS { self.constructions.push(Construction { - binding_type: name, + binding_type: name.to_string(), args, }); } @@ -1173,9 +1184,9 @@ fn interpret_streams(constructions: &[Construction], master: &MasterEnumTable) - StackVal::CodingType(name) if slot.is_none() => { slot = coding_slot(name); if slot == Some(CodingSlot::Audio) { - coding_type = Some(name.clone()); + coding_type = Some(name.to_string()); } else if slot.is_none() { - unknown_coding = Some(name.clone()); + unknown_coding = Some(name.to_string()); } } StackVal::Int(n) => { @@ -1216,7 +1227,7 @@ fn interpret_streams(constructions: &[Construction], master: &MasterEnumTable) - let codec_hint = coding_type .as_deref() .map(coding_type_to_codec_hint) - .map(str::to_string) + .map(neutralise) .unwrap_or_default(); // Neither `+= 1` nor `saturating_add` is correct: saturation caused a @@ -1246,7 +1257,7 @@ fn interpret_streams(constructions: &[Construction], master: &MasterEnumTable) - // Resolve language ordinal → enum value string via master // table; then route through vocab::lang for ISO code + variant. - let lang_value = master.value("Language", lang_ord).unwrap_or("").to_string(); + let lang_value = neutralise(master.value("Language", lang_ord).unwrap_or("")); let (language, variant) = match vocab::lang(&lang_value) { Some(li) => (li.code.to_string(), li.variant.to_string()), None if !lang_value.is_empty() => (lang_value.clone(), String::new()), @@ -1289,6 +1300,16 @@ fn interpret_streams(constructions: &[Construction], master: &MasterEnumTable) - out } +// Disc-authored text reaches the label list and from there track names and CLI output: +// control characters are escaped, everything else is kept as authored. +fn neutralise(s: &str) -> String { + if s.chars().any(char::is_control) { + s.escape_debug().to_string() + } else { + s.to_string() + } +} + // Which stream list each binding type enumerates, learned from the constructions that DID // resolve a language; a type that resolved as both kinds is left out (no consistent answer). fn slot_kinds(constructions: &[Construction]) -> HashMap<&str, Option> { @@ -1997,6 +2018,25 @@ mod tests { ); } + // A master enum's own getstatics every constant (old-javac $VALUES): + // counting it would set the 40% bar and drop the real binding classes. + #[test] + fn find_binding_classes_skips_the_master_enum_itself() { + let master_classes: HashSet<&str> = ["LanguageEnum"].into_iter().collect(); + let lang = class_with_getstatic_refs("LanguageEnum", "LanguageEnum", 70); + let audio = class_with_getstatic_refs("Audio", "LanguageEnum", 6); + let subs = class_with_getstatic_refs("Subs", "LanguageEnum", 8); + let zip = build_zip(&[ + ("LanguageEnum.class", lang), + ("Audio.class", audio), + ("Subs.class", subs), + ]); + let mut archive = open_jar(zip); + let candidates = find_binding_classes(&mut archive, &master_classes, &mut unbounded()); + let names: Vec<&str> = candidates.iter().map(|(n, _)| n.as_str()).collect(); + assert_eq!(names, vec!["Subs.class", "Audio.class"]); + } + #[test] fn find_binding_classes_empty_master_set_yields_no_candidates() { let master_classes: HashSet<&str> = HashSet::new(); @@ -2008,6 +2048,31 @@ mod tests { // ── Phase D: decode_binding (Jar-level short-circuit wrapper) ─────────── + // Classes ahead of the binding class in the jar are not inflated, so they cannot + // spend the budget the walk needs for the binding class itself. + #[test] + fn decode_binding_does_not_charge_classes_ahead_of_the_target() { + let target = class_with_simple_construction("Ignored"); + let need = target.len() as u64; + let decoy = class_with_getstatic_refs("Ignored2", "LanguageEnum", 40); + assert!(decoy.len() as u64 > need); + let zip = build_zip(&[ + ("com/bydeluxe/Decoy.class", decoy), + ("com/bydeluxe/Target.class", target), + ]); + let mut archive = open_jar(zip); + let mut budget = need; + let ctors = decode_binding( + &mut archive, + "com/bydeluxe/Target.class", + &lang_enum_master(), + &mut budget, + ) + .constructions; + assert_eq!(ctors.len(), 1); + assert_eq!(budget, 0); + } + #[test] fn decode_binding_finds_named_class_and_stops_at_first_match() { // Matches by Jar entry path, not `this_class` name. Target path has a @@ -2623,7 +2688,7 @@ mod tests { ); for v in &decoder.stack { match v { - StackVal::NewObj(name) => assert_eq!(name, "AudioSlot"), + StackVal::NewObj(name) => assert_eq!(&**name, "AudioSlot"), other => panic!("expected NewObj(\"AudioSlot\") x2, got {other:?}"), } } @@ -3091,6 +3156,13 @@ mod tests { // Two-class Deluxe jar on a disc: a 70-value Language enum and a binding class // of four `new Slot(Lang.English)`, each preceded by `extra` bytecode. fn deluxe_disc(extra: &[u8]) -> (crate::udf::fixture::MemDisc, crate::udf::UdfFs) { + deluxe_disc_with(&[(extra, 4)]) + } + + // As `deluxe_disc`, over one binding class per `(extra bytecode, binding count)`. + fn deluxe_disc_with( + binds: &[(&[u8], usize)], + ) -> (crate::udf::fixture::MemDisc, crate::udf::UdfFs) { use crate::udf::fixture::{DirSpec, MemDisc, build_udf_skeleton, file_with, lay_dir}; let mut values = vec!["English", "French", "Spanish", "Dutch"]; let rest: Vec = (4..70).map(|i| format!("L{i}")).collect(); @@ -3128,28 +3200,38 @@ mod tests { CpInfo::Utf8("com/bydeluxe/Bind".into()), CpInfo::Class { name_index: 16 }, ]; - let mut code = Vec::new(); - for _ in 0..4 { - code.extend([NEW, 0, 11, DUP]); - code.extend_from_slice(extra); - code.extend([GETSTATIC, 0, 9, INVOKESPECIAL, 0, 15, POP]); + let bind_class = |extra: &[u8], count: usize| { + let mut code = Vec::new(); + for _ in 0..count { + code.extend([NEW, 0, 11, DUP]); + code.extend_from_slice(extra); + code.extend([GETSTATIC, 0, 9, INVOKESPECIAL, 0, 15, POP]); + } + code.push(RETURN); + encode_class( + &cp, + 17, + &[MethodSpec { + name_index: 1, + descriptor_index: 2, + code_attr_name_index: 3, + max_stack: 4, + code, + }], + ) + }; + let mut entries = vec![("com/bydeluxe/Lang.class".to_string(), lang)]; + for (i, (extra, count)) in binds.iter().enumerate() { + entries.push(( + format!("com/bydeluxe/Bind{i}.class"), + bind_class(extra, *count), + )); } - code.push(RETURN); - let bind = encode_class( - &cp, - 17, - &[MethodSpec { - name_index: 1, - descriptor_index: 2, - code_attr_name_index: 3, - max_stack: 4, - code, - }], - ); - let jar = build_zip(&[ - ("com/bydeluxe/Lang.class", lang), - ("com/bydeluxe/Bind.class", bind), - ]); + let entries: Vec<(&str, Vec)> = entries + .iter() + .map(|(n, b)| (n.as_str(), b.clone())) + .collect(); + let jar = build_zip(&entries); let dir = |name: &str, icb, files, subdirs| DirSpec { name: name.to_string(), icb_lba: icb, @@ -3188,6 +3270,122 @@ mod tests { assert_eq!(r.confidence, super::super::Confidence::Low); } + // The inflate budget decides confidence: the smallest budget that still completes + // the parse spends it to zero, so the result is not High; one more byte is. + #[test] + fn a_spent_inflate_budget_is_not_high_confidence() { + let (mut disc, udf) = deluxe_disc(&[]); + let (mut lo, mut hi) = (0u64, 1u64 << 20); + while lo + 1 < hi { + let mid = (lo + hi) / 2; + if parse_with_budget(&mut disc, &udf, mid).is_some() { + hi = mid; + } else { + lo = mid; + } + } + let tight = parse_with_budget(&mut disc, &udf, hi).expect("completes at the minimum"); + assert_eq!(tight.confidence, super::super::Confidence::Low); + let roomy = parse_with_budget(&mut disc, &udf, hi + 1).expect("completes"); + assert_eq!(roomy.confidence, super::super::Confidence::High); + } + + // Drift in any binding class downgrades the union, whichever class carries it. + #[test] + fn drift_in_any_binding_class_is_not_high_confidence() { + let drifty: &[u8] = &[0xA8, 0, 3]; + for binds in [[(&[][..], 4), (drifty, 4)], [(drifty, 4), (&[][..], 4)]] { + let (mut disc, udf) = deluxe_disc_with(&binds); + let r = parse(&mut disc, &udf).expect("deluxe labels"); + assert_eq!(r.confidence, super::super::Confidence::Low); + } + } + + // The union across binding classes is bounded by the same cap as each walk. + #[test] + fn constructions_across_binding_classes_are_capped() { + let half = MAX_CONSTRUCTIONS / 2 + 100; + let (mut disc, udf) = deluxe_disc_with(&[(&[], half), (&[], half)]); + let r = parse(&mut disc, &udf).expect("deluxe labels"); + assert_eq!(r.labels.len(), MAX_CONSTRUCTIONS); + } + + // A control character in a CodingType name or language value never reaches the + // label text raw. + #[test] + fn disc_authored_control_characters_are_escaped_in_labels() { + let master = MasterEnumTable::from(&[( + "Language", + MasterEnum { + class_name: "LanguageEnum".into(), + values: vec!["Xx\x1b[31mYy".into()], + fields: Vec::new(), + }, + )]); + let constructions = vec![Construction { + binding_type: "ng".into(), + args: vec![ + StackVal::EnumRef { + kind: "Language", + ordinal: 0, + }, + StackVal::CodingType("\x1b]0;pwn\x07ATMOS_AUDIO".into()), + ], + }]; + let out = interpret_streams(&constructions, &master); + assert_eq!(out.len(), 1); + for text in [&out[0].codec_hint, &out[0].language, &out[0].name] { + assert!(!text.chars().any(char::is_control), "{text:?}"); + } + assert!(out[0].codec_hint.contains("ATMOS_AUDIO")); + } + + // An "English SDH" Language value carries the qualifier the Purpose enum left unset. + #[test] + fn language_value_supplies_the_qualifier_when_purpose_does_not() { + let master = MasterEnumTable::from(&[( + "Language", + MasterEnum { + class_name: "LanguageEnum".into(), + values: vec!["English SDH".into(), "English RNIB".into()], + fields: Vec::new(), + }, + )]); + let slot = |ord: u16| Construction { + binding_type: "SubtitleSlot".into(), + args: vec![StackVal::EnumRef { + kind: "Language", + ordinal: ord, + }], + }; + let out = interpret_streams(&[slot(0), slot(1)], &master); + assert_eq!(out[0].qualifier, LabelQualifier::Sdh); + assert_eq!(out[1].qualifier, LabelQualifier::DescriptiveService); + } + + // The documented Purpose ordinal mapping, every row: purpose and qualifier. + #[test] + fn deluxe_purpose_every_ordinal_maps_as_documented() { + use LabelPurpose::*; + for (ord, want) in [ + (0, Normal), + (1, Commentary), + (2, Normal), + (3, Normal), + (4, Descriptive), + (5, Score), + (6, Normal), + (7, Descriptive), + (8, Normal), + ] { + assert_eq!( + deluxe_purpose_to_label(ord), + (want, LabelQualifier::None), + "ordinal {ord}" + ); + } + } + // ── interpret_streams + deluxe_purpose_to_label tests ─────────────────── #[test] diff --git a/src/labels/fox.rs b/src/labels/fox.rs index c597504a..e463adb8 100644 --- a/src/labels/fox.rs +++ b/src/labels/fox.rs @@ -528,6 +528,45 @@ mod tests { assert_eq!(feature_hint(doc).and_then(|h| h.playlist_id), Some(900)); } + /// A full tie (same stream count, same `durs`) keeps the first feature. + #[test] + fn full_tie_keeps_the_first_feature() { + let doc = r#" + + + "#; + assert_eq!(feature_hint(doc).and_then(|h| h.playlist_id), Some(800)); + } + + /// Stream count outranks duration: a shorter but richer feature wins. + #[test] + fn stream_count_outranks_duration() { + let doc = r#" + + + + "#; + assert_eq!(feature_hint(doc).and_then(|h| h.playlist_id), Some(900)); + } + + /// Attribute values from the untrusted manifest are matched case-insensitively. + #[test] + fn manifest_attribute_values_are_case_insensitive() { + let doc = r#" + + + "#; + let labels = labels_from_dcx(doc); + assert_eq!(labels.len(), 3); + assert_eq!(labels[0].purpose, LabelPurpose::Descriptive); + assert_eq!(labels[1].qualifier, LabelQualifier::Sdh); + assert_eq!(labels[2].qualifier, LabelQualifier::Forced); + } + /// A stated sub-minute `name="feature"` is a decoy and is skipped in favour /// of the real (longer) feature. #[test] diff --git a/src/labels/jar.rs b/src/labels/jar.rs index b5eef34c..6b8761f5 100644 --- a/src/labels/jar.rs +++ b/src/labels/jar.rs @@ -383,6 +383,23 @@ where }) } +/// Parse the one `.class` entry named `entry_name` (its full jar path) and call `f` on it. +/// Every other entry is skipped by name, so it is neither inflated nor charged to `budget`. +pub fn try_class_budgeted( + archive: &mut ZipArchive, + entry_name: &str, + budget: &mut u64, + mut f: F, +) -> Option +where + F: FnMut(&ClassFile) -> Option, +{ + let is_target = |n: &str| n == entry_name; + try_each_entry(archive, is_target, MAX_CLASS_BYTES, budget, |_, bytes| { + f(&ClassFile::parse(bytes).ok()?) + }) +} + /// [`try_each_class_budgeted`] visiting every class (no short-circuit). pub fn for_each_class_budgeted( archive: &mut ZipArchive, @@ -710,6 +727,53 @@ mod tests { ); } + #[test] + fn is_jar_matches_files_with_a_case_insensitive_extension() { + let entry = |name: &str, is_dir| crate::udf::DirEntry { + name: name.to_string(), + is_dir, + meta_lba: 0, + size: 0, + entries: Vec::new(), + }; + assert!(is_jar(&entry("00000.jar", false))); + assert!(is_jar(&entry("00000.JAR", false))); + assert!(!is_jar(&entry("00000.jar", true))); + assert!(!is_jar(&entry("00000.xml", false))); + } + + // Directory entries and upper-case `.CLASS` bytecode are never offered as resources. + #[test] + fn try_each_resource_skips_directory_entries_and_upper_case_class() { + use std::io::{Cursor, Write as _}; + let mut buf = Vec::new(); + { + let mut w = zip::ZipWriter::new(Cursor::new(&mut buf)); + let opts = zip::write::SimpleFileOptions::default() + .compression_method(zip::CompressionMethod::Stored); + w.add_directory("assets/", opts).unwrap(); + w.start_file("com/x/Menu.CLASS", opts).unwrap(); + w.write_all(MINIMAL_CLASS).unwrap(); + w.start_file("assets/a.xml", opts).unwrap(); + w.write_all(b"").unwrap(); + w.finish().unwrap(); + } + let mut jar = open(buf); + let mut seen: Vec = Vec::new(); + let mut budget = u64::MAX; + let _: Option<()> = try_each_resource( + &mut jar, + |_| true, + 1024, + &mut budget, + |name, _| { + seen.push(name.to_string()); + None + }, + ); + assert_eq!(seen, ["assets/a.xml"]); + } + // An entry the name filter rejects is never inflated: it costs no budget. #[test] fn unwanted_resources_are_not_inflated() { diff --git a/src/labels/mod.rs b/src/labels/mod.rs index 256d97a9..850eee8b 100644 --- a/src/labels/mod.rs +++ b/src/labels/mod.rs @@ -188,11 +188,13 @@ pub(crate) fn name_words(s: &str) -> Vec { let mut words = Vec::new(); for part in s.split(|c: char| !c.is_ascii_alphabetic()) { let mut word = String::new(); + // The boundary test reads the source char: `word` holds it lower-cased. + let mut prev_lower = false; for c in part.chars() { - if c.is_ascii_uppercase() && word.chars().last().is_some_and(|l| l.is_ascii_lowercase()) - { + if c.is_ascii_uppercase() && prev_lower { words.push(std::mem::take(&mut word)); } + prev_lower = c.is_ascii_lowercase(); word.push(c.to_ascii_lowercase()); } if !word.is_empty() { @@ -300,8 +302,16 @@ pub fn apply( titles: &mut [DiscTitle], ) -> Option { let (labels, winner_hint) = - std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| extract(reader, udf))) - .unwrap_or_default(); + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| extract(reader, udf))) { + Ok(r) => r, + Err(payload) => { + tracing::warn!( + panic = %panic_message(payload.as_ref()), + "label extraction panicked; streams left unlabelled" + ); + Default::default() + } + }; if !labels.is_empty() { apply_labels(&labels, titles); } @@ -311,11 +321,26 @@ pub fn apply( let hint = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { resolve_feature_hint(reader, udf, winner_hint, titles) })) - .ok() - .flatten(); + .unwrap_or_else(|payload| { + tracing::warn!( + panic = %panic_message(payload.as_ref()), + "feature-hint resolution panicked; no hint" + ); + None + }); hint.filter(|h| !h.is_empty()) } +// A caught panic's message, control characters escaped (it can carry disc-derived text). +fn panic_message(payload: &(dyn std::any::Any + Send)) -> String { + let msg = payload + .downcast_ref::<&str>() + .copied() + .or_else(|| payload.downcast_ref::().map(String::as_str)) + .unwrap_or("non-string panic payload"); + msg.escape_debug().to_string() +} + // The disc's feature-playlist hint: the winning parser's manifest hint when it // carries one, else the generic BD-J menu-walk (`bdj_feature::resolve`). Kept // separate from extraction so a hint survives an empty/absent label set. @@ -1170,7 +1195,8 @@ pub struct LabelAnalysis { /// MPLS itself was the chosen parser. Diagnostic for the labels-analyze /// tool. pub gap_fill_added: usize, - /// Per-playlist chapter summary: `(playlist_filename, chapter_count, duration_secs)`. + /// Per-playlist chapter summary rows ([`ChapterSummary`]): playlist filename, + /// chapter count, duration in seconds. /// Sourced from MPLS PlaylistMark entries with `mark_type == 1` /// (chapter entries). Ordered by playlist filename. Empty if no /// MPLS files have parseable marks, or the disc isn't Blu-ray. @@ -1259,6 +1285,15 @@ pub(crate) fn read_jar_file( mod registry_tests { use super::*; + #[test] + fn name_words_splits_camel_case_but_not_all_caps() { + assert_eq!(name_words("MAIN_FEATURE"), ["main", "feature"]); + assert_eq!(name_words("FEATURE_A"), ["feature", "a"]); + assert_eq!(name_words("Feature_TRAILER"), ["feature", "trailer"]); + assert_eq!(name_words("MainFeature_A"), ["main", "feature", "a"]); + assert_eq!(name_words("FeatureHD"), ["feature", "hd"]); + } + fn dir_entry( name: &str, is_dir: bool, @@ -1454,22 +1489,6 @@ mod registry_tests { let none: Vec<(&'static str, ParseResult)> = Vec::new(); assert!(select_result(&none).is_none()); } - - /// Per-parser sanity: every parser has both detect and parse - /// hooked up. Catches accidental nullification (e.g. someone - /// stubbing `parse` to always-None during a refactor). - #[test] - fn parsers_registry_all_entries_populated() { - for (name, detect, parse) in PARSERS { - // Function pointers can't be null in safe Rust, so this just fails to - // compile if the tuple shape changes (e.g. a 4th field is added) - // without updating callers; it also marks "these parsers exist." - let _ = (name, detect, parse); - } - // Non-empty invariant is covered separately by - // `parsers_registry_order_locked`, whose assert_eq! on the expected - // order fails if PARSERS is empty. - } } // ── MPLS floor merge tests ───────────────────────────────────────────────── @@ -3398,3 +3417,347 @@ mod feature_hint_pass_tests { ); } } + +// ── Label text tables, ordering, and extraction ─────────────────────────── + +#[cfg(test)] +mod label_table_tests { + use super::*; + use crate::disc::{AudioChannels, Codec, HdrFormat, Stream, SubtitleStream}; + use crate::udf::fixture::*; + + fn vendor(t: StreamLabelType, n: u16) -> StreamLabel { + StreamLabel { + stream_id: None, + stream_number: n, + stream_type: t, + language: "eng".into(), + name: String::new(), + purpose: LabelPurpose::Normal, + qualifier: LabelQualifier::None, + codec_hint: String::new(), + variant: String::new(), + } + } + + fn derived(clip: &str, pid: u16, n: u16) -> StreamLabel { + StreamLabel { + stream_id: Some(StreamId { + clip_id: clip.into(), + pid, + }), + ..vendor(StreamLabelType::Audio, n) + } + } + + #[test] + fn panic_message_escapes_control_characters() { + let p = std::panic::catch_unwind(|| panic!("bad \x1b[31m disc")).unwrap_err(); + assert_eq!(panic_message(p.as_ref()), "bad \\u{1b}[31m disc"); + } + + #[test] + fn iso639_b_codes_map_to_their_t_codes() { + for (b, t) in [ + ("alb", "sqi"), + ("arm", "hye"), + ("baq", "eus"), + ("bur", "mya"), + ("chi", "zho"), + ("cze", "ces"), + ("dut", "nld"), + ("fre", "fra"), + ("geo", "kat"), + ("ger", "deu"), + ("gre", "ell"), + ("ice", "isl"), + ("mac", "mkd"), + ("mao", "mri"), + ("may", "msa"), + ("per", "fas"), + ("rum", "ron"), + ("slo", "slk"), + ("tib", "bod"), + ("wel", "cym"), + ] { + assert_eq!(to_iso639_t(b), t, "{b}"); + assert_eq!(to_iso639_t(&b.to_ascii_uppercase()), t, "{b}"); + assert!(same_language(b, t), "{b}"); + } + assert_eq!(to_iso639_t("eng"), "eng"); + } + + #[test] + fn audio_channel_layouts_have_their_own_text() { + for (ch, want) in [ + (AudioChannels::Mono, "1.0"), + (AudioChannels::Stereo, "2.0"), + (AudioChannels::Stereo21, "2.1"), + (AudioChannels::Surround30, "3.0"), + (AudioChannels::Surround31, "3.1"), + (AudioChannels::Quad, "4.0"), + (AudioChannels::Surround41, "4.1"), + (AudioChannels::Surround50, "5.0"), + (AudioChannels::Surround51, "5.1"), + (AudioChannels::Surround60, "6.0"), + (AudioChannels::Surround61, "6.1"), + (AudioChannels::Surround70, "7.0"), + (AudioChannels::Surround71, "7.1"), + ] { + assert_eq!( + generate_audio_label(&Codec::Ac3, &ch, false), + format!("Dolby Digital {want}") + ); + } + } + + #[test] + fn video_resolution_token_boundaries() { + let tok = |w, h, i| generate_video_label(&Codec::Hevc, (w, h), i, &HdrFormat::Sdr, false); + let name = Codec::Hevc.name(); + for (w, h, interlaced, want) in [ + (7680, 4320, false, "8K"), + (7679, 4320, false, "4K"), + (3840, 2160, false, "4K"), + (1920, 1080, false, "1080p"), + (1920, 1080, true, "1080i"), + (1280, 720, false, "720p"), + (720, 576, false, "576p"), + (720, 576, true, "576i"), + (720, 480, false, "480p"), + (720, 480, true, "480i"), + (640, 479, false, ""), + ] { + let want = if want.is_empty() { + name.to_string() + } else { + format!("{name} {want}") + }; + assert_eq!(tok(w, h, interlaced), want, "{w}x{h} i={interlaced}"); + } + } + + #[test] + fn atmos_marker_only_rides_a_dolby_carrier() { + let l = |c| generate_audio_label_atmos(&c, &AudioChannels::Surround71, false); + assert_eq!(l(Codec::DtsHdMa), "DTS-HD Master Audio 7.1"); + assert_eq!(l(Codec::Dts), "DTS 7.1"); + assert_eq!(l(Codec::Ac3), "Dolby Digital 7.1"); + } + + #[test] + fn atmos_hint_is_consistent_only_with_a_dolby_carrier() { + assert!(codec_hint_consistent("Atmos", &Codec::TrueHd)); + assert!(codec_hint_consistent("Atmos", &Codec::Ac3Plus)); + assert!(!codec_hint_consistent("Atmos", &Codec::DtsHdMa)); + assert!(!codec_hint_consistent("Atmos", &Codec::Dts)); + assert!(codec_hint_consistent("High Resolution", &Codec::DtsHdHr)); + assert!(codec_hint_consistent("DTS-HD HR", &Codec::DtsHdHr)); + assert!(!codec_hint_consistent("Master Audio", &Codec::DtsHdHr)); + assert!(!codec_hint_consistent("High Resolution", &Codec::DtsHdMa)); + } + + #[test] + fn merged_labels_order_by_slot_then_stream() { + let mut labels = vec![ + vendor(StreamLabelType::Audio, 3), + vendor(StreamLabelType::Audio, 1), + derived("00002", 0x1100, 1), + derived("00001", 0x1101, 1), + derived("00001", 0x1100, 1), + derived("00001", 0x1102, 0), + ]; + sort_labels(&mut labels); + let order: Vec<(Option<(&str, u16)>, u16)> = labels + .iter() + .map(|l| { + ( + l.stream_id.as_ref().map(|i| (i.clip_id.as_str(), i.pid)), + l.stream_number, + ) + }) + .collect(); + assert_eq!( + order, + [ + (None, 1), + (None, 3), + (Some(("00001", 0x1102)), 0), + (Some(("00001", 0x1100)), 1), + (Some(("00001", 0x1101)), 1), + (Some(("00002", 0x1100)), 1), + ] + ); + } + + // Off the authoritative path, a label binds by bare ordinal only when the stream + // STATES the same language; an unknown stream language is not agreement. + #[test] + fn ordinal_binding_needs_a_stated_subtitle_language() { + let mut title = DiscTitle { + playlist: "00800.mpls".into(), + playlist_id: 800, + duration_secs: 7200.0, + size_bytes: 0, + clips: Vec::new(), + streams: vec![Stream::Subtitle(SubtitleStream { + pid: 0x12A0, + codec: Codec::Pgs, + language: String::new(), + forced: false, + qualifier: LabelQualifier::None, + codec_data: None, + })], + chapters: Vec::new(), + extents: Vec::new(), + content_format: crate::disc::ContentFormat::BdTs, + codec_privates: Vec::new(), + }; + let labels = vec![StreamLabel { + qualifier: LabelQualifier::Forced, + ..vendor(StreamLabelType::Subtitle, 1) + }]; + apply_labels(&labels, std::slice::from_mut(&mut title)); + let Stream::Subtitle(s) = &title.streams[0] else { + unreachable!() + }; + assert!(!s.forced); + assert_eq!(s.qualifier, LabelQualifier::None); + } + + // Minimal MPLS: one play item on `clip` whose STN lists one primary audio stream `pid`. + fn mpls_bytes(clip: &[u8; 5], pid: u16) -> Vec { + let mut item = Vec::new(); + item.extend_from_slice(clip); + item.extend_from_slice(b"M2TS"); + item.extend_from_slice(&[0u8; 3]); + item.extend_from_slice(&0u32.to_be_bytes()); + item.extend_from_slice(&(7000u32 * 45000).to_be_bytes()); + item.extend_from_slice(&[0u8; 12]); // UO mask, misc, still + item.extend_from_slice(&[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]); + item.extend_from_slice(&[3, 0x01]); + item.extend_from_slice(&pid.to_be_bytes()); + item.extend_from_slice(&[5, 0x81, 0x61]); + item.extend_from_slice(b"eng"); + let mut pl = vec![0u8; 6]; + pl.extend_from_slice(&1u16.to_be_bytes()); + pl.extend_from_slice(&[0u8; 2]); + pl.extend_from_slice(&(item.len() as u16).to_be_bytes()); + pl.extend_from_slice(&item); + let pl_len = (pl.len() - 4) as u32; + pl[0..4].copy_from_slice(&pl_len.to_be_bytes()); + let mut buf = b"MPLS0200".to_vec(); + buf.extend_from_slice(&40u32.to_be_bytes()); + buf.extend_from_slice(&[0u8; 28]); + buf.extend_from_slice(&pl); + buf + } + + fn dir(name: &str, icb: u32, files: Vec, subdirs: Vec) -> DirSpec { + DirSpec { + name: name.to_string(), + icb_lba: icb, + dir_data_lba: icb + 1, + files, + subdirs, + } + } + + fn open(root: DirSpec) -> (MemDisc, UdfFs) { + let mut disc = MemDisc::new(); + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = crate::udf::read_filesystem(&mut disc).expect("fs"); + (disc, udf) + } + + // A framework winner keeps its own labels and hint, and the MPLS floor adds the + // streams it named no label for. + #[test] + fn extract_merges_the_mpls_floor_into_a_framework_winner() { + let dcx = br#" + + "#; + let (mut disc, udf) = open(dir( + "", + 10, + vec![], + vec![dir( + "BDMV", + 20, + vec![], + vec![ + dir( + "JAR", + 30, + vec![], + vec![dir( + "00000", + 40, + vec![file_with("dcx.xml", 100, 2000, dcx.to_vec(), true)], + vec![], + )], + ), + dir( + "PLAYLIST", + 50, + vec![file_with( + "00800.mpls", + 101, + 3000, + mpls_bytes(b"00001", 0x1100), + true, + )], + vec![], + ), + ], + )], + )); + let (labels, hint) = extract(&mut disc, &udf); + assert_eq!(labels.iter().filter(|l| l.stream_id.is_none()).count(), 1); + assert_eq!(labels.iter().filter(|l| l.stream_id.is_some()).count(), 1); + assert_eq!(hint.and_then(|h| h.playlist_id), Some(800)); + } + + // The first copy of a manifest that reads non-empty wins; names match without case. + #[test] + fn read_jar_file_skips_empty_copies_and_matches_names_without_case() { + let (mut disc, udf) = open(dir( + "", + 10, + vec![], + vec![dir( + "BDMV", + 20, + vec![], + vec![dir( + "JAR", + 30, + vec![], + vec![ + dir( + "00000", + 40, + vec![file_with("playlists.xml", 100, 2000, Vec::new(), true)], + vec![], + ), + dir( + "00001", + 50, + vec![file_with( + "PLAYLISTS.XML", + 101, + 3000, + b"

".to_vec(), + true, + )], + vec![], + ), + ], + )], + )], + )); + let got = read_jar_file(&mut disc, &udf, "playlists.xml"); + assert_eq!(got.as_deref(), Some(&b"

"[..])); + } +} diff --git a/src/labels/mpls_universal.rs b/src/labels/mpls_universal.rs index 0bb3b5f5..6857f622 100644 --- a/src/labels/mpls_universal.rs +++ b/src/labels/mpls_universal.rs @@ -15,6 +15,10 @@ use crate::sector::SectorSource; use crate::udf::UdfFs; use std::collections::HashSet; +// Retained-label cap: a crafted image can carry tens of thousands of playlists, each +// naming distinct clips with a full STN table. +const MAX_MPLS_LABELS: usize = 4096; + /// True iff `/BDMV/PLAYLIST/` exists and contains at least one /// `.mpls` file. Cheap directory walk only — no sector reads. pub fn detect(_reader: &mut dyn SectorSource, udf: &UdfFs) -> bool { @@ -54,6 +58,9 @@ pub fn parse(reader: &mut dyn SectorSource, udf: &UdfFs) -> Option let mut labels: Vec = Vec::new(); let mut seen: HashSet = HashSet::new(); for name in &mpls_names { + if labels.len() >= MAX_MPLS_LABELS { + break; + } let path = format!("/BDMV/PLAYLIST/{}", name); let Ok(data) = udf.read_file(reader, &path) else { continue; @@ -123,6 +130,9 @@ fn add_playlist_labels( clip_id: clip_id.clone(), pid: entry.pid, }; + if labels.len() >= MAX_MPLS_LABELS { + return; + } if !seen.insert(stream_id.clone()) { continue; } @@ -156,16 +166,18 @@ fn label_type_for(entry: &crate::mpls::StreamEntry) -> Option { match entry.stream_type { 2 | 5 if entry.coding_type == c::PG => Some(StreamLabelType::Subtitle), 2 | 5 => Some(StreamLabelType::Audio), + // PiP PG is the secondary-video overlay, not a stream of the title. + 3 if entry.secondary => None, 3 => Some(StreamLabelType::Subtitle), _ => None, } } fn has_mpls_extension(name: &str) -> bool { - // Case-insensitive ".mpls" suffix (discs use both cases). UDF names are - // decoded via from_utf8_lossy, so a multi-byte replacement char can straddle - // byte index n-5; `ends_with` on a lowercased copy avoids that boundary panic. - name.len() >= 5 && name.to_ascii_lowercase().ends_with(".mpls") + // Case-insensitive ".mpls" suffix (discs use both cases). Compared on a lowercased + // copy, never by slicing: UDF names are lossily decoded, so byte n-5 may fall + // inside a multi-byte replacement char. + name.to_ascii_lowercase().ends_with(".mpls") } // Lowercase + trim the raw 3-char ISO 639-2 code; if vocab::lang maps @@ -365,6 +377,104 @@ mod tests { build_labels(playlists) } + // Minimal MPLS: one play item on `clip` whose STN lists one primary audio stream `pid`. + fn mpls_bytes(clip: &[u8; 5], pid: u16) -> Vec { + let mut item = Vec::new(); + item.extend_from_slice(clip); + item.extend_from_slice(b"M2TS"); + item.extend_from_slice(&[0u8; 3]); + item.extend_from_slice(&0u32.to_be_bytes()); + item.extend_from_slice(&(7000u32 * 45000).to_be_bytes()); + item.extend_from_slice(&[0u8; 12]); // UO mask, misc, still + item.extend_from_slice(&[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]); + item.extend_from_slice(&[3, 0x01]); + item.extend_from_slice(&pid.to_be_bytes()); + item.extend_from_slice(&[5, 0x81, 0x61]); + item.extend_from_slice(b"eng"); + let mut pl = vec![0u8; 6]; + pl.extend_from_slice(&1u16.to_be_bytes()); + pl.extend_from_slice(&[0u8; 2]); + pl.extend_from_slice(&(item.len() as u16).to_be_bytes()); + pl.extend_from_slice(&item); + let pl_len = (pl.len() - 4) as u32; + pl[0..4].copy_from_slice(&pl_len.to_be_bytes()); + let mut buf = b"MPLS0200".to_vec(); + buf.extend_from_slice(&40u32.to_be_bytes()); + buf.extend_from_slice(&[0u8; 28]); + buf.extend_from_slice(&pl); + buf + } + + // The same stream named by two playlists yields one label; an unparseable playlist + // between them is skipped, not fatal. + #[test] + fn parse_dedups_across_playlists_and_skips_a_bad_one() { + use crate::udf::fixture::*; + let files = vec![ + file_with("00800.mpls", 32, 8200, mpls_bytes(b"00001", 0x1100), false), + file_with("00801.mpls", 33, 8300, b"not an mpls".to_vec(), false), + file_with("00802.mpls", 34, 8400, mpls_bytes(b"00001", 0x1100), false), + ]; + let root = DirSpec { + name: String::new(), + icb_lba: 10, + dir_data_lba: 11, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "BDMV".to_string(), + icb_lba: 20, + dir_data_lba: 21, + files: Vec::new(), + subdirs: vec![DirSpec { + name: "PLAYLIST".to_string(), + icb_lba: 30, + dir_data_lba: 31, + files, + subdirs: vec![], + }], + }], + }; + let mut disc = MemDisc::new(); + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = crate::udf::read_filesystem(&mut disc).expect("fs"); + let got = parse(&mut disc, &udf).expect("labels"); + assert_eq!(got.labels.len(), 1); + } + + #[test] + fn playlist_without_play_items_yields_no_labels() { + let mut pl = playlist_with(vec![audio_entry(0x1100, 0x81, 6, 1, "eng")]); + pl.play_items.clear(); + assert!(labels_from_playlists(&[pl]).is_empty()); + } + + #[test] + fn pip_pg_streams_are_not_labelled_and_take_no_slot() { + let mut pip = pg_entry(0x1B00, "eng"); + pip.secondary = true; + let pl = playlist_with(vec![pg_entry(0x1200, "eng"), pip, pg_entry(0x1201, "fra")]); + let labels = labels_from_playlists(&[pl]); + let got: Vec<(u16, u16)> = labels + .iter() + .map(|l| (l.stream_id.as_ref().unwrap().pid, l.stream_number)) + .collect(); + assert_eq!(got, [(0x1200, 1), (0x1201, 2)]); + } + + #[test] + fn retained_labels_are_capped() { + let playlists: Vec = (0..MAX_MPLS_LABELS + 50) + .map(|i| { + playlist_on( + &format!("{i:05}"), + vec![audio_entry(0x1100, 0x81, 6, 1, "eng")], + ) + }) + .collect(); + assert_eq!(labels_from_playlists(&playlists).len(), MAX_MPLS_LABELS); + } + #[test] fn mpls_audio_streams_become_labels() { // Two audio streams: English TrueHD combo 48k, French AC-3 5.1 48k. diff --git a/src/labels/paramount.rs b/src/labels/paramount.rs index 307f49cd..028bfa69 100644 --- a/src/labels/paramount.rs +++ b/src/labels/paramount.rs @@ -342,6 +342,60 @@ fn pick<'a>(cands: &[Candidate<'a>], key: Key, require_audio: bool) -> Option<&' mod tests { use super::*; + fn hint_id(doc: &str) -> Option { + feature_hint(doc).and_then(|h| h.playlist_id) + } + + // Tier 1 ranks exact `Feature` names by duration, not document order. + #[test] + fn exact_feature_tier_prefers_the_longest_duration() { + let doc = r#" + + + "#; + assert_eq!(hint_id(doc), Some(2)); + } + + // A zero-audio exact `Feature` still wins over an audio-bearing extra: its id feeds the hint. + #[test] + fn zero_audio_exact_feature_is_still_selected() { + let doc = r#" + + + "#; + assert_eq!(hint_id(doc), Some(800)); + } + + // With equal audio counts, tier 3 breaks the tie by the longer duration. + #[test] + fn most_audio_tier_breaks_a_tie_by_longer_duration() { + let doc = r#" + + + "#; + assert_eq!(hint_id(doc), Some(2)); + } + + // Exactly the minimum feature length is a feature; one second less is a decoy. + #[test] + fn minimum_feature_length_is_inclusive() { + let doc = |secs: u64| format!(r#""#); + assert_eq!(hint_id(&doc(60)), Some(800)); + assert_eq!(hint_id(&doc(59)), None); + } + + // Every duration attribute spelling arms the sub-minute decoy guard. + #[test] + fn every_duration_attribute_spelling_is_honoured() { + for key in ["duration", "durs", "dur", "runtime", "length", "len"] { + let doc = format!( + r#" + "# + ); + assert_eq!(hint_id(&doc), Some(800), "{key}"); + } + } + // Immunity pin, section-boundary half. #[test] fn a_playlists_stream_list_cannot_run_into_the_next_playlist() { diff --git a/src/labels/pixelogic.rs b/src/labels/pixelogic.rs index 57063b1e..8e4c07b3 100644 --- a/src/labels/pixelogic.rs +++ b/src/labels/pixelogic.rs @@ -1186,4 +1186,53 @@ mod tests { assert!(!is_stream_token("Video Stream 1")); assert!(!is_stream_token("entry-markEa9")); } + + // A notice-clip name (`deu_Warning`) has the token shape: it is recorded as an + // uncatalogued component, yields no label, and still occupies its stream slot. + #[test] + fn notice_clip_token_is_recorded_unlabelled_and_takes_a_slot() { + let mut acc = UnknownParts::default(); + assert!(parse_token_inner("deu_Warning", Some(&mut acc)).is_none()); + assert_eq!(acc.seen.iter().collect::>(), ["WARNING"]); + + let mut acc = UnknownParts::default(); + let labels = assign_labels( + &strs(&["FPL_Main", "Audio Stream 1", "deu_Warning", "eng_MLP_"]), + &mut acc, + ); + assert_eq!(labels.len(), 1); + assert_eq!(labels[0].stream_number, 3); + assert_eq!(acc.total, 1); + } + + // Lay `bluray_project.bin` into /BDMV/JAR/00000 and run the real `parse`. + fn parse_blob(blob: &[u8]) -> Option { + use crate::udf::fixture::*; + let dir = |name: &str, icb, data, files, subdirs| DirSpec { + name: name.to_string(), + icb_lba: icb, + dir_data_lba: data, + files, + subdirs, + }; + let file = file_with("bluray_project.bin", 100, 2000, blob.to_vec(), true); + let sub = dir("00000", 54, 55, vec![file], vec![]); + let jar = dir("JAR", 52, 53, vec![], vec![sub]); + let bdmv = dir("BDMV", 12, 13, vec![], vec![jar]); + let root = dir("", 10, 11, vec![], vec![bdmv]); + let mut disc = MemDisc::new(); + build_udf_skeleton(&mut disc, 10); + lay_dir(&mut disc, &root); + let udf = crate::udf::read_filesystem(&mut disc).expect("fs"); + parse(&mut disc, &udf) + } + + // Confidence is High only when every token component was catalogued. + #[test] + fn parse_confidence_reflects_uncatalogued_components() { + let known = parse_blob(b"FPL_Main\0Audio Stream 1\0eng_MLP_\0").expect("labels"); + assert_eq!(known.confidence, Confidence::High); + let unknown = parse_blob(b"FPL_Main\0Audio Stream 1\0eng_MLP_NEWFLAG_\0").expect("labels"); + assert_eq!(unknown.confidence, Confidence::Medium); + } } diff --git a/src/labels/vocab.rs b/src/labels/vocab.rs index b6c88368..252c38fb 100644 --- a/src/labels/vocab.rs +++ b/src/labels/vocab.rs @@ -495,6 +495,72 @@ fn has_word(haystack: &str, needle: &str) -> bool { mod tests { use super::*; + // ISO 639-1 -> 639-2/T, checked against an independently written list so a mistyped or + // swapped row cannot pass on the table's own shape. + #[test] + fn iso639_1_values_match_the_standard() { + let want = [ + "aa:aar", "ab:abk", "ae:ave", "af:afr", "ak:aka", "am:amh", "an:arg", "ar:ara", + "as:asm", "av:ava", "ay:aym", "az:aze", "ba:bak", "be:bel", "bg:bul", "bh:bih", + "bi:bis", "bm:bam", "bn:ben", "bo:bod", "br:bre", "bs:bos", "ca:cat", "ce:che", + "ch:cha", "co:cos", "cr:cre", "cs:ces", "cu:chu", "cv:chv", "cy:cym", "da:dan", + "de:deu", "dv:div", "dz:dzo", "ee:ewe", "el:ell", "en:eng", "eo:epo", "es:spa", + "et:est", "eu:eus", "fa:fas", "ff:ful", "fi:fin", "fj:fij", "fo:fao", "fr:fra", + "fy:fry", "ga:gle", "gd:gla", "gl:glg", "gn:grn", "gu:guj", "gv:glv", "ha:hau", + "he:heb", "hi:hin", "ho:hmo", "hr:hrv", "ht:hat", "hu:hun", "hy:hye", "hz:her", + "ia:ina", "id:ind", "ie:ile", "ig:ibo", "ii:iii", "ik:ipk", "io:ido", "is:isl", + "it:ita", "iu:iku", "ja:jpn", "jv:jav", "ka:kat", "kg:kon", "ki:kik", "kj:kua", + "kk:kaz", "kl:kal", "km:khm", "kn:kan", "ko:kor", "kr:kau", "ks:kas", "ku:kur", + "kv:kom", "kw:cor", "ky:kir", "la:lat", "lb:ltz", "lg:lug", "li:lim", "ln:lin", + "lo:lao", "lt:lit", "lu:lub", "lv:lav", "mg:mlg", "mh:mah", "mi:mri", "mk:mkd", + "ml:mal", "mn:mon", "mr:mar", "ms:msa", "mt:mlt", "my:mya", "na:nau", "nb:nob", + "nd:nde", "ne:nep", "ng:ndo", "nl:nld", "nn:nno", "no:nor", "nr:nbl", "nv:nav", + "ny:nya", "oc:oci", "oj:oji", "om:orm", "or:ori", "os:oss", "pa:pan", "pi:pli", + "pl:pol", "ps:pus", "pt:por", "qu:que", "rm:roh", "rn:run", "ro:ron", "ru:rus", + "rw:kin", "sa:san", "sc:srd", "sd:snd", "se:sme", "sg:sag", "si:sin", "sk:slk", + "sl:slv", "sm:smo", "sn:sna", "so:som", "sq:sqi", "sr:srp", "ss:ssw", "st:sot", + "su:sun", "sv:swe", "sw:swa", "ta:tam", "te:tel", "tg:tgk", "th:tha", "ti:tir", + "tk:tuk", "tl:tgl", "tn:tsn", "to:ton", "tr:tur", "ts:tso", "tt:tat", "tw:twi", + "ty:tah", "ug:uig", "uk:ukr", "ur:urd", "uz:uzb", "ve:ven", "vi:vie", "vo:vol", + "wa:wln", "wo:wol", "xh:xho", "yi:yid", "yo:yor", "za:zha", "zh:zho", "zu:zul", + ]; + assert_eq!(want.len(), 184); + for pair in want { + let (two, three) = pair.split_once(':').unwrap(); + assert_eq!(iso639_1_to_iso639_2(two), Some(three), "{two}"); + } + } + + #[test] + fn iso639_1_withdrawn_dvd_codes_resolve_to_their_replacements() { + for (old, three) in [ + ("iw", "heb"), + ("in", "ind"), + ("ji", "yid"), + ("jw", "jav"), + ("mo", "ron"), + ] { + assert_eq!(iso639_1_to_iso639_2(old), Some(three), "{old}"); + } + } + + #[test] + fn mandarin_chinese_compound_keeps_its_variant() { + assert_eq!( + lang("Mandarin Chinese"), + Some(LangInfo { + code: "zho", + variant: "Mandarin" + }) + ); + } + + #[test] + fn description_word_marks_descriptive_purpose() { + assert_eq!(purpose("English Description"), LabelPurpose::Descriptive); + assert_eq!(purpose("Descriptions"), LabelPurpose::Normal); + } + #[test] fn iso639_1_accepts_jw_and_mo_dvd_aliases() { assert_eq!(iso639_1_to_iso639_2("jw"), Some("jav")); diff --git a/src/mpls.rs b/src/mpls.rs index 1165e5bc..ea7eb80a 100644 --- a/src/mpls.rs +++ b/src/mpls.rs @@ -30,8 +30,8 @@ pub(crate) struct PlaylistMark { /// Chapter filters should test `== 1`, not `<= 1`. pub mark_type: u8, /// Which play item this mark belongs to. Carries the per-PlayItem - /// timebase needed to place a mark in a multi-PlayItem playlist; - /// the chapter builder does not consume it yet. + /// timebase needed to place a mark in a multi-PlayItem playlist; the + /// chapter builder resolves each mark against that PlayItem. pub play_item_ref: u16, /// Timestamp in 45kHz PTS ticks pub timestamp: u32, @@ -141,6 +141,12 @@ pub fn parse(data: &[u8]) -> Result { let item = &pl[pos + 2..pos + 2 + item_length]; if item.len() < 20 { + tracing::warn!( + target: "freemkv::mpls", + item_idx, + item_length, + "play item too short to read; it is dropped from the playlist" + ); pos += 2 + item_length; continue; } @@ -184,6 +190,7 @@ pub fn parse(data: &[u8]) -> Result { streams.push(entry); spos = next; } else { + stn_entry_unreadable("primary video"); break; } } @@ -193,6 +200,7 @@ pub fn parse(data: &[u8]) -> Result { streams.push(entry); spos = next; } else { + stn_entry_unreadable("primary audio"); break; } } @@ -206,6 +214,7 @@ pub fn parse(data: &[u8]) -> Result { streams.push(entry); spos = next; } else { + stn_entry_unreadable("PG subtitle"); break; } } @@ -214,6 +223,7 @@ pub fn parse(data: &[u8]) -> Result { if let Some((_, next)) = parse_stream_entry(item, spos, STREAM_CATEGORY_IG) { spos = next; } else { + stn_entry_unreadable("IG"); break; } } @@ -233,6 +243,7 @@ pub fn parse(data: &[u8]) -> Result { spos = next; } } else { + stn_entry_unreadable("secondary audio"); break; } } @@ -260,6 +271,7 @@ pub fn parse(data: &[u8]) -> Result { spos = next; } } else { + stn_entry_unreadable("secondary video"); break; } } @@ -273,6 +285,7 @@ pub fn parse(data: &[u8]) -> Result { streams.push(entry); spos = next; } else { + stn_entry_unreadable("Dolby Vision enhancement layer"); break; } } @@ -336,6 +349,15 @@ pub fn parse(data: &[u8]) -> Result { }) } +// An unreadable stream entry ends its STN category and every one after it. +fn stn_entry_unreadable(kind: &str) { + tracing::warn!( + target: "freemkv::mpls", + kind, + "unreadable STN stream entry; this and the later stream entries are dropped" + ); +} + impl Playlist { /// Total play-item running time in 45 kHz ticks (out_time - in_time, summed). pub fn duration_ticks(&self) -> u64 { @@ -380,7 +402,7 @@ fn parse_stream_entry(item: &[u8], pos: usize, stream_type: u8) -> Option<(Strea // PID location depends on stream-entry type (BD spec stream_entry()): type 1 // (PlayItem's Clip) → +2; type 2 (SubPath SubClip) → +4; type 3/4 (SubPath - // clip / DV enhancement layer) → +3. Previously only type 1 was handled. + // clip / DV enhancement layer) → +3. let pid_off = match item[pos + 1] { STREAM_ENTRY_PLAYITEM_CLIP => 2, STREAM_ENTRY_SUBPATH_SUBCLIP => 4, diff --git a/src/mux/codec/ac3.rs b/src/mux/codec/ac3.rs index 5f66e183..add51f5d 100644 --- a/src/mux/codec/ac3.rs +++ b/src/mux/codec/ac3.rs @@ -492,9 +492,9 @@ impl CodecParser for Ac3Parser { ); self.acc.clear(); self.held = None; - // Advance the cadence, as the other two paths out of this block do, - // so all three cannot disagree. Defensive: no input parsing frames - // and leaving a residue this large is currently reachable. + // Advance the cadence like the other two paths out of this block. + // No known input reaches this branch; it must still drop the held + // AU and advance, or a stale HeldAu would resume after the resync. self.flush_pts_ns = frame_pts_ns; } else { self.acc.drain(keep_from); diff --git a/src/mux/codec/adts.rs b/src/mux/codec/adts.rs index 25459165..a7121af9 100644 --- a/src/mux/codec/adts.rs +++ b/src/mux/codec/adts.rs @@ -16,10 +16,10 @@ const ADTS_SAMPLE_RATE_VALID: [u32; 16] = [ 0, ]; -/// ADTS header verdict for the packet head. -// Length of the fixed ADTS header without CRC. +/// Length of the fixed ADTS header without CRC. const ADTS_HEADER_BYTES: usize = 7; +/// ADTS header verdict for the packet head. enum AdtsVerdict { /// No 12-bit ADTS sync at the head — not an ADTS frame we can validate. NoSync, @@ -474,6 +474,119 @@ mod tests { assert_eq!(&asc[2..], &[0x05, 0x04, 0x00, 0x00, 0x20, 0x00]); } + fn bits_to_bytes(bits: &str) -> Vec { + let bits: String = bits.chars().filter(|c| !c.is_whitespace()).collect(); + assert_eq!(bits.len() % 8, 0, "byte-aligned"); + (0..bits.len() / 8) + .map(|i| u8::from_str_radix(&bits[i * 8..i * 8 + 8], 2).unwrap()) + .collect() + } + + fn pad8(bits: &str) -> String { + let bits: String = bits.chars().filter(|c| !c.is_whitespace()).collect(); + let pad = (8 - bits.len() % 8) % 8; + bits + &"0".repeat(pad) + } + + // An ADTS frame with channel_configuration 0 whose payload is `block`. + fn config0_frame(block: &[u8]) -> Vec { + let mut f = adts_frame(0); + f.extend_from_slice(block); + let total = f.len() as u32; + f[2] &= !1; + f[3] = (f[3] & 0x3C) | ((total >> 11) & 3) as u8; + f[4] = (total >> 3) as u8; + f[5] = (((total & 7) << 5) as u8) | 0x1F; + f + } + + // 5.1: 2 front, 1 back, 1 LFE, all three mixdown fields, a 3-byte comment. + const PCE_51_FIELDS: &str = "0000 01 0011 0010 0000 0001 01 000 0000 \ + 1 0001 1 0010 1 101 \ + 01000 10000 10001 0000"; + const PCE_51_COMMENT: &str = "00000011 10101010 01010101 11110000"; + + fn pce_51_block() -> Vec { + bits_to_bytes(&(pad8(&format!("101{PCE_51_FIELDS}")) + &PCE_51_COMMENT.replace(' ', ""))) + } + + #[test] + fn adts_pce_repacks_every_field_to_the_asc_start() { + let expect = bits_to_bytes(&(pad8(PCE_51_FIELDS) + &PCE_51_COMMENT.replace(' ', ""))); + assert_eq!(adts_pce(&pce_51_block()).unwrap(), expect); + } + + #[test] + fn adts_pce_without_mixdown_fields_and_with_assoc_and_cc_elements() { + // 1 front, 1 assoc data, 2 coupling channels: 5*(1+2) + 4*1 element bits. + let fields = "0000 01 0100 0001 0000 0000 00 001 0010 0 0 0 \ + 10000 \ + 0000 \ + 00001 10001"; + let block = bits_to_bytes(&(pad8(&format!("101{fields}")) + "00000000")); + let expect = bits_to_bytes(&(pad8(fields) + "00000000")); + assert_eq!(adts_pce(&block).unwrap(), expect); + } + + #[test] + fn adts_pce_truncated_comment_or_wrong_id_is_none() { + let block = pce_51_block(); + assert_eq!(adts_pce(&block[..block.len() - 1]), None); + let mut not_pce = block; + not_pce[0] ^= 0x80; + assert_eq!(adts_pce(¬_pce), None); + } + + #[test] + fn channel_config_0_asc_follows_the_pce_through_truncation_upgrade_and_change() { + let (mut config, mut changes) = (None, 0); + let block = pce_51_block(); + // A PCE cut off by the frame end leaves the 2-byte ASC. + let cut = config0_frame(&block[..4]); + adts_header(&cut, &mut config, &mut changes, 0).unwrap(); + assert_eq!(config.as_deref().map(<[u8]>::len), Some(2)); + // The first frame that carries the whole PCE upgrades it; not a change. + let full = config0_frame(&block); + adts_header(&full, &mut config, &mut changes, 0).unwrap(); + let asc = config.clone().unwrap(); + assert_eq!(&asc[2..], adts_pce(&block).unwrap()); + assert_eq!(changes, 0); + // A different layout is counted and the first config is kept. + let mut other = block; + other[1] ^= 0x10; + adts_header(&config0_frame(&other), &mut config, &mut changes, 0).unwrap(); + assert_eq!(changes, 1); + assert_eq!(config, Some(asc)); + } + + #[test] + fn adts_template_encodes_the_channel_configuration_across_both_header_bytes() { + // AAC-LC, 48 kHz: ASC byte 0 = 0x11, byte 1 = (rate low bit << 7) | (channels << 3). + for channels in 1u8..=7 { + let t = adts_template(&[0x11, 0x80 | (channels << 3)]).unwrap(); + assert_eq!( + t[2], + 0x40 | (3 << 2) | (channels >> 2), + "channels {channels}" + ); + assert_eq!(t[3], (channels & 3) << 6, "channels {channels}"); + } + assert_eq!(adts_template(&[0x11, 0x80]), None, "0 needs an in-band PCE"); + assert_eq!(adts_template(&[0x11, 0x80 | (8 << 3)]), None); + } + + #[test] + fn adts_frame_length_field_holds_13_bits_and_blocks_are_1_to_4() { + let t = adts_template(&[0x11, 0x90]).unwrap(); + let max = super::adts_frame(t, &vec![0u8; (1 << 13) - 1 - 7], 1).unwrap(); + assert_eq!(max.len(), (1 << 13) - 1); + assert_eq!(adts_frame_len(&max), Some((1 << 13) - 1)); + assert_eq!(super::adts_frame(t, &vec![0u8; (1 << 13) - 7], 1), None); + assert_eq!(super::adts_frame(t, &[0; 4], 0), None); + assert_eq!(super::adts_frame(t, &[0; 4], 5), None); + assert_eq!(super::adts_frame(t, &[0; 4], 4).unwrap()[6] & 3, 3); + } + #[test] fn valid_adts_is_kept() { let mut p = AdtsParser::new(); diff --git a/src/mux/codec/audio_frames.rs b/src/mux/codec/audio_frames.rs index b997fe48..12253ab8 100644 --- a/src/mux/codec/audio_frames.rs +++ b/src/mux/codec/audio_frames.rs @@ -22,6 +22,8 @@ use super::{Frame, PesPacket}; // Most bytes held after a lock while waiting for the next timestamp (see place_lock). const MAX_HOLD: usize = 64 * 1024; +// Most bytes buffered across packets before the stream is judged corrupt and the packet dropped. +const MAX_BUFFERED: usize = 1024 * 1024; pub(super) struct Header { pub bytes: usize, @@ -182,7 +184,7 @@ impl AudioFrames { self.framed = true; // Audio PES packets are small. Refuse pathological accumulation before // copying; a corrupt size must not grow memory without bound. - if self.buf.len().saturating_add(pes.data.len()) > 1024 * 1024 { + if self.buf.len().saturating_add(pes.data.len()) > MAX_BUFFERED { self.buf.clear(); self.tally.record_drop( facts.presentation_ns().unwrap_or(self.next_pts), diff --git a/src/mux/codec/dropgate.rs b/src/mux/codec/dropgate.rs index 8a342245..7a92693a 100644 --- a/src/mux/codec/dropgate.rs +++ b/src/mux/codec/dropgate.rs @@ -168,6 +168,15 @@ mod tests { assert!(!t.is_poisoned()); } + #[test] + fn a_negative_duration_is_counted_as_zero_lost_time() { + let mut t = DropTally::new("test"); + t.record_drop(0, -5_000, 512, "bad"); + t.record_collateral_drop(0, -1, 512, "resync-forward"); + assert_eq!(t.dropped_frames(), 2); + assert_eq!(t.dropped_duration_ns(), 0); + } + #[test] fn poisons_after_min_aus_over_half_dropped() { let mut t = DropTally::new("test"); diff --git a/src/mux/codec/dts.rs b/src/mux/codec/dts.rs index ec723708..31287a3f 100644 --- a/src/mux/codec/dts.rs +++ b/src/mux/codec/dts.rs @@ -23,10 +23,12 @@ pub struct DtsParser { /// its FIRST byte, so an AU whose core arrived in an earlier PES keeps that /// core's timestamp and source offset when its extensions arrive later. acc: super::pesbuf::PesBuf, - /// PTS of the access unit currently being assembled in `buf` (the unit + /// PTS of the access unit currently being assembled in `acc` (the unit /// starting at the first buffered core sync). Captured when that core /// frame's PES first arrived; the trailing extension-substream PES /// packets carry their own (later) PTS which must NOT override it. + /// `PTS_UNSET` after a gap or a forced flush; `front_pts` falls back to it + /// only when the front packet carries no timestamp. pending_pts: i64, /// The `front_pts` of the PREVIOUS emitted access unit. When the current /// AU's `front_pts` differs, it began a new PES → re-base to it. When it is @@ -73,8 +75,8 @@ impl DtsParser { } } - /// Number of access units dropped as undecodable so far. The mux/CLI reads - /// this to surface the count ("dropped N damaged DTS frames"). + /// Number of access units dropped as undecodable so far. Only the tally's + /// end-of-stream summary logs it; no mux or CLI consumer reads this accessor. pub fn dropped_frames(&self) -> u64 { self.tally.dropped_frames() } @@ -2450,6 +2452,66 @@ mod tests { assert_eq!(core_header_drop_reason(&d), Some(DropReason::PcmRes)); } + // CPF=1 inserts a 16-bit header CRC before the PCM resolution; the gate must skip exactly + // those bits. PCMR sits at absolute bits 95..=97 without the CRC, 111..=113 with it. + #[test] + fn a_crc_protected_core_header_reads_pcmr_after_the_16_crc_bits() { + let mut crc = make_dts_core(512); + crc[4] |= 0x02; // CPF + // The CRC bits are arbitrary, and set to a reserved code where PCMR sits without it. + crc[11] |= 0x01; + crc[12] = 0xFF; + crc[13] |= 0xFE; + assert_eq!(core_header_drop_reason(&crc), None); + + // Decodable PCMR 3 whose trailing neighbours would read as reserved code 4 if the + // skip ran two bits long. + let mut d = make_dts_core(512); + d[4] |= 0x02; + d[14] |= 0xC0; + assert_eq!(core_header_drop_reason(&d), None); + + // Reserved PCMR 7 is found at the shifted position. + let mut d = make_dts_core(512); + d[4] |= 0x02; + d[13] |= 0x01; + d[14] |= 0xC0; + assert_eq!(core_header_drop_reason(&d), Some(DropReason::PcmRes)); + } + + // Substreams over 64 KiB use the long header form (12-bit header size, 20-bit Fsize). + fn long_form_exss(total: usize, static_fields: Option<(&str, &str)>) -> Vec { + let mut bits = format!( + "{}{}1{}{:020b}", + "0".repeat(8), + "00", + "0".repeat(12), + total - 1 + ); + if let Some((rate, dur)) = static_fields { + bits.push_str(&format!("1{rate}{dur}")); + } + while bits.len() % 8 != 0 { + bits.push('0'); + } + let mut d = vec![0u8; total.max(4 + bits.len() / 8)]; + d[0..4].copy_from_slice(&DTS_HD_EXT_SYNC); + for (i, chunk) in bits.as_bytes().chunks(8).enumerate() { + d[4 + i] = u8::from_str_radix(std::str::from_utf8(chunk).unwrap(), 2).unwrap(); + } + d + } + + #[test] + fn exss_long_header_form_sizes_and_times_substreams_over_64_kib() { + let d = long_form_exss(70_000, Some(("10", "001"))); + assert_eq!(exss_frame_size(&d), Some(70_000)); + assert_eq!(exss_timing(&d), Some((1024, 48_000))); + let d = long_form_exss(1_000_000, None); + assert_eq!(exss_frame_size(&d), Some(1_000_000)); + assert_eq!(exss_timing(&d), None); + } + #[test] fn legal_multichannel_amode_is_not_dropped() { // ETSI TS 102 114 §5.3.1: AMODE is 6 bits with 16 LEGAL codes (0-15), only diff --git a/src/mux/codec/dvdsub.rs b/src/mux/codec/dvdsub.rs index 575b5a2d..ab54f932 100644 --- a/src/mux/codec/dvdsub.rs +++ b/src/mux/codec/dvdsub.rs @@ -45,7 +45,15 @@ impl DvdSubParser { // `complete` is only true when `self.pending` was `Some` above, but // borrow through `?` rather than a prod `unwrap()` on a fact the // compiler can't see across the two lines. - let (facts, _, data) = self.pending.take()?; + let (facts, size, data) = self.pending.take()?; + if data.len() < size { + tracing::warn!( + target: "mux", + have = data.len(), + declared = size, + "dvdsub: emitting a truncated subpicture unit" + ); + } // Guaranteed Some: `pending` is only ever created from a PES with // `pts.is_some()` (L054 dropped the no-PTS orphan path), so this // never takes the fallback; kept as unwrap_or, never a prod unwrap. @@ -165,7 +173,6 @@ impl CodecParser for DvdSubParser { // ── YCbCr → RGB conversion and palette formatting ───────────────────────── -// Byte-2-as-Cb is invisible on achromatic (Cb=Cr=128) palette entries. /// Convert a single YCbCr color to RGB, clamping to [0, 255]. /// /// Input: `[padding, Y, Cr, Cb]` (as stored in DVD IFO PGC data). Note the @@ -174,15 +181,17 @@ impl CodecParser for DvdSubParser { /// /// Returns `[R, G, B]`. /// -/// Uses full-range (JFIF) BT.601 coefficients, matching the VobSub `.idx` on-disk convention. +/// The DVD CLUT is studio-range (Y 16..=235, chroma 16..=240), so this applies +/// the limited-range BT.601 expansion: Y=16 maps to black, Y=235 to white. The +/// result is the full-range RGB a VobSub `.idx` palette carries. pub fn ycbcr_to_rgb(color: &[u8; 4]) -> [u8; 3] { - let y = color[1] as f64; - let cr = color[2] as f64; - let cb = color[3] as f64; + let y = 1.164 * (color[1] as f64 - 16.0); + let cr = color[2] as f64 - 128.0; + let cb = color[3] as f64 - 128.0; - let r = y + 1.402 * (cr - 128.0); - let g = y - 0.344 * (cb - 128.0) - 0.714 * (cr - 128.0); - let b = y + 1.772 * (cb - 128.0); + let r = y + 1.596 * cr; + let g = y - 0.392 * cb - 0.813 * cr; + let b = y + 2.017 * cb; [clamp_u8(r), clamp_u8(g), clamp_u8(b)] } @@ -424,22 +433,16 @@ mod tests { #[test] fn ycbcr_to_rgb_white() { - // White in YCbCr: Y=235, Cb=128, Cr=128 → R=235, G=235, B=235 + // Studio-range white: Y=235 at neutral chroma expands to full white. let color = [0x00, 235, 128, 128]; - let [r, g, b] = ycbcr_to_rgb(&color); - assert_eq!(r, 235); - assert_eq!(g, 235); - assert_eq!(b, 235); + assert_eq!(ycbcr_to_rgb(&color), [255, 255, 255]); } #[test] fn ycbcr_to_rgb_black() { - // Black: Y=16, Cb=128, Cr=128 → R=16, G=16, B=16 + // Studio-range black: Y=16 at neutral chroma is true black. let color = [0x00, 16, 128, 128]; - let [r, g, b] = ycbcr_to_rgb(&color); - assert_eq!(r, 16); - assert_eq!(g, 16); - assert_eq!(b, 16); + assert_eq!(ycbcr_to_rgb(&color), [0, 0, 0]); } #[test] @@ -465,9 +468,6 @@ mod tests { // Approximate red: Y=82, Cr=240, Cb=90 — on disc as [pad, Y, Cr, Cb]. let color = [0x00, 82, 240, 90]; let [r, g, b] = ycbcr_to_rgb(&color); - // R = 82 + 1.402*(240-128) = 82 + 156.9 ≈ 239 - // G = 82 - 0.344*(90-128) - 0.714*(240-128) = 82 + 13.1 - 79.97 ≈ 15 - // B = 82 + 1.772*(90-128) = 82 - 67.3 ≈ 15 assert!(r > 200, "R should be high for red, got {}", r); assert!(g < 30, "G should be low for red, got {}", g); assert!(b < 30, "B should be low for red, got {}", b); @@ -486,7 +486,7 @@ mod tests { "on-disc red [0,Y=76,Cr=255,Cb=85] must render red-dominant, \ got R={r} G={g} B={b} (R and B swapped => byte 2/3 are transposed)" ); - assert_eq!([r, g, b], [254, 0, 0], "exact full-range BT.601 red"); + assert_eq!([r, g, b], [255, 0, 0], "exact studio-range BT.601 red"); // And the converse: a saturated BLUE on-disc entry (Y=29, Cr=107, Cb=255) // must not come out red. @@ -545,8 +545,8 @@ mod tests { #[test] fn format_palette_hex_format() { - // Y=128, Cb=128, Cr=128 → R=128, G=128, B=128 → "808080" - let palette = vec![[0x00, 128, 128, 128]]; + // Y=126 at neutral chroma → R=G=B=128 → "808080" + let palette = vec![[0x00, 126, 128, 128]]; let result = format_palette(&palette, 0, 0); let text = String::from_utf8(result).unwrap(); assert_eq!(text, "palette: 808080\n"); @@ -556,7 +556,7 @@ mod tests { fn format_palette_emits_size_line_before_palette() { // With non-zero dimensions the `.idx` `size:` line is prepended ahead of // the palette so players place/scale the VobSub bitmap (PAL 720x576). - let palette = vec![[0x00, 128, 128, 128]]; + let palette = vec![[0x00, 126, 128, 128]]; let result = format_palette(&palette, 720, 576); let text = String::from_utf8(result).unwrap(); assert_eq!( @@ -569,7 +569,7 @@ mod tests { fn format_palette_omits_size_line_when_dimensions_unknown() { // 0 width/height (unknown resolution) omits the size line rather than // emitting a 0x0 frame; the palette line is still present. - let palette = vec![[0x00, 128, 128, 128]]; + let palette = vec![[0x00, 126, 128, 128]]; let result = format_palette(&palette, 0, 576); let text = String::from_utf8(result).unwrap(); assert_eq!(text, "palette: 808080\n", "no size line when a dim is 0"); @@ -717,12 +717,21 @@ mod tests { #[test] fn ycbcr_blue_channel_clamps_high() { - // B = Y + 1.772*(Cb-128). Y=128, Cb=255 → 128 + 1.772*127 ≈ 353 → clamp 255. + // B = 1.164*(Y-16) + 2.017*(Cb-128) ≈ 130 + 256 → clamp 255. // Cb is byte 3 in the on-disc [pad, Y, Cr, Cb] layout. let [_r, _g, b] = ycbcr_to_rgb(&[0x00, 128, 128, 255]); assert_eq!(b, 255, "blue clamps at 255"); } + #[test] + fn ycbcr_to_rgb_green_uses_both_chroma_terms() { + // Y=128, Cr=100, Cb=90: G = 1.164*112 - 0.392*(-38) - 0.813*(-28) = 167.9. + assert_eq!(ycbcr_to_rgb(&[0x00, 128, 100, 90])[1], 168); + // Each chroma term alone: Cr moves G by -0.813/step, Cb by -0.392/step. + assert_eq!(ycbcr_to_rgb(&[0x00, 128, 100, 128])[1], 153); + assert_eq!(ycbcr_to_rgb(&[0x00, 128, 128, 90])[1], 145); + } + #[test] fn format_palette_empty_is_just_prefix() { // An empty palette yields "palette: \n" (prefix + newline, no entries). @@ -733,9 +742,9 @@ mod tests { #[test] fn format_palette_pads_each_channel_to_two_hex_digits() { // Each RGB channel is formatted as exactly 2 hex digits (zero-padded). - // Y=16,neutral → 0x10 → "101010" (each channel two digits). - let result = format_palette(&[[0x00, 16, 128, 128]], 0, 0); - assert_eq!(String::from_utf8(result).unwrap(), "palette: 101010\n"); + // Y=20,neutral → 5 → "050505" (each channel two digits). + let result = format_palette(&[[0x00, 20, 128, 128]], 0, 0); + assert_eq!(String::from_utf8(result).unwrap(), "palette: 050505\n"); } // Text guard in codec/mod.rs can't see `source: facts.source` with no diff --git a/src/mux/codec/flac.rs b/src/mux/codec/flac.rs index 64796faf..3a7ed143 100644 --- a/src/mux/codec/flac.rs +++ b/src/mux/codec/flac.rs @@ -252,6 +252,33 @@ mod tests { assert_eq!(p.dropped_frames(), before + 1); } + // The blocking-strategy bit (0xFFF9 = variable blocksize) is masked off the sync test. + #[test] + fn variable_blocksize_frames_are_validated_too() { + let mut p = FlacParser::new(); + let mut good = make_flac_frame(100); + good[1] = 0xF9; + let n = good.len(); + let c = crc16_ansi(&good[..n - 2]); + good[n - 2] = (c >> 8) as u8; + good[n - 1] = c as u8; + assert_eq!(p.parse(&make_pes(good.clone(), Some(0))).len(), 1); + let mut bad = good; + bad[20] ^= 0xFF; + assert!(p.parse(&make_pes(bad, Some(90_000))).is_empty()); + assert_eq!(p.dropped_frames(), 1); + } + + #[test] + fn a_set_reserved_bit_is_not_a_flac_sync() { + let mut p = FlacParser::new(); + let mut f = make_flac_frame(100); + f[1] = 0xFA; // the mandatory-0 reserved bit set + f[20] ^= 0xFF; + assert_eq!(p.parse(&make_pes(f, Some(0))).len(), 1, "passed through"); + assert_eq!(p.dropped_frames(), 0); + } + #[test] fn non_flac_packet_passes_through() { // A packet without the FLAC sync isn't a frame we can validate — never diff --git a/src/mux/codec/h264.rs b/src/mux/codec/h264.rs index 1800cecf..a8423e22 100644 --- a/src/mux/codec/h264.rs +++ b/src/mux/codec/h264.rs @@ -212,9 +212,11 @@ impl CodecParser for H264Parser { let mut all_vcl_intra = true; let mut saw_vcl = false; let mut first_pic_done = false; - // Did this access unit already carry each param-set type in-band? - let mut emitted_sps = false; - let mut emitted_pps = false; + // In-band redefinitions, held aside so they lead the AU in SPS, PPS order: + // a PPS is parsed against the SPS before it, so a re-asserted PPS must + // never precede a redefined SPS. + let mut inband_sps = Vec::new(); + let mut inband_pps = Vec::new(); // Pre-sized to input length plus PARAM_REASSERT_HEADROOM so the mux hot // path avoids repeated reallocs and the keyframe param-set re-assert // below can be spliced in front without reallocating (see that site). @@ -232,12 +234,10 @@ impl CodecParser for H264Parser { // in-band on any change. In MVC passthrough these fall through to // the default arm so subset SPS/PPS stay in-band (self-contained AU). NAL_SPS if !mvc => { - emitted_sps |= - handle_param_set(&mut self.sps, &mut self.cur_sps, nal, &mut frame_data) + handle_param_set(&mut self.sps, &mut self.cur_sps, nal, &mut inband_sps); } NAL_PPS if !mvc => { - emitted_pps |= - handle_param_set(&mut self.pps, &mut self.cur_pps, nal, &mut frame_data) + handle_param_set(&mut self.pps, &mut self.cur_pps, nal, &mut inband_pps); } // Access unit delimiters: drop. Matroska H.264 frame data omits // AUDs (the container delimits access units), so keeping them @@ -278,7 +278,7 @@ impl CodecParser for H264Parser { } } - if frame_data.is_empty() { + if frame_data.is_empty() && inband_sps.is_empty() && inband_pps.is_empty() { return Vec::new(); } @@ -291,11 +291,17 @@ impl CodecParser for H264Parser { // Every keyframe is self-contained: re-assert active SPS/PPS in-band so a // decoder that dropped the set recovers, and a stale avcC re-apply can't - // revert it. Skipped per-type only when this AU already carried it. - if keyframe && !mvc { + // revert it. A type this AU redefined goes in-band instead, in its place. + { + let reassert = keyframe && !mvc; let mut prefix = Vec::with_capacity(PARAM_REASSERT_HEADROOM); - reassert_active(&mut prefix, &self.cur_sps, emitted_sps); - reassert_active(&mut prefix, &self.cur_pps, emitted_pps); + for (inband, cur) in [(&inband_sps, &self.cur_sps), (&inband_pps, &self.cur_pps)] { + if !inband.is_empty() { + prefix.extend_from_slice(inband); + } else if reassert { + reassert_active(&mut prefix, cur, false); + } + } if !prefix.is_empty() { // Splice prefix into frame_data in place, sized via // PARAM_REASSERT_HEADROOM to avoid the extra whole-frame alloc+copy @@ -1744,6 +1750,114 @@ mod tests { types } + // The parameter sets a later bare keyframe re-asserts are the ACTIVE ones, not the first-seen + // avcC copy: a player re-applies avcC at every keyframe, so reverting would undo the change. + #[test] + fn a_bare_keyframe_reasserts_the_active_param_sets_not_the_avcc_copy() { + let nals = |fd: &[u8]| -> Vec> { + let mut out = Vec::new(); + let mut off = 0; + while off + 4 <= fd.len() { + let len = + u32::from_be_bytes([fd[off], fd[off + 1], fd[off + 2], fd[off + 3]]) as usize; + out.push(fd[off + 4..off + 4 + len].to_vec()); + off += 4 + len; + } + out + }; + let au = |sps: Option, pps: Option, slice: &[u8]| { + let mut d = Vec::new(); + if let Some(b) = sps { + d.extend_from_slice(&[0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x1E, b]); + } + if let Some(b) = pps { + d.extend_from_slice(&[0x00, 0x00, 0x01, 0x68, b]); + } + d.extend_from_slice(&[0x00, 0x00, 0x01]); + d.extend_from_slice(slice); + d + }; + let mut parser = H264Parser::new(); + parser.parse(&make_pes( + au(Some(0xAA), Some(0x11), &[0x65, 0x10]), + Some(0), + )); + // Mid-title redefinition of both sets, on a non-keyframe access unit. + let f = parser.parse(&make_pes( + au(Some(0xBB), Some(0x22), &[0x41, 0x9A]), + Some(3000), + )); + assert_eq!(frame_nal_types(&f[0].data), vec![7, 8, 1]); + // A bare IDR: SPS then PPS, both the redefined bodies. + let f = parser.parse(&make_pes(au(None, None, &[0x65, 0x10]), Some(6000))); + let n = nals(&f[0].data); + assert_eq!(frame_nal_types(&f[0].data), vec![7, 8, 5]); + assert_eq!(n[0].last(), Some(&0xBB), "active SPS, not the avcC copy"); + assert_eq!(n[1].last(), Some(&0x22), "active PPS, not the avcC copy"); + } + + #[test] + fn slice_types_map_to_coding_types() { + use CodingType::{B, I, P}; + let expect = [P, B, I, P, I, P, B, I, P, I]; + for (st, want) in expect.into_iter().enumerate() { + assert_eq!(h264_slice_coding_type(st as u32), Some(want), "{st}"); + } + assert_eq!(h264_slice_coding_type(10), None); + assert_eq!(h264_slice_coding_type(u32::MAX), None); + } + + #[test] + fn avcc_oversized_pps_returns_none() { + let mut data = vec![0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x1E, 0xAA]; + data.extend_from_slice(&[0x00, 0x00, 0x01, 0x68]); + data.extend_from_slice(&vec![0x11u8; 70_000]); + let mut parser = H264Parser::new(); + parser.parse(&make_pes(data, Some(0))); + assert!( + parser.codec_private().is_none(), + "oversized PPS must not produce a truncated avcC" + ); + } + + // ISO 14496-15 §5.3.3.1.2: every profile whose SPS carries chroma_format_idc and bit depths + // gets the 4 avcC extension bytes; Baseline/Main/Extended must not. + #[test] + fn avcc_extension_bytes_follow_exactly_the_chroma_carrying_profiles() { + for profile in [ + 100u8, 110, 122, 144, 244, 44, 83, 86, 118, 128, 138, 139, 134, 135, + ] { + let sps = build_high_profile_sps(profile, 1, 0, 0); + let mut parser = H264Parser::new(); + feed_sps_pps(&mut parser, &sps); + let cp = parser.codec_private().expect("avcC"); + assert_eq!(cp.len(), sps.len() + 14 + 4, "profile {profile}"); + } + for profile in [66u8, 77, 88] { + let sps = build_high_profile_sps(profile, 1, 0, 0); + let mut parser = H264Parser::new(); + feed_sps_pps(&mut parser, &sps); + let cp = parser.codec_private().expect("avcC"); + assert_eq!(cp.len(), sps.len() + 14, "profile {profile}"); + } + } + + // An IDR whose SPS changed but whose PPS did not: the re-asserted PPS must follow + // the new SPS (§7.3.2.2: a PPS is parsed against its SPS). + #[test] + fn reasserted_pps_follows_a_redefined_sps() { + let mut parser = H264Parser::new(); + let au = |sps: u8| { + let mut d = vec![0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x1E, sps]; + d.extend_from_slice(&[0x00, 0x00, 0x01, 0x68, 0x11]); + d.extend_from_slice(&[0x00, 0x00, 0x01, 0x65, 0x10, 0x20]); + d + }; + parser.parse(&make_pes(au(0xAA), Some(0))); + let f = parser.parse(&make_pes(au(0xAB), Some(90000))); + assert_eq!(frame_nal_types(&f[0].data), vec![7, 8, 5]); + } + #[test] fn keyframes_self_contained_and_redefinition_emitted() { let mut parser = H264Parser::new(); diff --git a/src/mux/codec/hevc.rs b/src/mux/codec/hevc.rs index c0582914..a0bf0dee 100644 --- a/src/mux/codec/hevc.rs +++ b/src/mux/codec/hevc.rs @@ -115,8 +115,8 @@ pub struct HevcParser { /// ids are 0..=63 (H.265 §7.4.3.3); an out-of-range id is ignored. pps_num_extra: [Option; HEVC_MAX_PPS_COUNT], // Splice-aware CRA→BLA rewrite for a non-seamless BD clip boundary (first - // CRA_NUT -> BLA_W_LP so NoRaslOutput discards dangling RASL). Armed by - // `mark_clip_boundary` AND PTS-backstep auto-detect in `parse` — not dead code. + // CRA_NUT -> BLA_W_LP so NoRaslOutput discards dangling RASL). Armed by PTS-backstep + // auto-detect in `parse`, or by the public `mark_clip_boundary` hook (no in-tree caller). pending_clip_boundary: bool, // Highest PES PTS seen, on a monotonic 64-bit timeline (33-bit PTS unwrapped // across 2^33 wraps — see `pts_wrap_offset`). Auto-detects a non-seamless clip @@ -446,12 +446,11 @@ impl CodecParser for HevcParser { let mut bla_au = false; // Picture coding type, MEASURED from the first coded slice's header. let mut coding_type: Option = None; - // Track whether THIS access unit already carried each param-set type - // in-band (a redefinition vs codecPrivate). Used after the scan to - // re-assert the active set at a keyframe the source left bare. - let mut emitted_vps = false; - let mut emitted_sps = false; - let mut emitted_pps = false; + // This AU's in-band redefinitions, held aside to lead the AU in VPS, SPS, PPS + // order: a re-asserted PPS ahead of a redefined SPS is dropped by the decoder. + let mut inband_vps = Vec::new(); + let mut inband_sps = Vec::new(); + let mut inband_pps = Vec::new(); // Pre-sized to input length plus PARAM_REASSERT_HEADROOM: UHD frames are // 150-300 KB and an unsized Vec otherwise reallocs 5-7x per frame; the // headroom also lets the keyframe param-set re-assert splice in without reallocating. @@ -493,20 +492,20 @@ impl CodecParser for HevcParser { match nal_type { NAL_VPS => { - emitted_vps |= handle_param_set( + handle_param_set( &mut self.vps, &mut self.cur_vps, &data[nal_start..end], - &mut frame_data, - ) + &mut inband_vps, + ); } NAL_SPS => { - emitted_sps |= handle_param_set( + handle_param_set( &mut self.sps, &mut self.cur_sps, &data[nal_start..end], - &mut frame_data, - ) + &mut inband_sps, + ); } NAL_PPS => { // Store the PPS under its own id so a later slice's @@ -519,12 +518,12 @@ impl CodecParser for HevcParser { { *slot = hevc_num_extra_slice_header_bits(pps); } - emitted_pps |= handle_param_set( + handle_param_set( &mut self.pps, &mut self.cur_pps, &data[nal_start..end], - &mut frame_data, - ) + &mut inband_pps, + ); } // Drop Access Unit Delimiters: Matroska HEVC frame data // omits AUDs (the container delimits access units), so @@ -569,18 +568,30 @@ impl CodecParser for HevcParser { } } - if frame_data.is_empty() { + if frame_data.is_empty() + && inband_vps.is_empty() + && inband_sps.is_empty() + && inband_pps.is_empty() + { return Vec::new(); } // A player re-applies hvcC param sets at every keyframe; if the active // set was redefined mid-title and the source stopped repeating it, the - // reversion desyncs CABAC. Re-assert in-band at every keyframe (self-heals). - if keyframe { + // reversion desyncs CABAC. Re-assert at every keyframe; a redefined type goes instead. + { let mut prefix = Vec::with_capacity(PARAM_REASSERT_HEADROOM); - reassert_active(&mut prefix, &self.cur_vps, emitted_vps); - reassert_active(&mut prefix, &self.cur_sps, emitted_sps); - reassert_active(&mut prefix, &self.cur_pps, emitted_pps); + for (inband, cur) in [ + (&inband_vps, &self.cur_vps), + (&inband_sps, &self.cur_sps), + (&inband_pps, &self.cur_pps), + ] { + if !inband.is_empty() { + prefix.extend_from_slice(inband); + } else if keyframe { + reassert_active(&mut prefix, cur, false); + } + } if !prefix.is_empty() { // Splice prefix into frame_data in place via PARAM_REASSERT_HEADROOM, // avoiding the extra whole-frame alloc+copy per keyframe the old @@ -696,14 +707,20 @@ impl CodecParser for HevcParser { // chromaFormat / bit depths — parse real values from the SPS RBSP; a // hardcoded 8-bit 4:2:0 is wrong for 10-bit Main 10 UHD (nearly all UHD // content). Fall back to it only if the SPS can't be parsed. - let chroma = parse_sps_chroma(sps).unwrap_or(SpsChroma { - chroma_format_idc: 1, - bit_depth_luma_minus8: 0, - bit_depth_chroma_minus8: 0, - max_sub_layers_minus1: 0, - temporal_id_nesting_flag: 0, - max_num_reorder_pics: None, - picture_period_ticks: None, + let chroma = parse_sps_chroma(sps).unwrap_or_else(|| { + tracing::warn!( + target: "freemkv::mux::hevc", + "HEVC SPS unparseable; hvcC declares 8-bit 4:2:0" + ); + SpsChroma { + chroma_format_idc: 1, + bit_depth_luma_minus8: 0, + bit_depth_chroma_minus8: 0, + max_sub_layers_minus1: 0, + temporal_id_nesting_flag: 0, + max_num_reorder_pics: None, + picture_period_ticks: None, + } }); // chromaFormat (6 reserved bits set + 2-bit chroma_format_idc) record.push(0xFC | (chroma.chroma_format_idc & 0x03)); @@ -771,6 +788,7 @@ struct SpsChroma { /// Reorder depth R of an HEVC SPS NAL (2-byte header included): /// `sps_max_num_reorder_pics[sps_max_sub_layers_minus1]` (H.265 §7.3.2.2 \[I\]). +#[cfg(test)] pub(crate) fn parse_sps_reorder(sps: &[u8]) -> Option { parse_sps_chroma(sps)?.max_num_reorder_pics } @@ -2657,6 +2675,39 @@ mod tests { // --- parameter-set redefinition (mid-title redefinition bug) --- + // An IRAP whose SPS changed but whose PPS did not: the re-asserted PPS must follow + // the new SPS, or the decoder parses it against the old one and drops it. + #[test] + fn reasserted_pps_follows_a_redefined_sps() { + let mut parser = HevcParser::new(); + let nal = |t: u8, body: u8| { + let mut v = vec![0x00, 0x00, 0x01]; + v.extend_from_slice(&hevc_nal_header(t)); + v.extend_from_slice(&[body, body]); + v + }; + let au = |sps: u8| { + let mut d = nal(32, 0x11); // VPS + d.extend(nal(33, sps)); // SPS + d.extend(nal(34, 0x33)); // PPS + d.extend(nal(19, 0x44)); // IDR_W_RADL + d + }; + let types = |fd: &[u8]| { + let (mut t, mut o) = (Vec::new(), 0usize); + while o + 4 <= fd.len() { + let len = u32::from_be_bytes([fd[o], fd[o + 1], fd[o + 2], fd[o + 3]]) as usize; + o += 4; + t.push((fd[o] >> 1) & 0x3F); + o += len; + } + t + }; + parser.parse(&make_pes(au(0x22), Some(0))); + let f = parser.parse(&make_pes(au(0x23), Some(1))); + assert_eq!(types(&f[0].data), vec![32, 33, 34, 19]); + } + // A parameter set REDEFINED mid-stream must be emitted INLINE so the decoder re-activates // it. #[test] @@ -3243,6 +3294,193 @@ mod tests { assert_eq!(cp[18], 0xF8 | 2); } + // PTL and the SPS fields are read off the emulation-prevention-STRIPPED RBSP; a real + // UHD SPS carries `00 00 03` inside general_constraint_indicator_flags. + #[test] + fn hvcc_reads_ptl_and_chroma_through_emulation_prevention_in_the_ptl() { + let sps = make_sps_with_chroma(1, 2, 2); + let mut rbsp = sps[2..].to_vec(); + rbsp[1] = 0x21; // profile + rbsp[2] = 0x60; // compatibility flags 0x60000000 + rbsp[6] = 0x90; // constraint flags 0x900000000000 + rbsp[12] = 0x7B; // level_idc + let escaped = emulation_prevent(&rbsp); + assert!(escaped.len() > rbsp.len(), "the zero runs were escaped"); + let mut stored = sps[..2].to_vec(); + stored.extend_from_slice(&escaped); + + assert_eq!(parse_sps_chroma(&stored).unwrap().chroma_format_idc, 1); + let cp = codec_private_from_sps(&stored); + assert_eq!(cp[1], 0x21); + assert_eq!(&cp[2..6], &[0x60, 0, 0, 0]); + assert_eq!(&cp[6..12], &[0x90, 0, 0, 0, 0, 0]); + assert_eq!(cp[12], 0x7B); + assert_eq!(cp[16], 0xFC | 1); + assert_eq!(cp[17], 0xF8 | 2); + assert_eq!(cp[18], 0xF8 | 2); + } + + // The gap flag on a PES must reach the frame (both emit paths), or the resync gate never + // arms for HEVC. + #[test] + fn pes_discontinuity_propagates_to_frame() { + for reorder in [false, true] { + for flag in [true, false] { + let mut pes = make_pes(cra_au(&[0x10]), Some(0)); + pes.discontinuity = flag; + let mut parser = HevcParser::new().with_ps_reorder(reorder); + let mut frames = parser.parse(&pes); + frames.extend(parser.flush()); + assert_eq!(frames.len(), 1, "reorder {reorder}"); + assert_eq!(frames[0].discontinuity, flag, "reorder {reorder}"); + } + } + } + + // Other SEI messages sharing the NAL (or an HDR10 message already captured) are skipped by + // their declared size, so the wanted message behind them is still found. + #[test] + fn hdr10_sei_is_found_behind_other_and_already_captured_messages() { + let idr = nal_bytes(19, &[0xEC]); + let pps = nal_bytes(NAL_PPS, &[0xC0]); + let mastering = |max_lum| { + sei_message( + SEI_MASTERING_DISPLAY_COLOUR_VOLUME, + &mastering_payload([1, 2, 3], [4, 5, 6], 7, 8, max_lum, 10), + ) + }; + let cll = sei_message(SEI_CONTENT_LIGHT_LEVEL_INFO, &cll_payload(1000, 400)); + let other = |n: usize| sei_message(1, &vec![0x5A; n]); + + let mut one_nal = HevcParser::new(); + let mut au = pps.clone(); + au.extend_from_slice(&sei_nal(&[ + other(5), + mastering(10_000_000), + other(3), + cll.clone(), + ])); + au.extend_from_slice(&idr); + one_nal.parse(&make_pes(au, Some(0))); + assert_eq!( + one_nal + .sei_mastering + .map(|m| m.max_display_mastering_luminance), + Some(10_000_000) + ); + assert!(one_nal.sei_content_light.is_some()); + + let mut later = HevcParser::new(); + let mut first = pps.clone(); + first.extend_from_slice(&sei_nal(&[mastering(10_000_000)])); + first.extend_from_slice(&idr); + later.parse(&make_pes(first, Some(0))); + assert!(later.sei_content_light.is_none()); + let mut second = pps.clone(); + second.extend_from_slice(&sei_nal(&[mastering(1), other(4), cll.clone()])); + second.extend_from_slice(&idr); + later.parse(&make_pes(second, Some(3750))); + assert_eq!( + later + .sei_mastering + .map(|m| m.max_display_mastering_luminance), + Some(10_000_000), + "the repeat is skipped, not adopted" + ); + assert!( + later.sei_content_light.is_some(), + "content light behind a skipped repeat" + ); + } + + #[test] + fn hdr10_sei_in_a_suffix_nal_is_captured() { + let mut sei = sei_nal(&[sei_message( + SEI_CONTENT_LIGHT_LEVEL_INFO, + &cll_payload(1000, 400), + )]); + sei[3] = NAL_SEI_SUFFIX << 1; + let mut au = nal_bytes(NAL_PPS, &[0xC0]); + au.extend_from_slice(&nal_bytes(19, &[0xEC])); + au.extend_from_slice(&sei); + let mut parser = HevcParser::new(); + parser.parse(&make_pes(au, Some(0))); + assert!(parser.sei_content_light.is_some()); + } + + // Only the first slice segment of a picture carries the slice_type this reads; a later + // segment's bits would otherwise be taken for one. + #[test] + fn a_non_first_slice_segment_reports_no_coding_type() { + for nal_type in [1u8, 19] { + let nal = nal_bytes(nal_type, &[0x6C, 0x00]); + // 0x6C = first_slice_segment_in_pic_flag 0, then bits that would decode as an + // I slice (no_output_of_prior_pics 1, pps id 0, slice_type 2) if the flag were ignored. + assert_eq!( + hevc_first_slice_coding_type(&nal[3..], nal_type, |_| Some(0)), + None, + "NAL {nal_type}" + ); + } + } + + #[test] + fn hvcc_oversized_vps_or_pps_returns_none() { + for oversized in [32u8, 34] { + let mut data = Vec::new(); + for t in [32u8, 33, 34] { + data.extend_from_slice(&[0x00, 0x00, 0x01]); + data.extend_from_slice(&hevc_nal_header(t)); + let n = if t == oversized { 70_000 } else { 2 }; + data.extend_from_slice(&vec![0x11u8; n]); + } + let mut parser = HevcParser::new(); + parser.parse(&make_pes(data, Some(0))); + assert!( + parser.codec_private().is_none(), + "oversized NAL type {oversized} must not produce a truncated hvcC" + ); + } + } + + // A bare IRAP re-asserts the ACTIVE parameter sets, not the first-seen hvcC copy: a player + // re-applies hvcC at every keyframe, so reverting would undo a mid-title redefinition. + #[test] + fn a_bare_keyframe_reasserts_the_active_param_sets_not_the_hvcc_copy() { + let au = |sets: Option, slice: Vec| { + let mut d = Vec::new(); + if let Some(b) = sets { + d.extend_from_slice(&nal_bytes(32, &[0xA0, b])); + d.extend_from_slice(&nal_bytes(33, &[0xB0, b])); + d.extend_from_slice(&nal_bytes(NAL_PPS, &[0xC0 | b])); + } + d.extend_from_slice(&slice); + d + }; + let bodies = |fd: &[u8]| -> Vec<(u8, u8)> { + let mut out = Vec::new(); + let mut off = 0; + while off + 4 <= fd.len() { + let len = + u32::from_be_bytes([fd[off], fd[off + 1], fd[off + 2], fd[off + 3]]) as usize; + out.push(((fd[off + 4] >> 1) & 0x3F, fd[off + 4 + len - 1])); + off += 4 + len; + } + out + }; + let mut parser = HevcParser::new(); + parser.parse(&make_pes(au(Some(1), nal_bytes(19, &[0xEC])), Some(0))); + // Redefinition of all three sets on a non-keyframe access unit. + let f = parser.parse(&make_pes(au(Some(2), nal_bytes(1, &[0xD0])), Some(3000))); + assert_eq!(bodies(&f[0].data).len(), 4); + let f = parser.parse(&make_pes(au(None, nal_bytes(19, &[0xEC])), Some(6000))); + assert_eq!( + bodies(&f[0].data), + vec![(32, 2), (33, 2), (34, 0xC2), (19, 0xEC)], + "VPS, SPS, PPS are the redefined ones, in hierarchy order" + ); + } + // --- BitReader unit tests (exp-Golomb + bit reads) --- #[test] diff --git a/src/mux/codec/lpcm.rs b/src/mux/codec/lpcm.rs index 9bfc5c2a..3644e5ba 100644 --- a/src/mux/codec/lpcm.rs +++ b/src/mux/codec/lpcm.rs @@ -966,6 +966,30 @@ mod tests { assert_eq!(got, pcm); } + // Odd counts are coded with a pad channel: the payload_size field and the layout must say so, + // and the parser must read the bytes back. + #[test] + fn bd_payloads_pad_odd_channel_counts_and_round_trip() { + for channels in 1u8..=8 { + let h = bd_header(channels, 48_000, None, 24).unwrap(); + let n = usize::from(channels); + let pcm: Vec = (0..5 * n * 3).map(|i| (i % 251) as u8 + 1).collect(); + let parts = bd_payloads(&pcm, h); + assert_eq!(parts.len(), 1); + let coded = (n + n % 2) * 3; + let p = &parts[0].1; + assert_eq!(p.len(), 4 + 5 * coded, "{channels} ch"); + assert_eq!( + usize::from(u16::from_be_bytes([p[0], p[1]])), + 5 * coded, + "payload_size of {channels} ch" + ); + assert_eq!(&p[2..4], &h); + let got = LpcmParser::new().parse(&make_pes(p.clone(), Some(0))); + assert_eq!(got[0].data, pcm, "{channels} ch round trip"); + } + } + #[test] fn bd_payloads_are_5ms_pes_like_ffmpeg_blurayenc() { // 20000 stereo samples @ 48 kHz -> 83 full 240-sample PES + one of 80. @@ -1091,6 +1115,86 @@ mod tests { } } + // The channel_assignment table: count and the reorder into WAVE channel order, by value. + #[test] + fn bd_channel_assignments_map_to_wave_order() { + // (assignment, source channel values in BD order, expected WAVE-order values) + let cases: [(u8, &[u8], &[u8]); 10] = [ + (1, &[1], &[1]), + (3, &[1, 2], &[1, 2]), + (4, &[1, 2, 3], &[1, 2, 3]), + (5, &[1, 2, 3], &[1, 2, 3]), + (6, &[1, 2, 3, 4], &[1, 2, 3, 4]), + (7, &[1, 2, 3, 4], &[1, 2, 3, 4]), + (8, &[1, 2, 3, 4, 5], &[1, 2, 3, 4, 5]), + (9, &[1, 2, 3, 4, 5, 6], &[1, 2, 3, 6, 4, 5]), + (10, &[1, 2, 3, 4, 5, 6, 7], &[1, 2, 3, 5, 6, 4, 7]), + (11, &[1, 2, 3, 4, 5, 6, 7, 8], &[1, 2, 3, 8, 5, 6, 4, 7]), + ]; + for (assign, src, want) in cases { + // 16-bit, coded channels padded to even. + let mut pcm: Vec = src.iter().flat_map(|&c| [c, c]).collect(); + if src.len() % 2 == 1 { + pcm.extend_from_slice(&[0, 0]); + } + let mut p = LpcmParser::new(); + let f = p.parse(&make_pes(bd(assign, 1, &pcm), Some(0))); + let expect: Vec = want.iter().flat_map(|&c| [c, c]).collect(); + assert_eq!(f[0].data, expect, "assignment {assign}"); + } + } + + // Timeline advance across a PTS-less PES uses the header's sample rate (10 ms of stereo + // 16-bit audio per packet). + #[test] + fn a_ptsless_packet_advances_by_the_headers_sample_rate() { + for (code, hz) in [(1u8, 48_000usize), (4, 96_000), (5, 192_000)] { + let mut pes = vec![0x00, 0x00, (3 << 4) | code, 1 << 6]; + pes.extend(vec![0u8; hz / 100 * 4]); + let mut p = LpcmParser::new(); + p.parse(&make_pes(pes.clone(), Some(0))); + let f = p.parse(&make_pes(pes, None)); + assert_eq!(f[0].pts_ns, 10_000_000, "BD rate code {code} = {hz} Hz"); + } + for (freq, hz) in [(0u8, 48_000usize), (1, 96_000), (2, 44_100), (3, 32_000)] { + let mut pes = vec![0x00, (freq << 4) | 1, 0x80]; + pes.extend(vec![0u8; hz / 100 * 4]); + let mut p = LpcmParser::new_dvd(); + p.parse(&make_pes(pes.clone(), Some(0))); + let f = p.parse(&make_pes(pes, None)); + assert_eq!(f[0].pts_ns, 10_000_000, "DVD rate code {freq} = {hz} Hz"); + } + } + + // The M2TS re-mux header, against the pcm-blurayenc.c tables: assignment for each count, + // rate nibble, and the bits field. + #[test] + fn bd_header_encodes_assignment_rate_and_depth() { + for (channels, assign) in [ + (1u8, 1u8), + (2, 3), + (3, 4), + (4, 6), + (5, 8), + (6, 9), + (7, 10), + (8, 11), + ] { + for (hz, code) in [(48_000u32, 1u8), (96_000, 4), (192_000, 5)] { + assert_eq!( + bd_header(channels, hz, None, 24), + Some([(assign << 4) | code, 0xC0]), + "{channels} ch {hz} Hz" + ); + } + } + assert_eq!(bd_header(2, 48_000, None, 16).unwrap()[1], 0x40); + assert_eq!(bd_header(9, 48_000, None, 24), None); + // A source layout byte is reused only when it agrees with the count and rate. + assert_eq!(bd_header(6, 48_000, Some(0x71), 24), Some([0x91, 0xC0])); + assert_eq!(bd_header(4, 48_000, Some(0x71), 24), Some([0x71, 0xC0])); + } + #[test] fn dvd_bits_needed_is_the_smallest_lossless_depth() { assert_eq!(dvd_bits_needed(&[1, 2, 0, 3, 4, 0]), 16); diff --git a/src/mux/codec/mp2_channels.rs b/src/mux/codec/mp2_channels.rs index f198bdf2..dc4e14ee 100644 --- a/src/mux/codec/mp2_channels.rs +++ b/src/mux/codec/mp2_channels.rs @@ -541,7 +541,7 @@ struct Composite { /// What one stored frame says about its channels. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum Frame { +pub(crate) enum ChannelFrame { /// No valid 11172-3 header: not a frame this module can judge. NoHeader, /// A clean base frame whose multichannel extension verified (§2.5.3.1). @@ -553,22 +553,22 @@ pub(crate) enum Frame { } /// Judges one stored frame against 11172-3 and 13818-3. -pub(crate) fn inspect(frame: &[u8]) -> Frame { +pub(crate) fn inspect(frame: &[u8]) -> ChannelFrame { inspect_within(frame, frame.len() * 8) } // As `inspect`, reading no further than bit `end` (tests cut at exact field bits). -fn inspect_within(frame: &[u8], end: usize) -> Frame { +fn inspect_within(frame: &[u8], end: usize) -> ChannelFrame { let Some(h) = Header::parse(frame) else { - return Frame::NoHeader; + return ChannelFrame::NoHeader; }; let nch = h.nch(); let base = match walk_base(frame, &h, end) { Ok(base) => base, Err(why @ (Fallback::FrameCrc | Fallback::Truncated(_))) => { - return Frame::Damaged { nch, why }; + return ChannelFrame::Damaged { nch, why }; } - Err(why) => return Frame::Base { nch, why }, + Err(why) => return ChannelFrame::Base { nch, why }, }; // The frame's own length bounds mc_extension_data_part1 (11172-3 §2.4.3.1 slots). let frame_end = h @@ -576,8 +576,8 @@ fn inspect_within(frame: &[u8], end: usize) -> Frame { .map_or(end, |n| (n * 8).min(end)) .min(frame.len() * 8); match mc_extension(frame, &h, &base, frame_end) { - Ok(mc) => Frame::Multichannel { nch, mc }, - Err(why) => Frame::Base { nch, why }, + Ok(mc) => ChannelFrame::Multichannel { nch, mc }, + Err(why) => ChannelFrame::Base { nch, why }, } } @@ -737,7 +737,7 @@ impl ChannelTracker { } self.frames += 1; match inspect(frame) { - Frame::Multichannel { nch, mc } => { + ChannelFrame::Multichannel { nch, mc } => { self.base = Some(nch); // §2.5.2.13: "'1' extension bit stream present" - the rest is not in the track. let count = if mc.ext_bit_stream_present { @@ -757,18 +757,18 @@ impl ChannelTracker { } } // A clean frame without a verified mc_header (§2.5.3.1) breaks the run. - Frame::Base { nch, why } => { + ChannelFrame::Base { nch, why } => { self.base = Some(nch); self.last = Some(why); self.run = None; } // A failed frame CRC covers the header too, so its nch is not trusted; like a // header-less frame it breaks the run, which must be consecutive. - Frame::Damaged { why, .. } => { + ChannelFrame::Damaged { why, .. } => { self.last = Some(why); self.run = None; } - Frame::NoHeader => self.run = None, + ChannelFrame::NoHeader => self.run = None, } if self.frames >= MAX_FRAMES && self.run.is_none() { self.settled = true; diff --git a/src/mux/codec/mp2_channels_tests.rs b/src/mux/codec/mp2_channels_tests.rs index 06d8307f..dfae5d7f 100644 --- a/src/mux/codec/mp2_channels_tests.rs +++ b/src/mux/codec/mp2_channels_tests.rs @@ -383,7 +383,7 @@ fn mc(centre: u8, surround: u8, lfe: bool) -> Mc { fn multichannel(f: &[u8]) -> Option { match inspect(f) { - Frame::Multichannel { mc, .. } => Some(mc), + ChannelFrame::Multichannel { mc, .. } => Some(mc), _ => None, } } @@ -730,7 +730,7 @@ fn ffmpeg_encoded_frames_walk_to_the_end_of_every_frame() { f.len() * 8 - base.mc_start ); // §2.5.3.1: no valid mc_crc_check in MPEG-1 frames, so no multichannel. - assert!(matches!(inspect(f), Frame::Base { nch: c, .. } if c == nch)); + assert!(matches!(inspect(f), ChannelFrame::Base { nch: c, .. } if c == nch)); n += 1; } } @@ -753,7 +753,10 @@ fn frame_crc_is_checked_when_protection_bit_is_zero() { }; // §2.4.3.1: header bits "starting with bit_rate_index and ending with emphasis" plus the // Table 3-B.5 audio_data bits (allocation and scfsi for Layer II). - assert!(matches!(inspect(&frame(crc)), Frame::Base { nch: 2, .. })); + assert!(matches!( + inspect(&frame(crc)), + ChannelFrame::Base { nch: 2, .. } + )); let bad = Spec { bad_frame_crc: true, ..crc @@ -761,7 +764,7 @@ fn frame_crc_is_checked_when_protection_bit_is_zero() { // "If the words are not identical, a transmission error has occured in the protected field". assert_eq!( inspect(&frame(bad)), - Frame::Damaged { + ChannelFrame::Damaged { nch: 2, why: Fallback::FrameCrc } @@ -770,7 +773,7 @@ fn frame_crc_is_checked_when_protection_bit_is_zero() { flipped[3] ^= 0x01; // emphasis bit: inside the protected header bits assert_eq!( inspect(&flipped), - Frame::Damaged { + ChannelFrame::Damaged { nch: 2, why: Fallback::FrameCrc } @@ -792,7 +795,7 @@ fn multichannel_needs_a_valid_mc_crc_check() { }; assert_eq!( inspect(&frame(bad)), - Frame::Base { + ChannelFrame::Base { nch: 2, why: Fallback::McCrc } @@ -877,7 +880,7 @@ fn centre_10_is_not_defined_and_falls_back() { // §2.5.2.13 centre: "'10' not defined". assert_eq!( inspect(&f), - Frame::Base { + ChannelFrame::Base { nch: 2, why: Fallback::UndefinedCentre } @@ -929,7 +932,7 @@ fn ext_one_mc_data_beyond_the_base_part_cannot_be_verified() { let f = frame(with_mc(STEREO_256, huge)); assert!(matches!( inspect(&f), - Frame::Base { + ChannelFrame::Base { why: Fallback::Truncated(_), .. } @@ -1198,7 +1201,7 @@ fn unsupported_frames_give_the_header_count() { ), ]; for (s, why) in cases { - assert_eq!(inspect(&frame(s)), Frame::Base { nch: 2, why }); + assert_eq!(inspect(&frame(s)), ChannelFrame::Base { nch: 2, why }); } } @@ -1256,13 +1259,13 @@ fn truncation_at_each_field_boundary_names_the_field() { f => f, }); let want = if base_field { - Frame::Damaged { nch: 2, why } + ChannelFrame::Damaged { nch: 2, why } } else { - Frame::Base { nch: 2, why } + ChannelFrame::Base { nch: 2, why } }; assert_eq!(got, want, "{field}"); } - assert_eq!(inspect(&f[..3]), Frame::NoHeader); + assert_eq!(inspect(&f[..3]), ChannelFrame::NoHeader); } /// Opus defect 1: a failed first frame must not lock in the base count. diff --git a/src/mux/codec/mpeg2.rs b/src/mux/codec/mpeg2.rs index f2388b38..d9f7f3de 100644 --- a/src/mux/codec/mpeg2.rs +++ b/src/mux/codec/mpeg2.rs @@ -33,13 +33,12 @@ const PICTURE_TYPE_I: u8 = 1; #[cfg(test)] const MAX_AU_BUFFER: usize = 8 * 1024 * 1024; -/// Cap on frames held awaiting the first PES PTS anchor. A DVD stamps a PTS in -/// the first VOBU (~0.5 s ≈ 15 frames); this leaves generous slack. If no PTS -/// ever arrives within the cap, buffered frames are released on a 0 base. +/// Cap on frames buffered in one GOP. A DVD GOP is ~15 frames; a run reaching the +/// cap is force-flushed as its own GOP, and each split re-locks its origin and +/// display-order PTS from the frames it holds. const MAX_PENDING_FRAMES: usize = 600; -// Byte cap on frames held awaiting the first PES PTS anchor; mirrors the AC-3/DTS/PGS byte -// caps. +// Byte cap on one buffered GOP; mirrors the AC-3/DTS/PGS byte caps. const MAX_PENDING_BYTES: usize = 8 * 1024 * 1024; /// Frame rate table (index from sequence header frame_rate_code). @@ -451,7 +450,7 @@ fn picture_coding_flags(au: &[u8]) -> (bool, bool, bool, bool) { /// codec-agnostic [`CodingType`]: 1 → I, 3 → B, else (2 = P, 4 = D) → P. fn coding_type_from_raw(raw: u8) -> CodingType { match raw { - 1 => CodingType::I, + PICTURE_TYPE_I => CodingType::I, 3 => CodingType::B, _ => CodingType::P, } @@ -541,6 +540,55 @@ mod tests { } } + #[test] + fn a_second_top_field_starts_a_new_pair_instead_of_partnering_the_first() { + use crate::mux::codec::coding::FieldOrder; + let mut p = Mpeg2Parser::new(); + let mut frames = Vec::new(); + // top, top, bottom: the bottom field pairs with the SECOND top field. + for (i, (ct, e2)) in [(1u8, 0x01u8), (2, 0x01), (2, 0x02)] + .into_iter() + .enumerate() + { + let mut au = Vec::new(); + if i == 0 { + au.extend_from_slice(&make_seq_header(720, 576, 3, 3)); + } + au.extend_from_slice(&make_picture_header(ct)); + let mut ext = pic_coding_ext(0, 0, 0, false); + ext[6] = e2; + au.extend_from_slice(&ext); + frames.extend(p.parse(&make_pes(au, None))); + } + frames.extend(p.flush()); + assert_eq!(frames.len(), 3); + for f in &frames { + assert_eq!(f.coding.unwrap().field_order(), Some(FieldOrder::Tff)); + } + } + + #[test] + fn a_pts_jump_in_a_later_gop_relocks_the_timeline_origin() { + // 25 fps = 40 ms. GOP 1 anchors at 0; GOP 2 carries a PES PTS of 10 s + // (a clip boundary), so its frames follow it, not GOP 1's extrapolation. + let mut p = Mpeg2Parser::new(); + let mut g1 = make_seq_header(720, 480, 3, 3); + g1.extend_from_slice(&gop()); + g1.extend_from_slice(&make_picture_header_tr(1, 0)); + g1.extend_from_slice(&[0xAA; 10]); + g1.extend_from_slice(&make_picture_header_tr(2, 1)); + g1.extend_from_slice(&[0xBB; 10]); + let mut frames = p.parse(&make_pes(g1, Some(0))); + let mut g2 = gop(); + g2.extend_from_slice(&make_picture_header_tr(1, 0)); + g2.extend_from_slice(&[0xCC; 10]); + frames.extend(p.parse(&make_pes(g2, Some(10 * 90_000)))); + frames.extend(p.flush()); + assert_eq!(frames.len(), 3); + assert_eq!(frames[1].pts_ns, 40_000_000); + assert_eq!(frames[2].pts_ns, 10_000_000_000); + } + #[test] fn gop_header_resets_stale_lone_field() { use crate::mux::codec::coding::FieldOrder; diff --git a/src/mux/codec/mpegaudio.rs b/src/mux/codec/mpegaudio.rs index 463fc2e6..860aadad 100644 --- a/src/mux/codec/mpegaudio.rs +++ b/src/mux/codec/mpegaudio.rs @@ -421,6 +421,31 @@ mod tests { } } + // A learned free-format size excludes padding: a padded frame (padding bit set) is one slot + // longer, so the next header is looked for one byte later. + #[test] + fn free_format_frames_with_alternating_padding_keep_their_exact_bytes() { + let plain = free_frame(); + let mut padded = free_frame(); + padded[2] |= 0x02; // padding bit + padded.push(0xAA); // Layer III padding is one slot + let (plain, padded) = (plain.as_slice(), padded.as_slice()); + // Starting on a padded frame learns the size from a padded spacing. + for frames in [ + [plain, padded, plain, padded, plain, padded], + [padded, plain, padded, plain, padded, plain], + ] { + let mut p = MpegAudioParser::new(); + let mut f = p.parse(&make_pes(frames.concat(), Some(0))); + f.extend(p.flush()); + assert_eq!(f.len(), 6); + for (got, want) in f.iter().zip(frames) { + assert_eq!(got.data, want); + } + assert_eq!(p.dropped_frames(), 0); + } + } + #[test] fn free_format_frame_split_across_pes_reassembles() { let data = free_frame().repeat(2); diff --git a/src/mux/codec/pgs.rs b/src/mux/codec/pgs.rs index 6dabff17..4583f0cb 100644 --- a/src/mux/codec/pgs.rs +++ b/src/mux/codec/pgs.rs @@ -333,6 +333,7 @@ impl CodecParser for PgsParser { // let it fall to the non-PCS arm (would pollute the pending set): close // any pending set with a fallback and drop the header to resync on next PCS. if is_pcs && pes.data.len() <= PCS_NUM_OBJECTS_OFFSET { + self.skip_to_pcs = true; return self.emit_pending(None).into_iter().collect(); } @@ -350,6 +351,17 @@ impl CodecParser for PgsParser { Some(_) => match pts { Some(start) => { self.skip_to_pcs = false; + // A gap before this PCS cut the held set off from its END; + // closing it would emit a malformed display set. + if pes.discontinuity + && !self + .pending + .as_ref() + .is_some_and(|(_, d)| Self::ends_with_end(d)) + { + self.pending = None; + } + self.clear_scan_offset = 0; out.extend(self.emit_pending(Some(start))); // The set's facts are THIS packet's — the one that opened // it. `start` is that packet's PTS by construction. @@ -359,6 +371,7 @@ impl CodecParser for PgsParser { // store it with a 0 sentinel (wrong start, absurd duration). // Flush any prior pending with a fallback and skip this one. None => { + self.skip_to_pcs = true; out.extend(self.emit_pending(None)); } }, @@ -377,6 +390,7 @@ impl CodecParser for PgsParser { out.extend(self.emit_pending(None)); } self.pending = None; + self.clear_scan_offset = 0; self.skip_to_pcs = true; } if self.skip_to_pcs { @@ -634,6 +648,66 @@ mod tests { assert!(parser.flush().is_empty()); } + #[test] + fn pcs_after_gap_drops_an_incomplete_pending_set() { + let mut parser = PgsParser::new(); + let _ = parser.parse(&make_pes(pcs_bytes(1), Some(90000))); + let mut next = make_pes(pcs_bytes(1), Some(180000)); + next.discontinuity = true; + assert!( + parser.parse(&next).is_empty(), + "a set cut off before its END is not emitted" + ); + let tail = parser.flush(); + assert_eq!(tail.len(), 1, "the post-gap PCS opens a fresh set"); + assert_eq!(tail[0].pts_ns, 2_000_000_000); + } + + #[test] + fn pcs_after_gap_emits_a_complete_pending_set() { + let mut parser = PgsParser::new(); + let _ = parser.parse(&make_pes(pcs_bytes(1), Some(90000))); + let _ = parser.parse(&make_pes(vec![0x80, 0x00, 0x00], Some(90000))); + let mut next = make_pes(pcs_bytes(1), Some(180000)); + next.discontinuity = true; + let out = parser.parse(&next); + assert_eq!(out.len(), 1); + assert_eq!(out[0].duration_ns, Some(1_000_000_000)); + } + + #[test] + fn segments_after_a_dropped_pcs_are_skipped_until_the_next_pcs() { + let mut short_pcs = pcs_bytes(1); + short_pcs.truncate(PCS_NUM_OBJECTS_OFFSET); + for (dropped, pts) in [(short_pcs, Some(90000)), (pcs_bytes(1), None)] { + let mut parser = PgsParser::new(); + assert!(parser.parse(&make_pes(dropped, pts)).is_empty()); + for seg in [ + vec![0x17, 0x00, 0x00], + vec![0x15, 0x00, 0x02, 0xAA, 0xBB], + vec![0x80, 0x00, 0x00], + ] { + assert!( + parser.parse(&make_pes(seg, Some(90000))).is_empty(), + "orphan segments of a lost PCS are not emitted" + ); + } + let _ = parser.parse(&make_pes(pcs_bytes(1), Some(180000))); + assert_eq!(parser.flush().len(), 1, "the next PCS resyncs"); + } + } + + #[test] + fn a_gap_resets_the_clear_scan_offset() { + let mut parser = PgsParser::new(); + let _ = parser.parse(&make_pes(pcs_bytes(0), Some(90000))); + assert_ne!(parser.clear_scan_offset, 0, "the clear PCS was walked"); + let mut gap = make_pes(vec![0x17, 0x00, 0x00], None); + gap.discontinuity = true; + let _ = parser.parse(&gap); + assert_eq!(parser.clear_scan_offset, 0); + } + #[test] fn pending_buffer_is_capped() { let mut parser = PgsParser::new(); @@ -976,6 +1050,18 @@ mod tests { } } + // The share test multiplies a disc-derived count; it must not wrap at the top of u32. + #[test] + fn the_share_test_does_not_overflow_on_huge_counts() { + assert!(demotable(facts(u32::MAX, 1), true, u32::MAX)); + assert!(demotable(facts(u32::MAX / 2, 1), true, u32::MAX)); + assert!(!demotable( + facts(DEMOTE_MIN_DISPLAY_SETS, 1), + true, + u32::MAX + )); + } + // A track that mixes forced and non-forced sets needs no sibling corroboration: the flag is // in use ON THIS TRACK. #[test] diff --git a/src/mux/codec/reorder.rs b/src/mux/codec/reorder.rs index 04b6c289..9556667a 100644 --- a/src/mux/codec/reorder.rs +++ b/src/mux/codec/reorder.rs @@ -357,6 +357,27 @@ mod tests { assert_eq!(&got[..4], &[0, dur, 2 * dur, 3 * dur]); } + // The slots between two anchors exclude the held GOP's frames displayed before its own anchor. + #[test] + fn calibration_discounts_the_held_gops_leading_display_offset() { + use CodingType::*; + let dur = 40_000_000i64; + let mut r = SparsePtsReorder::new(); + let mut got: Vec = Vec::new(); + // Open first GOP, decode I B B P: the I displays third (slot 2), the P fourth. + for (k, ct) in [I, B, B, P].into_iter().enumerate() { + let p = (k == 0).then_some(2 * dur); + got.extend(r.push(p, frame(ct, k == 0)).iter().map(|f| f.pts_ns)); + } + // The next GOP's I is anchored two slots after the first anchor. + for (k, ct) in [I, P].into_iter().enumerate() { + let p = (k == 0).then_some(4 * dur); + got.extend(r.push(p, frame(ct, k == 0)).iter().map(|f| f.pts_ns)); + } + got.extend(r.flush().iter().map(|f| f.pts_ns)); + assert_eq!(&got[..4], &[2 * dur, 0, dur, 3 * dur]); + } + #[test] fn no_pts_collisions_within_a_gop() { use CodingType::*; diff --git a/src/mux/codec/truehd.rs b/src/mux/codec/truehd.rs index 3079695d..f57ab6ab 100644 --- a/src/mux/codec/truehd.rs +++ b/src/mux/codec/truehd.rs @@ -36,9 +36,9 @@ const AU_DURATION_NS: i64 = 833_333; // kHz constant would run ~8.95% fast on these (rare) streams. const AU_DURATION_NS_441: i64 = 907_029; -// Hard cap on the reassembly buffer: a valid AU is well under 32 KiB, so -// past this the stream is malformed — drop and resync. Parity with the -// AC-3/DTS/PGS caps. +// Backstop on the reassembly buffer, parity with the AC-3/DTS/PGS caps. The parse loop +// already drains every complete AU, so the residue never exceeds one AU (8190 bytes) and +// this does not fire today; it bounds memory if that loop changes. const MAX_TRUEHD_BUF: usize = 256 * 1024; pub struct TrueHdParser { @@ -1495,6 +1495,8 @@ mod tests { assert_eq!(truehd_lfe(0x4B), 0, "7.0 has no LFE"); assert_eq!(truehd_lfe(0xF << 15), 1, "6ch LFE bit when 8ch is empty"); assert_eq!(truehd_lfe(0x03 | (0xF << 15)), 0, "8ch wins over 6ch"); + assert_eq!(truehd_lfe(0x1B << 15), 0, "6ch mask with every bit but LFE"); + assert_eq!(truehd_lfe(0x04 << 15), 1, "6ch LFE bit alone"); assert_eq!(truehd_lfe(0), 0); } diff --git a/src/mux/codec/vc1.rs b/src/mux/codec/vc1.rs index 23767858..6b19ec75 100644 --- a/src/mux/codec/vc1.rs +++ b/src/mux/codec/vc1.rs @@ -486,6 +486,63 @@ mod tests { assert!(raw_during.iter().filter(|f| f.pts_ns == 0).count() >= 4); } + // The PES PTS anchors the reconstructed display timeline: with anchors 5 frames apart the + // frame duration calibrates to a fifth of the spacing, and each GOP's frames follow. + #[test] + fn vc1_ps_reorder_stamps_display_order_pts_from_the_pes_anchors() { + let seq = [0x00, 0x00, 0x01, SC_SEQUENCE_HEADER, 0xC0, 0, 0, 0, 0, 0]; + let pic = |ptype: u8| vec![0x00, 0x00, 0x01, SC_FRAME, ptype]; + let mut p = Vc1Parser::new().with_ps_reorder(true); + let mut frames = Vec::new(); + for anchor in [0i64, 18_750] { + for (k, ptype) in [0xC0u8, 0x00, 0x80, 0x00, 0x80].into_iter().enumerate() { + let data = if k == 0 { + [&seq[..], &pic(ptype)].concat() + } else { + pic(ptype) + }; + frames.extend(p.parse(&make_pes(data, (k == 0).then_some(anchor)))); + } + } + frames.extend(p.flush()); + let d = 208_333_333 / 5; + let gop = |origin: i64| [0, 2, 1, 4, 3].map(|i| origin + i * d); + let want: Vec = gop(0).into_iter().chain(gop(208_333_333)).collect(); + let got: Vec = frames.iter().map(|f| f.pts_ns).collect(); + assert_eq!(got, want); + } + + // An advanced-profile sequence header may carry `00 00 03` emulation prevention; the + // resolution and INTERLACE bit are read from the de-escaped bytes. + #[test] + fn sequence_header_fields_are_read_through_emulation_prevention() { + // De-escaped: PROFILE 3 byte, 00, then W-1 = 0 (12 bits), H-1 = 31 (12 bits), + // then INTERLACE set at bit 41. Bytes 1..=3 are 00 00 00, so an encoder escapes them. + let sh = [ + 0x00, + 0x00, + 0x01, + SC_SEQUENCE_HEADER, + 0xC0, + 0x00, + 0x00, + 0x03, + 0x00, + 0x1F, + 0x40, + ]; + assert_eq!(parse_vc1_resolution(&sh), Some((2, 64))); + assert_eq!(parse_vc1_interlace(&sh), Some(true)); + } + + #[test] + fn a_pes_ending_in_a_bare_start_code_prefix_passes_through_without_panicking() { + let data = vec![0xAA, 0x00, 0x00, 0x01]; + let f = Vc1Parser::new().parse(&make_pes(data.clone(), Some(0))); + assert_eq!(f.len(), 1); + assert_eq!(f[0].data, data); + } + /// Build a VC-1 PES with sequence header + entry point + frame start code. fn build_vc1_iframe_pes() -> Vec { let mut data = Vec::new(); diff --git a/src/mux/decode_ts/tests.rs b/src/mux/decode_ts/tests.rs index b184402d..68a1701b 100644 --- a/src/mux/decode_ts/tests.rs +++ b/src/mux/decode_ts/tests.rs @@ -351,6 +351,69 @@ fn h264_paff_pairs_come_from_the_slice_header() { ); } +#[test] +fn start_up_spacing_is_floored_at_two_field_periods() { + // PAFF at R = 1 with no stated frame rate: T is the smallest first-field gap (1500), + // below 2 x the 1800-tick field delta, so the floor keeps the second field's start-up + // DTS below the next unit's. + let sps = H264Sps { + frame_mbs_only: false, + height_map_units: 34, + vui: Some(H264Vui { + restriction: Some((1, 2)), + ..Default::default() + }), + ..Default::default() + }; + let (gap, field) = (1_500, 1_800); + let mut frames = Vec::new(); + for u in 0..4i64 { + for bottom in [false, true] { + let mut es = if u == 0 && !bottom { + length_prefixed(&[sps.nal()]) + } else { + Vec::new() + }; + let slice = h264_slice(&sps, u == 0 && !bottom, 0, u as u32, Some(bottom)); + es.extend(length_prefixed(&[slice])); + frames.push((B + u * gap + bottom as i64 * field, es)); + } + } + // The start-up window: first field, its second field, and the next unit. + frames.truncate(3); + let mut d = DtsDeriver::for_codec(Codec::H264, None); + let dts = run(&mut d, &frames); + assert_eq!(dts[0], Some(B - 2 * field)); + assert_eq!(d.counters(), DtsCounters::default(), "no order violation"); + assert_conformant(&frames, &dts); +} + +#[test] +fn reorder_depth_is_clamped_and_the_order_statistic_window_is_exact_at_the_clamp() { + // A corrupt SPS claiming 200 reordered frames is held to MAX_REORDER. + let sps = sps_timed(200); + let d = DtsDeriver::for_codec(Codec::H264, Some(&avcc(&sps.nal()))); + assert_eq!(d.params().0, MAX_REORDER); + + // In-order delivery at R = 16 over more than KEEP units: DTS_k is the PTS of unit k - 16. + let sps = sps_timed(MAX_REORDER as u32); + let mut d = DtsDeriver::for_codec(Codec::H264, Some(&avcc(&sps.nal()))); + let n = 3 * KEEP as i64; + let frames: Vec<_> = (0..n) + .map(|k| (B + k * F, h264_frame(&sps, k == 0, k as u32))) + .collect(); + let dts = run(&mut d, &frames); + let want: Vec<_> = (0..n) + .map(|k| Some(B + (k - MAX_REORDER as i64) * F)) + .collect(); + assert_eq!(dts, want); + assert_eq!( + d.core.top.len(), + KEEP, + "only KEEP unit PTS values are retained" + ); +} + #[test] fn h264_field_coded_r_upgrade_allots_one_slot_per_field() { // 25 fps PAFF: clip 1 R = 0 (I/P), clip 2 IDR activates R = 2 (B-pyramid). The diff --git a/src/mux/demux_sink.rs b/src/mux/demux_sink.rs index 0061b814..703631ad 100644 --- a/src/mux/demux_sink.rs +++ b/src/mux/demux_sink.rs @@ -396,8 +396,10 @@ impl EsWriter for PgsSupWriter { let pts90 = ns_to_90k(pts_ns); let is_pcs = f.data.first() == Some(&SEG_PCS) && f.data.len() > 13; let mut written = 0; + // An authored clear states the real end, even past a duration the parser capped. + let is_clear = is_pcs && f.data[13] == 0; if let Some((end, width, height)) = self.pending_clear { - if end < pts_ns || (end == pts_ns && !is_pcs) { + if (end < pts_ns && !is_clear) || (end == pts_ns && !is_pcs) { let clear = Self::synthetic_clear_display_set(width, height); let pts = ns_to_90k(end); written += Self::emit_segments(&clear, pts, pts, w)?.0; @@ -611,7 +613,7 @@ pub(crate) fn chapters_ogm(chapters: &[Chapter]) -> String { let name = if c.name.is_empty() { n.to_string() } else { - c.name.clone() + escape_controls(&c.name, &[]) }; s.push_str(&format!("CHAPTER{n:02}={h:02}:{m:02}:{sec:02}.{ms:03}\n")); s.push_str(&format!("CHAPTER{n:02}NAME={name}\n")); @@ -620,11 +622,28 @@ pub(crate) fn chapters_ogm(chapters: &[Chapter]) -> String { } fn xml_escape(s: &str) -> String { - s.replace('&', "&") + // XML 1.0 forbids C0 controls other than tab / LF / CR. + escape_controls(s, &['\t', '\n', '\r']) + .replace('&', "&") .replace('<', "<") .replace('>', ">") } +// Disc-derived text in a line-oriented or XML file: control chars (and U+FFFE / U+FFFF) +// outside `allow` are written in their `escape_debug()` form, as `Error::DirNameCollision` +// does, instead of raw. +fn escape_controls(s: &str, allow: &[char]) -> String { + let mut out = String::with_capacity(s.len()); + for c in s.chars() { + if (c.is_control() && !allow.contains(&c)) || matches!(c, '\u{FFFE}' | '\u{FFFF}') { + out.extend(c.escape_debug()); + } else { + out.push(c); + } + } + out +} + // Replace path-hostile characters (control chars incl. NUL) in a filename component; // disc-derived text (e.g. STN language codes) is unvalidated. fn sanitize(s: &str) -> String { @@ -1521,6 +1540,84 @@ mod tests { } } + // The parser caps a display at 30 s; an authored clear later than that is the real end. + #[test] + fn pgs_sup_authored_clear_beyond_a_capped_duration_is_not_preempted() { + let mut writer = PgsSupWriter::default(); + let mut bytes = Vec::new(); + for frame in [ + sup_frame(0, Some(30_000_000_000), true), + sup_frame(40_000_000_000, Some(0), false), + ] { + writer.write_frame(&mut bytes, &frame, frame.pts).unwrap(); + } + writer.finish(&mut bytes).unwrap(); + assert_eq!(sup_compositions(&bytes), [(0, 1), (3_600_000, 0)]); + } + + // A continuation segment stamped exactly at the wipe time follows the clear, not precedes it. + #[test] + fn pgs_sup_clear_precedes_a_non_pcs_frame_at_the_wipe_time() { + let mut writer = PgsSupWriter::default(); + let mut bytes = Vec::new(); + let display = sup_frame(0, Some(2_000_000_000), true); + writer.write_frame(&mut bytes, &display, 0).unwrap(); + let mut end = sup_frame(2_000_000_000, None, false); + end.data = vec![SEG_END, 0, 0]; + writer.write_frame(&mut bytes, &end, end.pts).unwrap(); + writer.finish(&mut bytes).unwrap(); + let mut types = Vec::new(); + let mut pos = 0; + while pos < bytes.len() { + let size = usize::from(u16::from_be_bytes([ + bytes[pos + SUP_HEADER_LEN + 1], + bytes[pos + SUP_HEADER_LEN + 2], + ])); + types.push(bytes[pos + SUP_HEADER_LEN]); + pos += SUP_HEADER_LEN + PGS_SEG_HEADER_LEN + size; + } + // display PCS + END, clear PCS + END, then the continuation END. + assert_eq!(types, [SEG_PCS, SEG_END, SEG_PCS, SEG_END, SEG_END]); + } + + // sub:// / demux:// finish() must flush the writer so the last subtitle is cleared. + #[test] + fn sink_finish_writes_the_last_pgs_subtitles_synthetic_clear() { + let dir = tempdir(); + let title = title_with(vec![subtitle_stream(Codec::Pgs, "eng")], vec![None]); + let opts = DemuxOptions { + base: "Sub".to_string(), + export_chapters: false, + ..Default::default() + }; + let mut sink = DemuxSink::create(&dir, &title, &opts).unwrap(); + sink.write(&sup_frame(0, Some(2_000_000_000), true)) + .unwrap(); + sink.finish().unwrap(); + let sup = std::fs::read_dir(&dir) + .unwrap() + .map(|e| e.unwrap().path()) + .find(|p| p.extension().is_some_and(|e| e == "sup")) + .expect("a .sup file"); + let bytes = std::fs::read(&sup).unwrap(); + assert_eq!(sup_compositions(&bytes), [(0, 1), (180_000, 0)]); + let _ = std::fs::remove_dir_all(&dir); + } + + // Hostile chapter names must not forge OGM lines or make the XML ill-formed. + #[test] + fn chapter_names_with_control_chars_are_neutralised() { + let chaps = vec![Chapter { + time_secs: 0.0, + name: "x\nCHAPTER99=00:00:00.000\u{1}y".to_string(), + }]; + let ogm = chapters_ogm(&chaps); + assert_eq!(ogm.lines().count(), 2, "{ogm:?}"); + assert!(!ogm.lines().any(|l| l.starts_with("CHAPTER99"))); + let xml = chapters_xml(&chaps); + assert!(!xml.contains('\u{1}'), "{xml:?}"); + } + #[test] fn pgs_sup_legacy_duration_only_input_clears_before_a_later_display() { let mut writer = PgsSupWriter::default(); diff --git a/src/mux/demux_thread.rs b/src/mux/demux_thread.rs index 44abdf1e..8adb96ff 100644 --- a/src/mux/demux_thread.rs +++ b/src/mux/demux_thread.rs @@ -545,6 +545,55 @@ mod tests { assert!(saw, "PS PES must be demuxed and delivered"); } + // A PS input commonly ends with no program-end code: the last PES is only completed by + // flush() and must still be delivered, before the Eof, and every PES is stamped with its + // position in the whole input, across buffers. + #[test] + fn ps_tail_without_a_program_end_is_flushed_with_its_source() { + let (pf_tx, pf_rx) = bounded::>>(4); + let (rc_tx, _rc_rx) = bounded::>(4); + let ps = super::super::ps::PsDemuxer::new(); + let (_dt, rx) = DemuxThread::spawn_zero_copy( + pf_rx, + rc_tx, + (), + &crate::ctx::Ctx::default(), + None, + Some(ps), + ) + .unwrap(); + + // A bounded PES (11 bytes), then, in the next buffer, an unbounded one. + let first = vec![ + 0x00, 0x00, 0x01, 0xE0, 0x00, 0x05, 0x80, 0x00, 0x00, 0x77, 0x88, + ]; + let second = vec![ + 0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x00, 0x00, 0x99, 0xAA, + ]; + pf_tx.send(Ok(first.clone())).unwrap(); + pf_tx.send(Ok(second)).unwrap(); + drop(pf_tx); + + let batches = collect_batches(&rx, Duration::from_secs(5)); + assert!(matches!(batches.last(), Some(DemuxBatch::Eof))); + let got: Vec<(Vec, u64)> = batches + .iter() + .filter_map(|b| match b { + DemuxBatch::Ps(p) => Some(p), + _ => None, + }) + .flatten() + .map(|x| (x.data.clone(), x.source.expect("stamped").byte)) + .collect(); + assert_eq!( + got, + vec![ + (vec![0x77, 0x88], 0), + (vec![0x99, 0xAA], first.len() as u64) + ] + ); + } + #[test] fn no_demuxer_configured_still_recycles_and_eofs() { // With neither ts nor ps set, the worker must still recycle buffers diff --git a/src/mux/driver.rs b/src/mux/driver.rs index 8e5f553d..2d9a8ba7 100644 --- a/src/mux/driver.rs +++ b/src/mux/driver.rs @@ -164,14 +164,19 @@ pub fn mux_url( ) -> std::io::Result { let source = match super::source::open_source(url, crate::disc::ScanOptions::default(), ctx) { Ok(s) => s, - Err(Error::Halted) if ctx.halt.is_cancelled() => { - return Ok(MuxOutcome::stopped_before_output(0, 0)); - } - Err(e) => return Err(e.into()), + Err(e) => return open_failed(e, ctx), }; mux_with_keys(source, keys, dest_url, opts, ctx) } +// A failed open: the stopped outcome when it is a Stop, the error otherwise. +fn open_failed(e: Error, ctx: &Ctx) -> std::io::Result { + match e { + Error::Halted if ctx.halt.is_cancelled() => Ok(MuxOutcome::stopped_before_output(0, 0)), + e => Err(e.into()), + } +} + // `mux_with_keys` before its one halt rule. fn mux_routed( source: Source, @@ -251,7 +256,7 @@ pub(crate) fn open_pes( match set.or(keyless.as_ref()) { Some(set) => { let title = live_keyed_title(&*reader, title, set, opts)?; - live_keyed(reader, title, format, set, opts, ctx) + live_keyed(reader, title, set, opts, ctx) } None => { let mut title = title; @@ -344,7 +349,7 @@ fn session_pes( let staged = session.staged_reader().ok_or_else(not_ready)?; let title = live_keyed_title(staged, title, set, opts)?; let reader = session.take_reader().ok_or_else(not_ready)?; - live_keyed(reader, title, format, set, opts, ctx) + live_keyed(reader, title, set, opts, ctx) } None => { let mut title = title; @@ -411,7 +416,6 @@ fn live_keyed_title( fn live_keyed( reader: Box, title: DiscTitle, - _format: crate::disc::ContentFormat, set: &crate::keys::KeyRing, opts: &MuxOptions, ctx: &Ctx, @@ -998,6 +1002,19 @@ impl WriteSink { }) } + // End an output that `cause` cut short. The cause is the title's verdict: ending the + // output can fail (an mkv with no frames is `MkvInvalid`), which is logged, never returned. + fn end_cut_short(self, cause: &str) -> Result { + let bytes = self.bytes.clone(); + self.end(false).or_else(|e| { + tracing::warn!(target: "mux", error = %e, "ending the output {cause} cut short failed"); + Ok(SinkClose { + bytes: bytes.load(Ordering::Relaxed), + undelivered: Vec::new(), + }) + }) + } + fn apply_late_configs(&mut self) -> Result<(), Error> { let late = std::mem::take(&mut *lock_late(&self.late_configs)); for (track, cp) in late { @@ -1026,13 +1043,15 @@ impl Sink for WriteSink { } fn close(self) -> Result { - let complete = !self.read_failed.load(Ordering::Relaxed); - self.end(complete) + match self.read_failed.load(Ordering::Relaxed) { + true => self.end_cut_short("a read failure"), + false => self.end(true), + } } // A stopped title is incomplete too: a wire sink must not end it cleanly. fn close_stopped(self) -> Result { - self.end(false) + self.end_cut_short("a stop") } } @@ -1146,6 +1165,38 @@ mod tests { assert!(log.is_empty(), "a failed sink is never finalised: {log:?}"); } + // A strict read failure before any frame lands: the mkv's zero-frame `MkvInvalid` on + // ending the output is a skippable stub code and must not replace the read error. + #[test] + fn a_read_failure_before_any_frame_is_not_a_skippable_stub() { + let dir = tempfile::tempdir().expect("tempdir"); + let url = format!("mkv://{}", dir.path().join("o.mkv").display()); + let mut fs = FakeStream::new(1).with_frames(10); + fs.fail_read_at = Some(0); + let ctx = crate::ctx::Ctx::new(Halt::new()); + let err = drive_mux(Box::new(fs), &url, &ctx, None, None).expect_err("a read failure"); + assert_eq!( + crate::error::error_code(&err), + Some(crate::error::E_DISC_READ), + "got {err}" + ); + assert!(!crate::error::is_skippable_title_stub(&err)); + } + + // A Stop after the mkv output opens but before any frame reaches it: a halted outcome, + // never the zero-frame `MkvInvalid` from ending the empty output. + #[test] + fn a_stop_before_any_frame_reaches_an_open_mkv_is_halted_not_an_error() { + let dir = tempfile::tempdir().expect("tempdir"); + let url = format!("mkv://{}", dir.path().join("o.mkv").display()); + let halt = Halt::new(); + let fs = FakeStream::new(1).with_frames(10).cancels(halt.clone(), 0); + let ctx = crate::ctx::Ctx::new(halt.clone()); + let o = drive_mux(Box::new(fs), &url, &ctx, None, None).expect("a stop is not an error"); + assert!(o.output_opened && !o.completed && o.halted, "{o:?}"); + assert_eq!(o.bytes_written, 0); + } + // A stop ends the output incomplete; a clean drain finishes it. #[test] fn only_a_clean_drain_finishes_the_output() { @@ -1401,6 +1452,72 @@ mod tests { } } + // A never-syncing TrueHD title with `n` non-sync frames of `size` bytes each. + fn unsynced_truehd_stream(n: usize, size: usize) -> FakeStream { + use crate::disc::{AudioChannels, AudioStream, Codec, LabelPurpose, SampleRate, Stream}; + let mut s = FakeStream::new(0); + s.info.streams = vec![Stream::Audio(AudioStream { + pid: 0x1100, + codec: Codec::TrueHd, + channels: AudioChannels::Surround51, + language: "eng".into(), + sample_rate: SampleRate::S48, + secondary: false, + purpose: LabelPurpose::Normal, + label: String::new(), + })]; + for pts in 0..n as i64 { + s.frames.push_back(PesFrame { + discard_padding_ns: 0, + track: 0, + pts, + keyframe: true, + data: vec![0u8; size], + duration_ns: None, + source: None, + coding: None, + }); + } + s + } + + // A Stop during the TrueHD probe ends the mux before any output opens, for both a + // frame sink and a metadata sink, whether it shows as a cancelled halt or Err(Halted). + #[test] + fn a_stop_during_the_truehd_probe_stops_before_the_output_opens() { + let dir = tempfile::tempdir().unwrap(); + let json = format!("json://{}", dir.path().join("t.json").display()); + for dest in ["null://", json.as_str()] { + let halt = Halt::new(); + let s = unsynced_truehd_stream(5, 64).cancels(halt.clone(), 1); + let out = drive_mux(Box::new(s), dest, &Ctx::new(halt), None, None).unwrap(); + assert!(out.halted && !out.completed && !out.output_opened, "{dest}"); + + let s = unsynced_truehd_stream(5, 64).halt_errs_at(1); + let out = drive_mux(Box::new(s), dest, &Ctx::default(), None, None).unwrap(); + assert!(!out.completed && !out.output_opened, "{dest} Err(Halted)"); + } + } + + // The probe reads at most TRUEHD_PROBE_BYTES before the output opens; a title that never + // syncs is not buffered whole. + #[test] + fn the_truehd_probe_is_byte_capped() { + let mut s = unsynced_truehd_stream(20, 1 << 20); + let reads = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + s.read_observer = Some(reads.clone()); + let at_open = Arc::new(std::sync::atomic::AtomicUsize::new(usize::MAX)); + let (seen, counter) = (at_open.clone(), reads.clone()); + let ctx = Ctx::default().with_events(Arc::new(move |e: &Event<'_>| { + if let Event::OutputOpened { .. } = e { + seen.store(counter.load(Ordering::SeqCst), Ordering::SeqCst); + } + })); + drive_mux(Box::new(s), "null://", &ctx, None, None).unwrap(); + let n = at_open.load(Ordering::SeqCst); + assert!(n <= 10, "{n} MiB frames read before the output opened"); + } + fn mp2_ext_title() -> DiscTitle { use crate::disc::{AudioChannels, AudioStream, Codec, LabelPurpose, SampleRate, Stream}; let audio = |pid, label: &str| { @@ -1680,6 +1797,22 @@ mod tests { assert!(out.halted && !out.completed && !out.output_opened); } + // A Stop landing in `open_source` (the scan / bring-up) is the documented stopped + // outcome from `mux_url`, not an `E_HALTED` error; any other open failure stays an error. + #[test] + fn a_stop_during_mux_urls_open_is_a_halted_outcome() { + let ctx = Ctx::default(); + ctx.halt.cancel(); + let out = open_failed(Error::Halted, &ctx).expect("a Stop is not an error"); + assert!(out.halted && !out.completed && !out.output_opened); + assert!(open_failed(Error::NoStreams, &ctx).is_err()); + let live = Ctx::default(); + assert!( + open_failed(Error::Halted, &live).is_err(), + "Err(Halted) with no Stop requested is not a stopped outcome" + ); + } + // ── halt mid-pump stops cleanly with completed=false, no panic. ── #[test] fn halt_mid_pump_stops_cleanly() { @@ -1700,6 +1833,10 @@ mod tests { .expect("halt is a clean stop, not an error"); assert!(!out.completed, "an interrupted mux is not complete"); assert!(out.output_opened, "the sink was opened before the halt"); + assert!( + out.halted, + "a Stop is reported as halted so callers map it to Cancelled" + ); } // A halt that ends the stream as Ok(None) can also release the header gate @@ -1740,6 +1877,7 @@ mod tests { None, ) .expect("a halt mid frame-read is a clean stop, not an Err"); + assert!(!out.halted, "halt was never cancelled: not a Stop"); assert!(!out.completed, "interrupted mux is not complete"); assert!(out.output_opened, "sink opened before the mid-read halt"); } diff --git a/src/mux/ebml.rs b/src/mux/ebml.rs index a8396f8d..4f5e1693 100644 --- a/src/mux/ebml.rs +++ b/src/mux/ebml.rs @@ -611,7 +611,7 @@ pub const CHAPTER_UID: u32 = 0x73C4; pub const CHAPTER_TIME_START: u32 = 0x91; pub const CHAPTER_DISPLAY: u32 = 0x80; pub const CHAP_STRING: u32 = 0x85; -pub const EDITION_FLAG_DEFAULT: u32 = 0xDB; +pub const EDITION_FLAG_DEFAULT: u32 = 0x45DB; pub const CHAPTER_FLAG_HIDDEN: u32 = 0x98; pub const CHAPTER_FLAG_ENABLED: u32 = 0x4598; pub const CHAP_LANGUAGE: u32 = 0x437C; diff --git a/src/mux/fvi_sink.rs b/src/mux/fvi_sink.rs index ad5d89f6..bbf58afc 100644 --- a/src/mux/fvi_sink.rs +++ b/src/mux/fvi_sink.rs @@ -338,6 +338,24 @@ mod tests { assert_eq!(n, [0, 1]); } + // Known provenance (playlist, volume id) and picture count are carried into the header. + #[test] + fn header_carries_playlist_volume_id_and_picture_count_when_known() { + let source = SourceInfo { + playlist: "00800.mpls".into(), + volume_id: "DISC_LABEL".into(), + ..SourceInfo::default() + }; + let mut header = MapHeader::from_title(&mpeg2_title(), source); + header.picture_count = Some(7); + let mut out = Vec::new(); + write_fvi_header(&mut out, &header).unwrap(); + let v: serde_json::Value = serde_json::from_slice(&out).unwrap(); + assert_eq!(v["source"]["playlist"], "00800.mpls"); + assert_eq!(v["source"]["volume_id"], "DISC_LABEL"); + assert_eq!(v["picture_count"], 7); + } + #[test] fn sink_writes_header_and_only_video_records() { let dir = tempfile::tempdir().unwrap(); diff --git a/src/mux/header_gate.rs b/src/mux/header_gate.rs index 55a85871..798ad42e 100644 --- a/src/mux/header_gate.rs +++ b/src/mux/header_gate.rs @@ -183,6 +183,32 @@ mod tests { assert!(waiting(&g)); } + // After a forward discontinuity the clock rebases: normal PTS past the jump counts again. + #[test] + fn the_clock_runs_again_after_a_forward_pts_jump() { + let mut g = HeaderGate::default(); + for pts in [0, S, 100 * S, 101 * S, 102 * S, 103 * S, 104 * S] { + g.account(0, pts, 0); + } + assert!(!waiting(&g), "5 s of content on either side of the jump"); + } + + // A subtitle track has no codec_private and never holds the headers. + #[test] + fn subtitle_tracks_never_gate() { + let mut t = DiscTitle::empty(); + t.streams + .push(Stream::Subtitle(crate::disc::SubtitleStream { + pid: 0x1200, + codec: Codec::Pgs, + language: "eng".into(), + forced: false, + qualifier: crate::disc::LabelQualifier::None, + codec_data: None, + })); + assert!(HeaderGate::default().ready(&t, |_| None)); + } + // Decode-order reordering (B-frames) must not inflate elapsed time. #[test] fn reordered_pts_count_once() { diff --git a/src/mux/hevc/mod.rs b/src/mux/hevc/mod.rs index 3763cc30..a303c95e 100644 --- a/src/mux/hevc/mod.rs +++ b/src/mux/hevc/mod.rs @@ -495,6 +495,23 @@ mod tests { assert_eq!(&out[..], &[0x00, 0x00, 0x00, 0x01, 0x40, 0x01]); } + // An hvcC ending 1-2 bytes into the next array header is truncated, not read past its end. + #[test] + fn hvcc_cut_inside_an_array_header_is_truncated_not_out_of_bounds() { + for tail in [&[32u8][..], &[32, 0]] { + let mut hvcc = vec![0u8; 22]; + hvcc.push(2); // numArrays = 2 + hvcc.push(32); + hvcc.extend_from_slice(&1u16.to_be_bytes()); + hvcc.extend_from_slice(&2u16.to_be_bytes()); + hvcc.extend_from_slice(&[0x40, 0x01]); + hvcc.extend_from_slice(tail); + let (out, truncated) = hvcc_parse(&hvcc); + assert!(truncated, "tail {tail:?}"); + assert_eq!(out.unwrap(), [0x00, 0x00, 0x00, 0x01, 0x40, 0x01]); + } + } + // --- length_prefixed_to_annex_b additional branches --- #[test] diff --git a/src/mux/m2ts.rs b/src/mux/m2ts.rs index bd76d31f..8e6917ba 100644 --- a/src/mux/m2ts.rs +++ b/src/mux/m2ts.rs @@ -162,8 +162,7 @@ impl M2tsStream { Repack::Adts(h) } DiscStream::Audio(a) if a.codec == crate::disc::Codec::Lpcm => { - let h = lpcm_bd_header(a, cp.as_deref()); - if h.is_none() { + let Some(h) = lpcm_bd_header(a, cp.as_deref()) else { tracing::warn!( target: "mux", track = i, @@ -171,11 +170,11 @@ impl M2tsStream { ); route.push(None); continue; - } + }; // Advertise the layout actually packed, not a rejected source byte. let depth = super::codec::lpcm::output_depth(cp.as_deref()); - cp = h.map(|h| super::codec::lpcm::tagged_layout(h[0], depth)); - h.map_or(Repack::Verbatim, Repack::Lpcm) + cp = Some(super::codec::lpcm::tagged_layout(h[0], depth)); + Repack::Lpcm(h) } _ => Repack::Verbatim, }; @@ -1176,4 +1175,116 @@ mod tests { fn mvc_dependent_view_arriving_first_still_matches_the_base_dts() { mvc_dependent_dts(true); } + + // An AAC track whose config cannot be signalled in ADTS is left out, not written as raw AUs. + #[test] + fn aac_without_a_usable_config_is_omitted() { + let mut title = make_title(); + title.streams.push(audio(0x1100, Codec::Aac)); + title.codec_privates.push(None); + let data = [fake_idr_pes_data(), vec![0x21, 0x10, 0x04]]; + let (pids, ts) = mux_one_frame_each(&title, &data); + assert_eq!(pids.len(), 1, "only the video track is advertised"); + assert!( + !ts.as_chunks::<192>().0.iter().any(|p| p[5] & 0x1F == 0x11), + "no packets on the AAC PID" + ); + } + + // Two tracks asking for one PID are not interleaved on it: the second moves. + #[test] + fn tracks_sharing_a_pid_get_distinct_pids() { + let mut title = make_title(); + title.streams = vec![ac3_audio(0x1100), ac3_audio(0x1100)]; + title.codec_privates = vec![None, None]; + let a = vec![0x0B, 0x77, 1, 2]; + let b = vec![0x0B, 0x77, 3, 4]; + let (pids, ts) = mux_one_frame_each(&title, &[a.clone(), b.clone()]); + assert_ne!(pids[0], pids[1], "{pids:x?}"); + let mut demux = crate::mux::ts::TsDemuxer::new(&pids); + let mut pes = demux.feed(&ts); + pes.extend(demux.flush()); + let got = |pid| pes.iter().find(|p| p.pid == pid).map(|p| p.data.clone()); + assert_eq!(got(pids[0]), Some(a)); + assert_eq!(got(pids[1]), Some(b)); + } + + // An MPEG-2 multichannel extension has no BD-TS binding: never written, reported lost + // once its packets arrive. + #[test] + fn an_mp2_extension_track_is_left_out_and_reported() { + let mut title = make_title(); + let DiscStream::Audio(base) = audio(0x00C0, Codec::Mp2) else { + unreachable!() + }; + let ext = crate::disc::AudioStream { + pid: 0x00D0, + label: crate::disc::MP2_EXTENSION_LABEL.into(), + ..base.clone() + }; + title.streams.push(DiscStream::Audio(base)); + title.streams.push(DiscStream::Audio(ext)); + title.codec_privates = vec![None; 3]; + let shared = std::sync::Arc::new(std::sync::Mutex::new(Vec::::new())); + let mut stream = M2tsStream::create(SharedSink(shared.clone()), &title).unwrap(); + assert!( + stream.undelivered_streams().is_empty(), + "nothing arrived yet" + ); + stream + .write(&frame(0, 0, true, fake_idr_pes_data())) + .unwrap(); + stream + .write(&frame(2, 0, true, vec![0x7F, 0xF0, 1, 2])) + .unwrap(); + assert_eq!(stream.undelivered_streams(), vec![2]); + stream.finish().unwrap(); + drop(stream); + let (meta, _) = ts_after_header(&shared.lock().unwrap()); + assert_eq!(meta.streams.len(), 2, "video and the base only"); + } + + // The PMT stream_type of each codec reads back as that codec. + #[test] + fn stream_types_read_back_as_their_codecs() { + let mut title = make_title(); + let codecs = [ + Codec::Ac3, + Codec::Dts, + Codec::TrueHd, + Codec::Ac3Plus, + Codec::DtsHdHr, + Codec::DtsHdMa, + ]; + for (k, c) in codecs.into_iter().enumerate() { + title.streams.push(audio(0x1100 + k as u16, c)); + } + title + .streams + .push(DiscStream::Subtitle(crate::disc::SubtitleStream { + pid: 0x1200, + codec: Codec::Pgs, + language: "eng".into(), + forced: false, + qualifier: crate::disc::LabelQualifier::None, + codec_data: None, + })); + title.codec_privates = vec![None; title.streams.len()]; + let mut data = vec![fake_idr_pes_data()]; + data.extend((0..codecs.len() + 1).map(|_| vec![0x0B, 0x77, 1, 2])); + let (_, ts) = mux_one_frame_each(&title, &data); + let got: Vec = crate::mux::ts::scan_streams(&ts) + .expect("PAT/PMT present") + .iter() + .map(|s| match s { + DiscStream::Video(v) => v.codec, + DiscStream::Audio(a) => a.codec, + DiscStream::Subtitle(t) => t.codec, + }) + .collect(); + let mut want = vec![Codec::Hevc]; + want.extend(codecs); + want.push(Codec::Pgs); + assert_eq!(got, want); + } } diff --git a/src/mux/meta.rs b/src/mux/meta.rs index 5f3d71f9..3f341ee7 100644 --- a/src/mux/meta.rs +++ b/src/mux/meta.rs @@ -675,6 +675,94 @@ mod tests { assert_eq!(cursor.position() as usize % BD_SOURCE_PACKET_BYTES, 0); } + // Every stream field the receiver acts on survives an FMKV hop (network://, stdio://, m2ts://). + #[test] + fn stream_fields_round_trip_through_the_header() { + use crate::disc::{ + AudioChannels, AudioStream, LabelPurpose, LabelQualifier, MVC_DEPENDENT_LABEL, + SampleRate, SubtitleStream, + }; + let mut t = video_title(HdrFormat::Hdr10, ColorSpace::Bt2020); + if let Stream::Video(v) = &mut t.streams[0] { + v.label = MVC_DEPENDENT_LABEL.into(); + v.secondary = true; + } + for (i, purpose) in [ + LabelPurpose::Commentary, + LabelPurpose::Descriptive, + LabelPurpose::Score, + LabelPurpose::Ime, + ] + .into_iter() + .enumerate() + { + t.streams.push(Stream::Audio(AudioStream { + pid: 0x1100 + i as u16, + codec: Codec::Dts, + channels: AudioChannels::Surround71, + language: "fra".into(), + sample_rate: SampleRate::S96, + secondary: true, + purpose, + label: format!("audio {i}"), + })); + } + for qualifier in [ + LabelQualifier::Sdh, + LabelQualifier::DescriptiveService, + LabelQualifier::Forced, + ] { + t.streams.push(Stream::Subtitle(SubtitleStream { + pid: 0x1200, + codec: Codec::Pgs, + language: "deu".into(), + forced: true, + qualifier, + codec_data: None, + })); + } + let back = M2tsMeta::from_title(&t).to_title(); + let Stream::Video(v) = &back.streams[0] else { + panic!("video first") + }; + assert!(v.is_mvc_dependent()); + let dbg = |s: &crate::disc::Stream| format!("{s:?}"); + let (want, got): (Vec<_>, Vec<_>) = t + .streams + .iter() + .zip(&back.streams) + .map(|(a, b)| (dbg(a), dbg(b))) + .unzip(); + assert_eq!(got, want); + } + + // The wire track is a u8: a header declaring more than 256 streams is refused. + #[test] + fn a_header_with_more_than_256_streams_is_refused() { + let title_of = |n: usize| { + let mut t = DiscTitle::empty(); + for pid in 0..n { + t.streams + .push(Stream::Subtitle(crate::disc::SubtitleStream { + pid: pid as u16, + codec: Codec::Pgs, + language: String::new(), + forced: false, + qualifier: crate::disc::LabelQualifier::None, + codec_data: None, + })); + } + t + }; + let read = |n| { + let mut buf = Vec::new(); + write_header(&mut buf, &M2tsMeta::from_title(&title_of(n))).unwrap(); + read_header(&mut io::Cursor::new(buf)) + }; + assert!(read(256).unwrap().is_some()); + assert!(read(257).is_err()); + } + #[test] fn audio_and_subtitle_codec_private_round_trip() { use crate::disc::{AudioChannels, AudioStream, LabelPurpose, SampleRate, SubtitleStream}; diff --git a/src/mux/meta_sink.rs b/src/mux/meta_sink.rs index 253c53cc..b789505f 100644 --- a/src/mux/meta_sink.rs +++ b/src/mux/meta_sink.rs @@ -362,6 +362,40 @@ mod tests { assert!(vtt.contains("a & <b>"), "{vtt}"); } + #[test] + fn vtt_times_carry_hours_minutes_and_seconds() { + assert_eq!(vtt_time(3725.5), "01:02:05.500"); + assert_eq!(vtt_time(36_000.0), "10:00:00.000"); + } + + // json:// marks the MVC dependent view and only that. + #[test] + fn json_marks_the_mvc_dependent_view() { + use crate::disc::{ + Codec, ColorSpace, DiscTitle, FrameRate, HdrFormat, MVC_DEPENDENT_LABEL, Resolution, + Stream as DiscStream, VideoStream, + }; + let video = |label: &str| { + DiscStream::Video(VideoStream { + pid: 0x1011, + codec: Codec::H264, + resolution: Resolution::R1080p, + frame_rate: FrameRate::F23_976, + hdr: HdrFormat::Sdr, + color_space: ColorSpace::Bt709, + display_aspect: None, + secondary: false, + label: label.into(), + measured_cicp: None, + }) + }; + let mut t = DiscTitle::empty(); + t.streams = vec![video(""), video(MVC_DEPENDENT_LABEL)]; + let v = title_json(&t); + assert_eq!(v["video"][0]["mvc"], false); + assert_eq!(v["video"][1]["mvc"], true); + } + #[test] fn chapters_format_selected_by_extension() { let xml = chapters_content(&chaps(), Some("xml")); diff --git a/src/mux/mkv.rs b/src/mux/mkv.rs index dff006f2..88145c3c 100644 --- a/src/mux/mkv.rs +++ b/src/mux/mkv.rs @@ -688,6 +688,51 @@ pub struct MkvMuxer { /// Per track: file offset of a Void reserved for a CodecPrivate that was not /// known at header time (AAC), filled by [`Self::set_codec_private`]. codec_private_reserves: std::collections::HashMap, + /// Per MPEG-2 video track: the DefaultDuration measured from the kept frames. + frame_rate_fixups: std::collections::HashMap, +} + +// Deferred DefaultDuration correction for an MPEG-2 video track: the header holds the +// sequence rate (29.97 for NTSC), but soft-telecined film is 23.976 coded frames a second. +// `finish()` rewrites it when the kept frames' PTS span measures another standard rate. +struct FrameRateFixup { + /// Absolute file offset of the DefaultDuration value, and its byte length. + value_offset: u64, + value_len: usize, + /// The value written up-front, in ns. + initial_ns: u64, + frames: u64, + min_pts_ns: i64, + max_pts_ns: i64, +} + +// Fewest kept frames whose PTS span is taken as a measured frame period. +const FRAME_RATE_MIN_FRAMES: u64 = 120; + +// The standard frame period (ns) within 0.5% of `measured`, if any. +fn standard_frame_period_ns(measured: f64) -> Option { + use crate::disc::FrameRate; + [ + FrameRate::F23_976, + FrameRate::F24, + FrameRate::F25, + FrameRate::F29_97, + FrameRate::F30, + FrameRate::F50, + FrameRate::F59_94, + FrameRate::F60, + ] + .iter() + .map(|r| { + let (num, den) = r.as_fraction(); + (1_000_000_000u64 * den as u64) / num as u64 + }) + .find(|&ns| (measured - ns as f64).abs() <= ns as f64 * 0.005) +} + +// Bytes of the shortest big-endian encoding `ebml::write_uint` gives `v`. +fn uint_len(v: u64) -> usize { + (8 - (v.leading_zeros() / 8) as usize).max(1) } // Bytes reserved for a late AAC CodecPrivate: ID (2) + size (1-2) + an ASC of @@ -991,6 +1036,8 @@ impl MkvMuxer { std::collections::HashMap::new(); let mut flag_interlaced_fixups: std::collections::HashMap = std::collections::HashMap::new(); + let mut frame_rate_fixups: std::collections::HashMap = + std::collections::HashMap::new(); // Per track: whether it emitted a conforming `mvcC` BlockAdditionMapping. // Filled below from the SAME built record that drives the CodecPrivate // mvcC extension, so the three MVC signals never diverge. @@ -1084,6 +1131,21 @@ impl MkvMuxer { ebml::DEFAULT_DURATION, track.default_duration_ns, )?; + if track.track_type == ebml::TRACK_TYPE_VIDEO && track.codec_id == ebml::CODEC_MPEG2 + { + let value_len = uint_len(track.default_duration_ns); + frame_rate_fixups.insert( + i, + FrameRateFixup { + value_offset: writer.stream_position()? - value_len as u64, + value_len, + initial_ns: track.default_duration_ns, + frames: 0, + min_pts_ns: i64::MAX, + max_pts_ns: i64::MIN, + }, + ); + } } // DefaultDecodedFieldDuration: production video always passes 0 here @@ -1329,6 +1391,7 @@ impl MkvMuxer { flag_interlaced_fixups, opening_capture: None, codec_private_reserves, + frame_rate_fixups, }) } @@ -1528,6 +1591,11 @@ impl MkvMuxer { } } } + if let Some(f) = self.frame_rate_fixups.get_mut(&track_idx) { + f.frames += 1; + f.min_pts_ns = f.min_pts_ns.min(pts_ns); + f.max_pts_ns = f.max_pts_ns.max(pts_ns); + } let raw_ticks = pts_ns / TIMESTAMP_SCALE_NS; // Clusters normally open on a video keyframe so Cues resolve to a seekable @@ -1951,6 +2019,20 @@ impl MkvMuxer { .collect::>(), )?; + // Correct an MPEG-2 track's DefaultDuration from its measured frame period. + let rate_patches: Vec<(u64, Vec)> = self + .frame_rate_fixups + .values() + .filter(|f| f.frames >= FRAME_RATE_MIN_FRAMES) + .filter_map(|f| { + let measured = (f.max_pts_ns - f.min_pts_ns) as f64 / (f.frames - 1) as f64; + let ns = standard_frame_period_ns(measured)?; + (ns != f.initial_ns && uint_len(ns) == f.value_len) + .then(|| (f.value_offset, ns.to_be_bytes()[8 - f.value_len..].to_vec())) + }) + .collect(); + self.patch_bytes(&rate_patches)?; + // Correct FlagInterlaced from the WHOLE-stream scan majority: the header // value came from only the FIRST coded picture. Rewrite the byte to the // dominant scan; on a demotion to progressive, Void the stale FieldOrder. @@ -2061,7 +2143,8 @@ impl MkvMuxer { None => return Err(crate::error::Error::MkvUnencodable.into()), }, ebml::CUES => cues_offset, - _ => 0, + // The SeekHead reserves only the targets above. + _ => return Err(crate::error::Error::MkvUnencodable.into()), }; self.writer .seek(std::io::SeekFrom::Start(fixup.value_offset))?; @@ -2975,6 +3058,87 @@ mod tests { ); } + // An NTSC MPEG-2 track declares 29.97 up-front; soft-telecined film (3:2 pulldown: frames + // 3 and 2 fields long) measures 23.976 coded frames a second, and DefaultDuration is + // rewritten to agree with the frame count. Real 29.97 video keeps its value. + #[test] + fn mpeg2_default_duration_follows_the_measured_frame_period() { + use std::sync::{Arc, Mutex}; + let field = 1_001_000_000_000i64 / 60_000; + for (pulldown, want) in [(true, 41_708_333u64), (false, 33_366_666)] { + let shared = Arc::new(Mutex::new(Cursor::new(Vec::new()))); + let mut track = make_video_track(); + track.codec_id = ebml::CODEC_MPEG2; + track.codec_private = None; + track.default_duration_ns = 33_366_666; + let mut muxer = + MkvMuxer::new(SharedWriter(shared.clone()), &[track], None, 0.0, &[]).unwrap(); + let mut pts = 0i64; + for i in 0..240u32 { + muxer + .write_frame_at( + 0, + pts, + i % 12 == 0, + &[0x01, i as u8], + None, + None, + None, + None, + ) + .unwrap(); + let fields = if pulldown && i % 2 == 0 { 3 } else { 2 }; + pts += field * fields; + } + muxer.finish().unwrap(); + let data = shared.lock().unwrap().clone().into_inner(); + let body = track_entry_child_body(&data, ebml::DEFAULT_DURATION).unwrap(); + let got = body.iter().fold(0u64, |a, &b| (a << 8) | b as u64); + assert_eq!(got, want, "pulldown {pulldown}"); + } + } + + // A progressive leader on an interlaced MPEG-2 title leaves the reserved FieldOrder to be + // filled from the majority field order: TFF and BFF titles must not swap. + #[test] + fn reserved_field_order_follows_the_measured_majority() { + use crate::mux::codec::FieldOrder; + use std::sync::{Arc, Mutex}; + for (scan, want) in [ + (FieldOrder::Tff, ebml::FIELD_ORDER_TFF), + (FieldOrder::Bff, ebml::FIELD_ORDER_BFF), + ] { + let shared = Arc::new(Mutex::new(Cursor::new(Vec::new()))); + let mut track = make_video_track(); + track.codec_id = ebml::CODEC_MPEG2; + let mut muxer = + MkvMuxer::new(SharedWriter(shared.clone()), &[track], None, 0.0, &[]).unwrap(); + let scans = [FieldOrder::Progressive, scan, scan, scan]; + for (i, scan) in scans.into_iter().enumerate() { + let pts = i as i64 * 40_000_000; + muxer + .write_frame_at( + 0, + pts, + i == 0, + &[1, 2, i as u8], + None, + None, + None, + Some(scan), + ) + .unwrap(); + } + muxer.finish().unwrap(); + let data = shared.lock().unwrap().clone().into_inner(); + let at = data + .windows(2) + .position(|w| w == [ebml::FIELD_ORDER as u8, 0x81]) + .expect("FieldOrder element present"); + assert_eq!(data[at + 2], want); + } + } + #[test] fn mkv_write_frame_creates_cluster() { let buf = Cursor::new(Vec::new()); @@ -5528,6 +5692,27 @@ mod tests { assert_eq!(groups[0].duration, 29_127, "SPU stop time, not a 1 s guess"); } + // The stop time is found past every command that precedes it in the first DCSQ. + #[test] + fn vobsub_stop_is_found_after_commands_with_arguments() { + let mut track = make_subtitle_track(); + track.codec_id = ebml::CODEC_VOBSUB; + let mut spu = vec![0, 0, 0, 6, 0xAB, 0xCD]; + // delay 0x0100, next = itself; STA_DSP, SET_COLOR, SET_CONTR, SET_DAREA, SET_DSPXA, STP_DSP. + spu.extend_from_slice(&[0x01, 0x00, 0, 6, 0x01]); + spu.extend_from_slice(&[0x03, 0x11, 0x22, 0x04, 0x33, 0x44]); + spu.extend_from_slice(&[0x05, 1, 2, 3, 4, 5, 6, 0x06, 1, 2, 3, 4, 0x02, 0xFF]); + let len = spu.len() as u16; + spu[..2].copy_from_slice(&len.to_be_bytes()); + let data = mux_with_durations(&[track], &[(0, 0, true, spu, None)]); + let groups = all_block_groups(&data); + assert_eq!(groups.len(), 1); + assert_eq!( + groups[0].duration, 29_127, + "the real stop, not the 10 s cap" + ); + } + // A VobSub SPU with no stop command ends at the next SPU on its track, else // after a 10 s cap. #[test] @@ -5557,10 +5742,9 @@ mod tests { ); let groups = all_block_groups(&data); assert_eq!(groups.len(), 1); - assert!( - groups[0].duration > 10_000, - "a self-terminating cue must not be hidden after 1 s (got {} ticks)", - groups[0].duration + assert_eq!( + groups[0].duration, 300_000, + "a self-terminating cue lasts the 30 s fallback, not 1 s" ); } @@ -5703,6 +5887,41 @@ mod tests { assert!(find_id(&data, ebml::CHANNELS).is_some(), "Channels present"); } + // A late AAC CodecPrivate fills the 16-byte reserve exactly, whatever its length: the + // TrackEntry still parses and the bytes after it are untouched. + #[test] + fn late_aac_codec_private_fills_the_reserve_exactly() { + for len in [2usize, 11, 12, 13] { + let mut aac = make_audio_track(); + aac.codec_id = ebml::CODEC_AAC; + let mut muxer = MkvMuxer::new(Cursor::new(Vec::new()), &[aac], None, 0.0, &[]).unwrap(); + let cp: Vec = (1..=len as u8).collect(); + assert!(muxer.set_codec_private(0, &cp).unwrap(), "len {len}"); + let data = muxer.writer.into_inner(); + let (te_start, te_size) = first_track_entry(&data); + let children = master_children(&data, te_start, te_size); + let (_, off, size) = *children + .iter() + .find(|(id, _, _)| *id == ebml::CODEC_PRIVATE) + .expect("CodecPrivate present"); + assert_eq!(&data[off..off + size as usize], &cp[..], "len {len}"); + let end = children.iter().map(|&(_, o, s)| o + s as usize).max(); + assert_eq!( + end, + Some(te_start + te_size), + "len {len}: children fill the entry" + ); + assert!(children.iter().any(|(id, _, _)| *id == ebml::AUDIO)); + } + let mut aac = make_audio_track(); + aac.codec_id = ebml::CODEC_AAC; + let mut muxer = MkvMuxer::new(Cursor::new(Vec::new()), &[aac], None, 0.0, &[]).unwrap(); + assert!( + !muxer.set_codec_private(0, &[0u8; 14]).unwrap(), + "does not fit" + ); + } + #[test] fn video_colour_element_emitted_only_when_hdr_metadata_present() { // A video track with colour metadata (matrix/transfer) must emit the diff --git a/src/mux/mkvstream.rs b/src/mux/mkvstream.rs index 17014fb6..9497143a 100644 --- a/src/mux/mkvstream.rs +++ b/src/mux/mkvstream.rs @@ -240,6 +240,8 @@ const MAX_BLOCK_SIZE: u64 = 64 * 1024 * 1024; // largest accepted SIMPLE_BLOCK p /// Largest accepted CODEC_PRIVATE payload. hvcC/avcC/setup blobs are a /// few KB in practice; 16 MiB is far above any legitimate value. const MAX_CODEC_PRIVATE: u64 = 16 * 1024 * 1024; +/// Largest total of decoded CODEC_PRIVATE retained across all TrackEntries. +const MAX_CODEC_PRIVATE_TOTAL: usize = 2 * MAX_CODEC_PRIVATE as usize; /// Largest accepted string element (TITLE/CODEC_ID/LANGUAGE/TRACK_NAME). const MAX_STRING_LEN: u64 = 64 * 1024; /// EBML unsigned-int elements are at most 8 bytes wide. @@ -404,7 +406,7 @@ impl PendingMux { .map(|(f, _)| f.pts) .filter(|&pts| in_ns.is_none_or(|i| pts >= i)) .fold(idr_pts, i64::min); - self.origin_lead_ns = idr_pts - origin; + self.origin_lead_ns = idr_pts.saturating_sub(origin); let before = self.buffered.len(); self.buffered.retain(|(f, _)| f.pts >= origin); self.dropped_pre_origin = (before - self.buffered.len()) as u64; @@ -811,7 +813,7 @@ impl MkvStream { // Provenance must survive the replay: these pre-muxer frames still carry // the byte offset they were read from — dropping it here would fall back // to the timestamp heuristic this change set exists to stop relying on. - muxer.write_frame_at_with_padding( + let replayed = muxer.write_frame_at_with_padding( f.track, f.pts, f.keyframe, @@ -821,7 +823,19 @@ impl MkvStream { f.source.map(|s| s.byte), f.coding.as_ref().and_then(|c| c.field_order()), f.discard_padding_ns, - )?; + ); + // A track-range reject writes nothing and is not fatal (see + // `fail_on_write_error`): drop that buffered frame, keep the file. + match replayed { + Err(e) if crate::error::error_code(&e) == Some(crate::error::E_MUX_TRACK_RANGE) => { + tracing::warn!( + target: "mux", + track = f.track, + "buffered frame for a track outside the muxer dropped" + ); + } + other => other?, + } } self.mode = Mode::Write(WriteMode::Active(Box::new(muxer))); Ok(()) @@ -1587,6 +1601,7 @@ fn parse_mkv_header(r: &mut impl Read, want_chapters: bool) -> io::Result = Vec::new(); let mut codec_privates: Vec<(u16, Vec)> = Vec::new(); + let mut codec_private_bytes = 0usize; let mut tracks = TrackTable::default(); let (id, size, _) = ebml::read_element_header(r)?; @@ -1701,6 +1716,10 @@ fn parse_mkv_header(r: &mut impl Read, want_chapters: bool) -> io::Result MAX_CODEC_PRIVATE_TOTAL { + return Err(crate::error::Error::MkvSourceInvalid.into()); + } codec_privates.push((t.number, cp)); } } else { @@ -2244,6 +2263,18 @@ fn parse_content_encodings(r: &mut impl Read, size: u64) -> io::Result) -> bool { + match codec_private.and_then(|cp| cp.get(16..20)) { + Some(fcc) => !matches!( + fcc.to_ascii_uppercase().as_slice(), + b"WVC1" | b"WMVA" | b"WMV3" + ), + None => false, + } +} + // Undo `steps` on one payload; the result may not exceed `cap` bytes (zlib bomb guard). fn decode_content(steps: &[Decode], mut data: Vec, cap: usize) -> io::Result> { for step in steps { @@ -2298,7 +2329,8 @@ fn parse_track(r: &mut impl Read, size: u64) -> io::Result { let (mut codec_id, mut lang, mut name) = (String::new(), String::from("eng"), String::new()); let mut bcp47: Option = None; let mut video = VideoMeta::default(); - let (mut sr, mut ch, mut forced) = (0.0f64, 0u8, false); + // RFC 9559 5.1.4.1.28.3: Channels defaults to 1. + let (mut sr, mut ch, mut forced) = (0.0f64, 1u8, false); let mut bit_depth = 0u64; let mut codec_priv: Option> = None; let mut enc = Encodings::default(); @@ -2409,8 +2441,8 @@ fn parse_track(r: &mut impl Read, size: u64) -> io::Result { } None => None, }; - // &str consts can't be `match` patterns, so compare via guards — this keeps - // the single source of truth in `ebml::CODEC_*` shared with the muxer. + // The PCM arms bind layout state, so the ladder is chained `if`s; the CodecIDs + // stay the `ebml::CODEC_*` consts shared with the muxer. Keep each ID in one arm. let cid = codec_id.as_str(); let (mut pcm, mut infer) = (None, None); let is_pcm = cid == ebml::CODEC_PCM_BE || cid == ebml::CODEC_PCM_LE; @@ -2433,7 +2465,7 @@ fn parse_track(r: &mut impl Read, size: u64) -> io::Result { Codec::Hevc } else if cid == ebml::CODEC_H264 { Codec::H264 - } else if cid == ebml::CODEC_VC1 { + } else if cid == ebml::CODEC_VC1 && !vfw_fourcc_is_foreign(codec_priv.as_deref()) { Codec::Vc1 } else if cid == ebml::CODEC_MPEG2 { Codec::Mpeg2 @@ -2765,11 +2797,17 @@ fn parse_block_counted( let frames = parse_block_raw(block, cluster_ts_ticks, ts_scale_ns, tracks, duration_ns)?; let mut kept = Vec::with_capacity(frames.len()); let mut lost = Vec::new(); + // One decoded-size budget for the whole block: a laced block's frames share it, so + // 256 frames each inflating to the cap cannot hold 256x the cap at once. + let mut budget = MAX_BLOCK_SIZE as usize; for mut f in frames { if let Some(steps) = tracks.decode.get(f.track).filter(|s| !s.is_empty()) { let len = f.data.len() as u64; - match decode_content(steps, std::mem::take(&mut f.data), MAX_BLOCK_SIZE as usize) { - Ok(d) => f.data = d, + match decode_content(steps, std::mem::take(&mut f.data), budget) { + Ok(d) => { + budget -= d.len(); + f.data = d; + } Err(_) => { lost.push(len); continue; @@ -5449,6 +5487,97 @@ mod tests { ); } + // Matroska with one TrackEntry per `(number, type, CodecID, extra)`, `extra` being + // pre-encoded children appended after the number, type and CodecID. + fn mkv_with_tracks(entries: &[(u64, u64, &str, &[u8])]) -> Vec { + let mut tracks = Vec::new(); + for (tnum, ttype, codec_id, extra) in entries { + let mut entry = Vec::new(); + ebml::write_uint(&mut entry, ebml::TRACK_NUMBER, *tnum).unwrap(); + ebml::write_uint(&mut entry, ebml::TRACK_TYPE, *ttype).unwrap(); + ebml::write_string(&mut entry, ebml::CODEC_ID, codec_id).unwrap(); + entry.extend_from_slice(extra); + ebml::write_id(&mut tracks, ebml::TRACK_ENTRY).unwrap(); + ebml::write_size(&mut tracks, entry.len() as u64).unwrap(); + tracks.extend_from_slice(&entry); + } + let mut out = Vec::new(); + ebml::write_id(&mut out, ebml::EBML).unwrap(); + ebml::write_size(&mut out, 0).unwrap(); + ebml::write_id(&mut out, ebml::SEGMENT).unwrap(); + ebml::write_unknown_size(&mut out).unwrap(); + ebml::write_id(&mut out, ebml::INFO).unwrap(); + ebml::write_size(&mut out, 0).unwrap(); + ebml::write_id(&mut out, ebml::TRACKS).unwrap(); + ebml::write_size(&mut out, tracks.len() as u64).unwrap(); + out.extend_from_slice(&tracks); + out + } + + fn mkv_with_track_extra(ttype: u64, codec_id: &str, extra: &[u8]) -> Vec { + mkv_with_tracks(&[(1, ttype, codec_id, extra)]) + } + + // Retained CodecPrivate across all tracks is bounded, not just each blob. + #[test] + fn total_codec_private_across_tracks_is_bounded() { + let mut extra = Vec::new(); + ebml::write_binary( + &mut extra, + ebml::CODEC_PRIVATE, + &vec![0u8; MAX_CODEC_PRIVATE as usize], + ) + .unwrap(); + let ok = mkv_with_tracks(&[ + (1, 2, ebml::CODEC_AC3, &extra), + (2, 2, ebml::CODEC_AC3, &extra), + ]); + assert!(MkvStream::open(Cursor::new(ok)).is_ok()); + let many: Vec<_> = (1..=3) + .map(|n| (n, 2, ebml::CODEC_AC3, &extra[..])) + .collect(); + let err = MkvStream::open(Cursor::new(mkv_with_tracks(&many))).err(); + let code = format!("E{}", crate::error::E_MKV_SOURCE_INVALID); + assert!(err.is_some_and(|e| e.to_string().starts_with(&code))); + } + + // V_MS/VFW/FOURCC is any VFW codec: DivX / XviD must not be relabelled VC-1. + #[test] + fn vfw_track_is_vc1_only_when_the_fourcc_says_so() { + for (fourcc, want) in [ + (b"WVC1", Codec::Vc1), + (b"wvc1", Codec::Vc1), + (b"DIVX", Codec::Unknown(0)), + (b"XVID", Codec::Unknown(0)), + ] { + let mut bih = vec![0u8; 40]; + bih[16..20].copy_from_slice(fourcc); + let mut extra = Vec::new(); + ebml::write_binary(&mut extra, ebml::CODEC_PRIVATE, &bih).unwrap(); + let bytes = mkv_with_track_extra(1, ebml::CODEC_VC1, &extra); + let stream = MkvStream::open(Cursor::new(bytes)).unwrap(); + let got = stream.info().streams.first().map(stream_codec); + assert_eq!(got, Some(want), "fourcc {fourcc:?}"); + } + } + + // RFC 9559: an Audio master without Channels means one channel. + #[test] + fn audio_without_channels_element_reads_as_mono() { + let mut audio = Vec::new(); + ebml::write_float(&mut audio, ebml::SAMPLING_FREQUENCY, 48000.0).unwrap(); + let mut extra = Vec::new(); + ebml::write_id(&mut extra, ebml::AUDIO).unwrap(); + ebml::write_size(&mut extra, audio.len() as u64).unwrap(); + extra.extend_from_slice(&audio); + let bytes = mkv_with_track_extra(2, ebml::CODEC_AC3, &extra); + let stream = MkvStream::open(Cursor::new(bytes)).unwrap(); + let Some(crate::disc::Stream::Audio(a)) = stream.info().streams.first() else { + panic!("audio stream expected"); + }; + assert_eq!(a.channels, crate::disc::AudioChannels::Mono); + } + fn three_track_title() -> crate::disc::DiscTitle { use crate::disc::{ AudioStream, ColorSpace, DiscTitle, FrameRate, HdrFormat, Resolution, Stream, @@ -6779,6 +6908,36 @@ mod tests { assert!(s.finish().is_err(), "a truncated MKV must not finish Ok"); } + // A buffered frame for a track the muxer lacks is rejected on its own at activation; + // it must not fail the video frame that triggered the replay or truncate the file. + #[test] + fn an_out_of_range_buffered_frame_does_not_fail_activation() { + let out = SharedOut::new(); + let mut s = MkvStream::create(Box::new(out.clone()), &h264_title(), None).unwrap(); + s.write(&av_frame(5, 0, true, vec![9])).unwrap(); + s.write(&av_frame(0, 0, true, vec![1, 2, 3])).unwrap(); + s.write(&av_frame(0, 1_000_000, false, vec![4])).unwrap(); + s.finish().unwrap(); + } + + // A saturated IDR pts against a far-negative buffered pts must not overflow the lead. + #[test] + fn origin_lead_saturates_on_extreme_timestamps() { + let mut p = PendingMux { + writer: Box::new(Cursor::new(Vec::new())), + tracks: Vec::new(), + timings: Vec::new(), + video_track: None, + opening_capture_path: None, + buffered: vec![(av_frame(1, i64::MIN / 2, false, vec![1]), None)], + buffered_bytes: 1, + origin_lead_ns: 0, + dropped_pre_origin: 0, + }; + p.set_origin(i64::MAX, None); + assert_eq!(p.origin_lead_ns, i64::MAX); + } + #[test] fn set_track_timing_rejects_bad_tracks_and_late_calls_with_codes() { let out = SharedOut::new(); @@ -7439,6 +7598,105 @@ mod readback_tests { v } + // EditionFlagDefault is the two-byte ID 0x45DB mkvmerge writes; the default edition wins. + #[test] + fn the_default_edition_is_read_by_its_registered_id() { + let atom = |name: &str| { + let mut a = Vec::new(); + ebml::write_uint(&mut a, ebml::CHAPTER_TIME_START, 0).unwrap(); + let mut d = Vec::new(); + ebml::write_string(&mut d, ebml::CHAP_STRING, name).unwrap(); + a.extend(el(ebml::CHAPTER_DISPLAY, &d)); + el(ebml::CHAPTER_ATOM, &a) + }; + let first = el(ebml::EDITION_ENTRY, &atom("first")); + let mut second = vec![0x45, 0xDB, 0x81, 0x01]; + second.extend(atom("second")); + let mut body = first; + body.extend(el(ebml::EDITION_ENTRY, &second)); + let chapters = parse_chapters(&body).unwrap(); + assert_eq!(chapters.len(), 1); + assert_eq!(chapters[0].name, "second"); + } + + fn encoding(order: u64, algo: u64, settings: &[u8]) -> Vec { + let mut comp = Vec::new(); + ebml::write_uint(&mut comp, CONTENT_COMP_ALGO, algo).unwrap(); + if !settings.is_empty() { + ebml::write_binary(&mut comp, CONTENT_COMP_SETTINGS, settings).unwrap(); + } + let mut body = Vec::new(); + ebml::write_uint(&mut body, CONTENT_ENCODING_ORDER, order).unwrap(); + body.extend(el(CONTENT_COMPRESSION, &comp)); + el(CONTENT_ENCODING, &body) + } + + fn parse_encodings(list: &[Vec]) -> io::Result { + let body = list.concat(); + parse_content_encodings(&mut Cursor::new(&body), body.len() as u64) + } + + #[test] + fn content_encodings_decode_highest_order_first() { + let enc = parse_encodings(&[encoding(0, 0, &[]), encoding(1, 3, b"AB")]).unwrap(); + assert_eq!( + enc.frames, + vec![Decode::Prefix(b"AB".to_vec()), Decode::Zlib] + ); + } + + #[test] + fn content_comp_algos_other_than_zlib_and_header_strip_are_undecodable() { + for algo in [1, 2] { + let enc = parse_encodings(&[encoding(0, algo, &[])]).unwrap(); + assert!(enc.undecodable, "algo {algo}"); + assert!(enc.frames.is_empty()); + } + } + + #[test] + fn content_encodings_per_track_are_capped() { + let n = MAX_CONTENT_ENCODINGS as u64; + let list = |n| (0..n).map(|o| encoding(o, 0, &[])).collect::>(); + assert!(parse_encodings(&list(n)).is_ok()); + assert!(parse_encodings(&list(n + 1)).is_err()); + } + + #[test] + fn display_unit_unknown_ignores_the_declared_display_size() { + let meta = |unit| VideoMeta { + pixel_width: 1920, + pixel_height: 1080, + display_width: Some(4), + display_height: Some(3), + display_unit: unit, + ..VideoMeta::default() + }; + let res = Resolution::R1080p; + assert_eq!(meta(0).display_aspect(res), Some((4, 3))); + assert_eq!(meta(DISPLAY_UNIT_UNKNOWN).display_aspect(res), None); + } + + #[test] + fn primaries_map_to_their_color_spaces() { + assert_eq!(color_space_from_primaries(5), ColorSpace::Bt470bg); + assert_eq!(color_space_from_primaries(6), ColorSpace::Smpte170m); + } + + #[test] + fn default_duration_matches_only_within_a_hundredth_of_a_percent() { + for (ns, want) in [ + (41_708_333, FrameRate::F23_976), + (41_666_667, FrameRate::F24), + (33_366_667, FrameRate::F29_97), + (33_333_333, FrameRate::F30), + (16_683_350, FrameRate::F59_94), + (16_666_667, FrameRate::F60), + ] { + assert_eq!(frame_rate_from_ns(ns), want, "{ns} ns"); + } + } + fn uint(id: u32, val: u64) -> Vec { let mut v = Vec::new(); ebml::write_uint(&mut v, id, val).unwrap(); @@ -7891,6 +8149,26 @@ mod readback_tests { assert_eq!(s.lost_bytes(), 5); } + // The decode cap is one budget per block: two laced frames that each fit the cap + // alone cannot both inflate (a 256-frame block would otherwise hold 256x the cap). + #[test] + fn laced_zlib_frames_share_one_block_budget() { + let frame = zlib(&vec![0u8; MAX_BLOCK_SIZE as usize * 5 / 8]); + // Fixed-size lacing (flags 0x04): count-1, then equal-size frames. + let mut block = vec![0x81, 0x00, 0x00, 0x84, 0x01]; + block.extend_from_slice(&frame); + block.extend_from_slice(&frame); + let enc = encodings(&compression(None, &[])); + let bytes = mkv( + &[entry(1, 17, ebml::CODEC_VOBSUB, &enc)], + &cluster_of(&[&block]), + ); + let mut s = open(bytes); + let frames = drain_all(&mut s); + assert_eq!(frames.len(), 1, "the second frame is past the block budget"); + assert_eq!(s.errors(), 1); + } + #[test] fn a_corrupt_compressed_codec_private_drops_the_track_not_the_file() { let mut extra = diff --git a/src/mux/mp4/audio.rs b/src/mux/mp4/audio.rs index 9cd6c6c3..1aeb6e7e 100644 --- a/src/mux/mp4/audio.rs +++ b/src/mux/mp4/audio.rs @@ -222,7 +222,7 @@ pub(super) fn dec3_box(c: &DolbyConfig) -> Vec { bx(b"dec3", &[b[3], b[4], b[5], b[6], b[7]]) } -/// Build an audio sample entry (`ac-3` / `ec-3` / `dtsc` / `dtsh`) with the +/// Build an audio sample entry (`ac-3` / `ec-3` / `dtsc` / `dtsh` / `mp4a`) with the /// given config box. `AudioSampleEntry` per ISO/IEC 14496-12 §12.2.3. pub(super) fn audio_sample_entry( fourcc: &[u8; 4], @@ -246,6 +246,12 @@ pub(super) fn audio_sample_entry( bx(fourcc, &e) } +/// The integer sample rate `audio_sample_entry` wrote into `entry`, if it is long enough. +pub(super) fn entry_sample_rate(entry: &[u8]) -> Option { + let rate = entry.get(32..36)?; + Some(u32::from_be_bytes([rate[0], rate[1], rate[2], rate[3]]) >> 16) +} + // ── DTS (dtsc/dtsh + ddts) ─────────────────────────────────────────────────── /// DTS core `SFREQ` (4-bit) → sample rate (Hz); 0 marks an invalid code @@ -485,7 +491,8 @@ pub(super) fn dolby_sample_entry( } /// (channels, rate) of an AudioSpecificConfig the `esds` can carry: a table rate and -/// channel configuration, no escaped object type, short enough for one-byte sizes. +/// channel configuration, no escaped object type, short enough for one-byte sizes +/// (the 100-byte cap keeps every enclosing descriptor body under 128). pub(super) fn aac_config(asc: &[u8]) -> Option<(u16, u32)> { const RATES: [u32; 12] = [ 96000, 88200, 64000, 48000, 44100, 32000, 24000, 22050, 16000, 12000, 11025, 8000, @@ -585,6 +592,40 @@ mod tests { f } + #[test] + fn aac_config_maps_channel_configurations() { + // AAC-LC, 44.1 kHz; the channel configuration sits in bits 6..3 of byte 1. + let cfg = |c: u8| aac_config(&[0x12, c << 3]); + assert_eq!(cfg(1), Some((1, 44_100))); + assert_eq!(cfg(6), Some((6, 44_100))); + assert_eq!(cfg(7), Some((8, 44_100)), "config 7 is 7.1"); + assert_eq!(cfg(0), None, "PCE layout is not describable"); + assert_eq!(cfg(8), None, "reserved"); + } + + #[test] + fn aac_config_refuses_escaped_object_types_and_oversized_configs() { + assert_eq!(aac_config(&[0xF8, 0x08]), None); + let mut asc = vec![0x12, 0x10]; + asc.resize(100, 0); + assert!(aac_config(&asc).is_some()); + asc.push(0); + assert_eq!(aac_config(&asc), None); + } + + #[test] + fn mpeg_audio_rate_follows_the_version_bits() { + let rate = |h1: u8| { + mpeg_sample_entry(Codec::Mp3, &[0xFF, h1, 0x00, 0xC0], &[]) + .and_then(|e| entry_sample_rate(&e)) + }; + assert_eq!(rate(0xFB), Some(44_100), "MPEG-1"); + assert_eq!(rate(0xF3), Some(22_050), "MPEG-2"); + assert_eq!(rate(0xE3), Some(11_025), "MPEG-2.5"); + assert_eq!(rate(0xF1), None, "reserved layer"); + assert_eq!(rate(0xEB), None, "reserved version"); + } + #[test] fn ac3_bsi_and_dac3() { let c = parse_dolby(&ac3_frame_5_1()).expect("parsed"); diff --git a/src/mux/mp4/mod.rs b/src/mux/mp4/mod.rs index 47485280..2a376b06 100644 --- a/src/mux/mp4/mod.rs +++ b/src/mux/mp4/mod.rs @@ -272,6 +272,13 @@ pub struct Mp4Sink { dropped: Vec<(usize, Mp4SkipReason)>, /// MPEG-2 multichannel extension tracks (no MP4 mapping), reported once packets arrive. excluded: super::ps::UnstoredExtensions, + /// Clip-join PTS correction, shared with the MKV muxer and the demux sink: a + /// multi-clip playlist's source PTS resets or jumps at each join. + timeline: crate::mux::timeline::TimelineContinuity, + /// The video stream that drives the timeline's epochs. + ref_video: Option, + /// Frames the timeline placed (the denominator for its drop count). + frames_mapped: u64, } impl Mp4Sink { @@ -371,6 +378,10 @@ impl Mp4Sink { writer.write_all(b"mdat")?; writer.write_all(&0u64.to_be_bytes())?; + let ref_video = tracks + .iter() + .find(|t| t.media == Media::Video) + .map(|t| t.stream_idx); Ok(Self { writer, title: title.clone(), @@ -383,6 +394,12 @@ impl Mp4Sink { finished: false, plan: report, excluded: super::ps::UnstoredExtensions::new(title, "MP4"), + timeline: crate::mux::timeline::TimelineContinuity::with_clips( + &title.clips, + title.content_format, + ), + ref_video, + frames_mapped: 0, dropped: Vec::new(), }) } @@ -467,11 +484,22 @@ impl PesSink for Mp4Sink { { // A rate the title could not name (e.g. 22.05 kHz MP3) comes from the entry. if self.tracks[slot].audio_timescale == 0 { - self.tracks[slot].audio_timescale = - u32::from_be_bytes([entry[32], entry[33], entry[34], entry[35]]) >> 16; + self.tracks[slot].audio_timescale = audio::entry_sample_rate(&entry).unwrap_or(0); } self.tracks[slot].audio_entry = Some(entry); } + // Onto the continuous timeline first; `None` is material outside the clip marks. + let is_video = self.tracks[slot].media == Media::Video; + let Some(pts_ns) = self.timeline.map( + frame.pts, + Some(frame.track) == self.ref_video, + frame.track, + is_video, + frame.source.map(|s| s.byte), + ) else { + return Ok(()); + }; + self.frames_mapped += 1; // Nothing decodes before the first video keyframe: it is not stored. if self.tracks[slot].media == Media::Video && !frame.keyframe @@ -479,7 +507,6 @@ impl PesSink for Mp4Sink { { return Ok(()); } - let pts_ns = frame.pts; let offset = self.mdat_start + 16 + self.mdat_payload; self.writer.write_all(&frame.data)?; self.mdat_payload += frame.data.len() as u64; @@ -497,6 +524,26 @@ impl PesSink for Mp4Sink { return Ok(()); } self.finished = true; + // As the MKV muxer and demux sink: a seam plan that dropped everything, or most + // of the title, is a failed mux, not a short file at exit 0. + let seam_dropped = self.timeline.dropped_total(); + if self.frames_mapped == 0 && seam_dropped > 0 { + return Err(crate::error::Error::SinkWroteNothing.into()); + } + if seam_dropped > self.frames_mapped { + return Err(crate::error::Error::SeamPlanDroppedMost { + dropped: seam_dropped, + written: self.frames_mapped, + } + .into()); + } + if seam_dropped > 0 { + tracing::info!( + target: "mux", + dropped = seam_dropped, + "frames outside the playlist's clip marks were dropped at clip joins" + ); + } // Every drop below is recorded in `self.dropped` so `final_report()` cannot // keep claiming a track the file doesn't have; a `tracing::warn` alone left // the crate's own public report lying about the output. @@ -580,7 +627,6 @@ impl PesSink for Mp4Sink { // ── per-track box assembly ─────────────────────────────────────────────────── -/// Build a track's `trak` box and return `(bytes, duration_seconds)`. /// Per-track start delay: how far each track's first sample lies after the earliest /// sample of all tracks (the shared t=0). fn start_delays_ns(tracks: &[Track]) -> Vec { @@ -588,7 +634,7 @@ fn start_delays_ns(tracks: &[Track]) -> Vec { let origin = tracks.iter().filter_map(first).min().unwrap_or(0); tracks .iter() - .map(|t| first(t).map_or(0, |f| f - origin)) + .map(|t| first(t).map_or(0, |f| f.saturating_sub(origin))) .collect() } @@ -612,6 +658,7 @@ fn build_edts(delay_ns: i64, media_secs: f64) -> Vec { bx(b"edts", &fullbox(b"elst", 1, 0, &body)) } +/// Build a track's `trak` box and return `(bytes, duration_seconds)`. fn build_trak_at(t: &Track, delay_ns: i64) -> (Vec, f64) { match t.media { Media::Video => build_video_trak_full(t, delay_ns), @@ -622,7 +669,8 @@ fn build_trak_at(t: &Track, delay_ns: i64) -> (Vec, f64) { fn build_video_trak_full(t: &Track, delay_ns: i64) -> (Vec, f64) { let timing = VideoTiming::derive(&t.samples); let media_dur = timing.total_duration(); - let secs = media_dur as f64 / timing.timescale as f64; + // Presentation, not decode, extent: lost frames leave CTS beyond `media_dur`. + let secs = timing.presentation_end() as f64 / timing.timescale as f64; let stsd = build_visual_stsd(t.codec, &t.codec_private, t.width, t.height, t.colr); let stbl = build_video_stbl(stsd, &t.samples, &timing); @@ -702,6 +750,12 @@ impl VideoTiming { fn total_duration(&self) -> u64 { self.cts.len() as u64 * self.sample_dur as u64 } + // Where the last frame to present ends; past `total_duration` when frames were lost. + fn presentation_end(&self) -> u64 { + let last = self.cts.iter().copied().max().unwrap_or(0).max(0) as u64; + self.total_duration() + .max(last.saturating_add(self.sample_dur as u64)) + } fn ctts(&self) -> Vec { self.cts .iter() @@ -1363,6 +1417,28 @@ mod tests { ); } + // A multi-clip playlist's source PTS restarts at a join: the sink must place the next + // clip after the first (the timeline corrector the MKV muxer uses), never on top of it. + #[test] + fn a_clip_join_pts_reset_continues_the_timeline() { + let t = title(vec![hevc_video()], vec![Some(vec![1, 2, 3, 4])]); + let mut s = Mp4Sink::create(std::io::Cursor::new(Vec::new()), &t).unwrap(); + for _clip in 0..2 { + for i in 0..150i64 { + let key = i % 25 == 0; + s.write(&frame(0, i * 40_000_000, key, vec![0xAB; 64])) + .unwrap(); + } + } + let pts: Vec = s.tracks[0].samples.iter().map(|x| x.pts_ns).collect(); + assert_eq!(pts.len(), 300); + assert!( + pts.windows(2).all(|w| w[1] > w[0]), + "clip 2 must follow clip 1: {:?}", + &pts[148..152] + ); + } + // An audio track whose frames never yield a parseable sample entry must be dropped from // moov, not written as an stsd around an empty entry, while the frames still reach mdat. #[test] @@ -2059,6 +2135,93 @@ mod tests { ); } + // Frames lost to a damaged span leave the last CTS past n*d; the edit list must still + // cover the whole presentation or players cut the tail. + #[test] + fn edit_list_covers_the_last_frame_when_frames_were_lost() { + let mut samples: Vec = (0..4) + .map(|i| Sample { + offset: 0, + size: 1, + pts_ns: i * 40_000_000, + keyframe: i == 0, + }) + .collect(); + samples.push(Sample { + offset: 0, + size: 1, + pts_ns: 10_000_000_000, + keyframe: false, + }); + let t = Track { + media: Media::Video, + track_id: 1, + stream_idx: 0, + codec: Codec::Hevc, + codec_private: vec![1], + width: 16, + height: 16, + colr: None, + language: [0x55, 0xC4], + audio_entry: None, + audio_timescale: 0, + samples, + }; + let (trak, secs) = build_trak_at(&t, 40_000_000); + let edts = find_child(&trak[8..], b"edts").unwrap(); + let e = find_child(edts, b"elst").unwrap(); + let seg = u64::from_be_bytes(e[28..36].try_into().unwrap()); + assert!(seg >= 10 * MOVIE_TIMESCALE as u64, "segment {seg}"); + assert!(secs >= 10.04, "{secs}"); + } + + // AAC / MP3 entries come from the MPEG branch; a rate the title could not name is + // taken from the entry for the mdhd timescale. + #[test] + fn mp3_track_gets_an_entry_and_its_rate_from_the_first_frame() { + let mut mp3 = audio(Codec::Mp3, "eng"); + if let DiscStream::Audio(a) = &mut mp3 { + a.sample_rate = SampleRate::Unknown; + } + let t = title(vec![hevc_video(), mp3], vec![Some(vec![1, 2, 3, 4]), None]); + let mut s = Mp4Sink::create(std::io::Cursor::new(Vec::new()), &t).unwrap(); + s.write(&frame(0, 0, true, vec![0xAB; 800])).unwrap(); + // MPEG-2 Layer III header, 22.05 kHz, mono. + let mp3_frame = vec![0xFF, 0xF3, 0x90, 0xC4, 0, 0, 0, 0]; + s.write(&frame(1, 0, true, mp3_frame.clone())).unwrap(); + s.write(&frame(1, 26_122_449, true, mp3_frame)).unwrap(); + s.finish().unwrap(); + assert!(s.undelivered_streams().is_empty()); + assert_eq!(s.tracks[1].audio_timescale, 22_050); + assert!(s.tracks[1].audio_entry.is_some()); + } + + // Extreme first timestamps (a clamped reader) must not overflow the start delay. + #[test] + fn start_delays_saturate_on_extreme_timestamps() { + let mk = |id, pts_ns| Track { + media: Media::Audio, + track_id: id, + stream_idx: 0, + codec: Codec::Ac3, + codec_private: Vec::new(), + width: 0, + height: 0, + colr: None, + language: [0x55, 0xC4], + audio_entry: None, + audio_timescale: 48_000, + samples: vec![Sample { + offset: 0, + size: 1, + pts_ns, + keyframe: true, + }], + }; + let delays = start_delays_ns(&[mk(1, i64::MAX), mk(2, i64::MIN)]); + assert_eq!(delays, vec![i64::MAX, 0]); + } + #[test] fn video_before_its_first_keyframe_is_not_stored() { let t = title(vec![hevc_video()], vec![Some(vec![1, 2, 3, 4])]); diff --git a/src/mux/mp4/read.rs b/src/mux/mp4/read.rs index 45d89087..0b2af6eb 100644 --- a/src/mux/mp4/read.rs +++ b/src/mux/mp4/read.rs @@ -179,6 +179,7 @@ impl Mp4Reader { // this track's sample count (indices past it are never read). let sizes = find_box(stbl, b"stsz") .map(|b| parse_stsz(b, sample_budget)) + .or_else(|| find_box(stbl, b"stz2").map(|b| parse_stz2(b, sample_budget))) .unwrap_or_default(); let n = sizes.len(); if n == 0 { @@ -820,7 +821,7 @@ fn audio_rate(entry: u32, timescale: u32, dts_hd: Option) -> u32 { !known(entry) && timescale > 0xFFFF && known(timescale) - && timescale.is_multiple_of(entry) + && (timescale.is_multiple_of(entry) || entry == timescale & 0xFFFF) } }; if wider { timescale } else { entry } @@ -1039,6 +1040,35 @@ fn parse_stsz(b: &[u8], max: usize) -> Vec { out } +/// stz2 (compact sample sizes, field size 4 / 8 / 16 bits) → per-sample sizes. +fn parse_stz2(b: &[u8], max: usize) -> Vec { + if b.len() < 12 { + return Vec::new(); + } + let field_size = b[7] as usize; + if !matches!(field_size, 4 | 8 | 16) { + return Vec::new(); + } + let count = (be32(b, 8) as usize).min(max); + let body = &b[12..]; + // `count` entries of `field_size` bits can't exceed the box body. + let mut out = Vec::with_capacity(count.min(body.len() * 8 / field_size)); + for i in 0..count { + let size = match field_size { + 4 => body + .get(i / 2) + .map(|&v| u32::from(if i % 2 == 0 { v >> 4 } else { v & 0xF })), + 8 => body.get(i).map(|&v| u32::from(v)), + _ => body + .get(i * 2..i * 2 + 2) + .map(|v| u32::from(u16::from_be_bytes([v[0], v[1]]))), + }; + let Some(size) = size else { break }; + out.push(size); + } + out +} + /// stco (32-bit) / co64 (64-bit) → chunk offsets. fn parse_stco(b: &[u8], is64: bool) -> Vec { // `< 8` vs `<= 8` is equivalent (same shape as `parse_stsd`, and stsc/stts/ctts/stss @@ -3039,6 +3069,14 @@ mod tests { p.extend_from_slice(&3u32.to_be_bytes()); // count = 3 (samples exist) mp4_box(b"stsz", &p) }; + // Compact form of the same three 10-byte samples (omit == "stz2" swaps it in). + let stz2 = { + let mut p = vec![0u8; 4]; // version+flags + p.extend_from_slice(&[0, 0, 0, 8]); // reserved, field_size = 8 + p.extend_from_slice(&3u32.to_be_bytes()); // count + p.extend_from_slice(&[10, 10, 10]); + mp4_box(b"stz2", &p) + }; let stco = { let mut p = Vec::new(); p.extend_from_slice(&[0, 0, 0, 0]); @@ -3068,9 +3106,12 @@ mod tests { }; let mut stbl = Vec::new(); stbl.extend_from_slice(&stsd); - if omit != b"stsz" { + if omit != b"stsz" && omit != b"stz2" { stbl.extend_from_slice(&stsz); } + if omit == b"stz2" { + stbl.extend_from_slice(&stz2); + } if omit != b"stco" { stbl.extend_from_slice(&stco); } @@ -3088,6 +3129,62 @@ mod tests { mp4_box(b"trak", &mp4_box(b"mdia", &mdia)) } + // Samples come out in global decode order, interleaved across tracks. + #[test] + fn samples_from_two_tracks_are_interleaved_by_decode_time() { + use std::io::Cursor; + let mut body = audio_trak_missing(b"____"); + body.extend(audio_trak_missing(b"____")); + let rd = + Mp4Reader::from_reader(Cursor::new(mp4_box(b"moov", &body)), "two".into()).unwrap(); + let order: Vec = rd.samples.iter().map(|s| s.track).collect(); + assert_eq!(order, vec![0, 1, 0, 1, 0, 1]); + } + + // The title duration is the longest track, whatever order the tracks come in. + #[test] + fn title_duration_is_the_longest_track() { + use std::io::Cursor; + let mut body = audio_trak(1_000); // 1 s + body.extend(audio_trak(48_000)); // ~21 ms + let rd = + Mp4Reader::from_reader(Cursor::new(mp4_box(b"moov", &body)), "dur".into()).unwrap(); + assert!((rd.info().duration_secs - 1.0).abs() < 1e-9); + } + + // A 16.16 samplerate whose integer part was truncated to u16 resolves to the mdhd rate. + #[test] + fn audio_rate_recovers_a_u16_truncated_entry_rate() { + assert_eq!(audio_rate(96_000 & 0xFFFF, 96_000, None), 96_000); + assert_eq!(audio_rate(48_000, 48_000, None), 48_000); + assert_eq!(audio_rate(44_100, 96_000, None), 44_100); + } + + // A compact `stz2` size table carries the track's samples like `stsz` does. + #[test] + fn stz2_sizes_populate_the_track_samples() { + use std::io::Cursor; + let moov = mp4_box(b"moov", &audio_trak_missing(b"stz2")); + let rd = Mp4Reader::from_reader(Cursor::new(moov), "stz2".into()).unwrap(); + assert_eq!(rd.samples.len(), 3); + assert_eq!(rd.samples[0].size, 10); + } + + #[test] + fn parse_stz2_decodes_each_field_size() { + let hdr = |fs: u8, n: u32| { + let mut b = vec![0, 0, 0, 0, 0, 0, 0, fs]; + b.extend_from_slice(&n.to_be_bytes()); + b + }; + let with = |fs, n, body: &[u8]| [hdr(fs, n), body.to_vec()].concat(); + assert_eq!(parse_stz2(&with(4, 3, &[0x12, 0x30]), 99), vec![1, 2, 3]); + assert_eq!(parse_stz2(&with(8, 2, &[7, 9]), 99), vec![7, 9]); + assert_eq!(parse_stz2(&with(16, 1, &[1, 2]), 99), vec![0x0102]); + assert!(parse_stz2(&with(12, 1, &[1, 2]), 99).is_empty()); + assert_eq!(parse_stz2(&with(8, 5, &[7, 9]), 99), vec![7, 9]); + } + // A track with samples (`stsz`) but no chunk-offset table (`stco`/`co64`) must be DROPPED, // not indexed with offsets near file byte 0. #[test] diff --git a/src/mux/mpg/mod.rs b/src/mux/mpg/mod.rs index 78bad33a..2e233a22 100644 --- a/src/mux/mpg/mod.rs +++ b/src/mux/mpg/mod.rs @@ -111,6 +111,9 @@ pub struct MpgSink { writer: Option, mux: Option>, excluded: super::ps::UnstoredExtensions, + /// Clip-join PTS correction shared with the MKV muxer and the other sinks: a + /// multi-clip playlist's source PTS resets or jumps at each join. + timeline: crate::mux::timeline::TimelineContinuity, origin_saturated: u64, frames: u64, finished: bool, @@ -292,6 +295,7 @@ impl MpgSink { .codec_privates .get(video_track) .and_then(|c| c.as_deref()); + let video_out = route[video_track].ok_or(crate::error::Error::MpgNoVideoTrack)?; let mut excluded = super::ps::UnstoredExtensions::new(title, "MPG"); excluded.retain(|i| route[i].is_none()); Ok(Self { @@ -305,7 +309,7 @@ impl MpgSink { route, outs, buffers, - video_out: 0, + video_out, deriver: DtsDeriver::for_codec(codec, cp), anchor: None, armed: false, @@ -318,6 +322,10 @@ impl MpgSink { writer: Some(writer), mux: None, excluded, + timeline: crate::mux::timeline::TimelineContinuity::with_clips( + &title.clips, + title.content_format, + ), origin_saturated: 0, frames: 0, finished: false, @@ -504,7 +512,9 @@ impl MpgSink { Ok(()) } - // The system header and program stream map of the first pack (MS-21 §2.7.8). + // The system header and program stream map of the first pack (MS-21 §2.7.8). Side + // effects: unseen extensions are unrouted and the video buffer is sized, so it runs once, + // before `Mux::new` clones `self.buffers`. fn first_pack_prefix(&mut self) -> Vec { // J23: an extension with no packets in the origin window is not described; its // packets, should they come later, are left out and reported like an orphan's. @@ -667,11 +677,21 @@ impl PesSink for MpgSink { return Ok(()); } let is_video = out == self.video_out; + // Onto the continuous timeline first; `None` is material outside the clip marks. + let Some(pts_ns) = self.timeline.map( + frame.pts, + is_video, + frame.track, + is_video, + frame.source.map(|s| s.byte), + ) else { + return Ok(()); + }; // Drop video before the first keyframe: nothing decodes without it (tsmux's guard). if is_video && !frame.keyframe && !self.armed { return Ok(()); } - let rel = self.rel(frame.pts); + let rel = self.rel(pts_ns); self.span = Some( self.span .map_or((rel, rel), |(lo, hi)| (lo.min(rel), hi.max(rel))), @@ -759,10 +779,33 @@ impl MpgSink { } } // Design §2.3 EOF: a short input is written, not failed; zero frames is MuxEmpty. + let seam_dropped = self.timeline.dropped_total(); + if self.frames == 0 && seam_dropped > 0 { + return Err(crate::error::Error::SinkWroteNothing.into()); + } + if seam_dropped > self.frames { + return Err(crate::error::Error::SeamPlanDroppedMost { + dropped: seam_dropped, + written: self.frames, + } + .into()); + } + if seam_dropped > 0 { + tracing::info!( + target: "mux", + dropped = seam_dropped, + "frames outside the playlist's clip marks were dropped at clip joins" + ); + } if self.frames == 0 { return Err(crate::error::Error::MuxEmpty.into()); } self.maybe_set_origin(true)?; + let Some(m) = self.mux.as_mut() else { + return Err(crate::error::Error::MuxEmpty.into()); + }; + let finished = m.finish(); + // After the final drain, so EOF-time padding and forced passes are counted. let c = self.counters(); if c != MpgCounters::default() { tracing::warn!( @@ -772,14 +815,13 @@ impl MpgSink { pstd_late_aus = c.pstd.late_aus, pts_gap_over_0_7s = c.pstd.pts_gaps, interleave_cap = c.pstd.interleave_cap, + padding_packs = c.pstd.padding_packs, + forced_eof = c.pstd.forced_eof, rebased_gaps = c.pstd.rebased_gaps, origin_saturated = c.origin_saturated, "mpg: program stream needed corrections (counted, not refused)" ); } - match self.mux.as_mut() { - Some(m) => m.finish(), - None => Err(crate::error::Error::MuxEmpty.into()), - } + finished } } diff --git a/src/mux/mpg/pstd.rs b/src/mux/mpg/pstd.rs index 68622a8f..48714ef8 100644 --- a/src/mux/mpg/pstd.rs +++ b/src/mux/mpg/pstd.rs @@ -324,17 +324,7 @@ impl Mux { fn commencement(s: &Stream, k: usize) -> Option { let head = s.queue.front()?; let q = s.queue.get(k)?; - let before: usize = if k == 0 { - 0 - } else { - head.au.data.len() - head.sent - + s.queue - .iter() - .skip(1) - .take(k - 1) - .map(|q| q.au.data.len()) - .sum::() - }; + let before = if k == 0 { 0 } else { Self::first_byte(s, k)? }; let own = if k == 0 { q.au.mark.checked_sub(head.sent)? } else { @@ -1029,6 +1019,86 @@ mod tests { assert_eq!(clean.counters().forced_eof, 0); } + // What no pack can ever take, even with the waits lifted, is an error at EOF, never a + // silent Ok with the AUs dropped: an LPCM AU shorter than one packing unit. + #[test] + fn eof_with_untakeable_aus_is_an_error() { + let spec = StreamSpec { + stream_id: pack::PRIVATE_STREAM_1, + payload: Payload::Lpcm { + sub_id: 0xA0, + channels: 2, + rate: 48_000, + }, + buffer: 0, + sparse: false, + av: true, + }; + let buf = BufferSpec { + stream_id: pack::PRIVATE_STREAM_1, + scale_1024: true, + size: 232, + }; + let mut m = Mux::new(Vec::new(), vec![spec], vec![buf], Vec::new(), 25_200); + m.push( + 0, + Au { + lpcm_bits: 16, + ..au(9_000, 3, 0) + }, + ); + let e = m.finish().expect_err("an AU that cannot be packetized"); + assert!( + e.to_string() + .contains(&crate::error::Error::MpgUnpacketized.to_string()), + "{e}" + ); + } + + // A declared audio track that never delivers is passed once the others lead it by the + // interleave cap, counted; before the cap the clock waits for it. + #[test] + fn a_silent_audio_track_is_passed_at_the_interleave_cap() { + let video = StreamSpec { + stream_id: 0xE0, + payload: Payload::Plain, + buffer: 0, + sparse: false, + av: true, + }; + let audio = StreamSpec { + stream_id: 0xC0, + buffer: 1, + ..video.clone() + }; + let bufs = vec![ + BufferSpec { + stream_id: 0xE0, + scale_1024: true, + size: 232, + }, + BufferSpec { + stream_id: 0xC0, + scale_1024: false, + size: 128, + }, + ]; + let mut m = Mux::new(Vec::new(), vec![video, audio], bufs, Vec::new(), 25_200); + m.push(0, au(90_000, 100, 0)); + for k in 1..=7u64 { + m.push(0, au(90_000 + k * 90_000, 100, 0)); + m.pump(false).unwrap(); + } + assert_eq!(m.counters().interleave_cap, 0, "7 s: still waiting"); + assert!(m.writer.is_empty(), "nothing is written while audio lags"); + for k in 8..=12u64 { + m.push(0, au(90_000 + k * 90_000, 100, 0)); + m.pump(false).unwrap(); + } + assert_eq!(m.counters().interleave_cap, 1); + assert!(!m.writer.is_empty()); + } + // B1: at EOF nothing is left behind. A zero-length AU used to wedge its stream and // every AU after it; `finish` returned Ok with them queued. #[test] diff --git a/src/mux/mpg/scan.rs b/src/mux/mpg/scan.rs index 6778b8eb..b1c9f829 100644 --- a/src/mux/mpg/scan.rs +++ b/src/mux/mpg/scan.rs @@ -557,3 +557,123 @@ mod dar_tests { assert_eq!(dar, None); } } + +#[cfg(test)] +mod probe_tests { + use super::*; + + // A sequence header for `w`x`h` with aspect code 1 and frame-rate code `rate`, then a + // sequence extension whose progressive_sequence bit is `progressive`. + fn video(w: u32, h: u32, rate: u8, progressive: bool) -> Vec { + vec![ + 0, + 0, + 1, + 0xB3, + (w >> 4) as u8, + ((w & 0xF) << 4) as u8 | (h >> 8) as u8, + h as u8, + 0x10 | rate, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 0xB5, + 0x14, + if progressive { 0x8A } else { 0x82 }, + ] + } + + #[test] + fn frame_rate_codes_map_in_order() { + let want = [ + FrameRate::F23_976, + FrameRate::F24, + FrameRate::F25, + FrameRate::F29_97, + FrameRate::F30, + FrameRate::F50, + FrameRate::F59_94, + FrameRate::F60, + ]; + for (i, rate) in want.into_iter().enumerate() { + let es = video(720, 576, i as u8 + 1, false); + assert_eq!( + probe_video(&es, Some(0x02)).unwrap().2, + rate, + "code {}", + i + 1 + ); + } + let es = video(720, 576, 0, false); + assert_eq!(probe_video(&es, Some(0x02)).unwrap().2, FrameRate::Unknown); + } + + #[test] + fn scan_type_follows_the_progressive_bit() { + let res = |h, p| probe_video(&video(720, h, 3, p), Some(0x02)).unwrap().1; + assert_eq!(res(480, false), Resolution::R480i); + assert_eq!(res(576, false), Resolution::R576i); + assert_eq!(res(576, true), Resolution::R576p); + assert_eq!(res(1080, false), Resolution::R1080i); + assert_eq!(res(1080, true), Resolution::R1080p); + } + + #[test] + fn mpeg_audio_rate_and_layer_come_from_the_header() { + let probe = |b1: u8, b2: u8| { + let (codec, _, rate) = probe_mpeg_audio(&[0xFF, b1, b2, 0xC4, 0, 0]); + (codec, rate) + }; + // MPEG-1 (ID bit set): 44.1, 48, 32 kHz; Layer II is 0xFD, Layer III 0xFB. + assert_eq!(probe(0xFD, 0x00), (Codec::Mp2, SampleRate::S44_1)); + assert_eq!(probe(0xFD, 0x04), (Codec::Mp2, SampleRate::S48)); + assert_eq!(probe(0xFD, 0x08), (Codec::Mp2, SampleRate::Unknown)); + assert_eq!(probe(0xFB, 0x00), (Codec::Mp3, SampleRate::S44_1)); + assert_eq!(probe(0xFB, 0x04), (Codec::Mp3, SampleRate::S48)); + // LSF (ID bit clear) never reads the MPEG-1 table. + assert_eq!(probe(0xF5, 0x00), (Codec::Mp2, SampleRate::Unknown)); + assert_eq!(probe(0xF5, 0x04), (Codec::Mp2, SampleRate::Unknown)); + } + + #[test] + fn lpcm_rate_and_channels_come_from_the_header() { + let probe = |b: u8| probe_lpcm(&[0, b, 0]); + assert_eq!(probe(0x01), (AudioChannels::Stereo, SampleRate::S48)); + assert_eq!(probe(0x11), (AudioChannels::Stereo, SampleRate::S96)); + assert_eq!(probe(0x21), (AudioChannels::Stereo, SampleRate::S44_1)); + assert_eq!(probe(0x31), (AudioChannels::Stereo, SampleRate::Unknown)); + } + + // A crafted map whose declared lengths overrun the packet is refused, never indexed. + #[test] + fn a_map_with_overrunning_lengths_is_refused() { + let ok = pack::psm( + &[], + &[pack::PsmEntry { + stream_type: 0x02, + stream_id: 0xE0, + descriptors: vec![], + }], + ) + .unwrap(); + assert!(parse_map(&ok).is_some()); + let refuse = |at: usize, v: u8| { + let mut m = ok.clone(); + m[at] = v; + let n = m.len() - 4; + let crc = pack::crc32(&m[..n]); + m[n..].copy_from_slice(&crc.to_be_bytes()); + assert!(parse_map(&m).is_none(), "byte {at} = {v}"); + }; + // program_stream_info_length, elementary_stream_map_length, then an entry's + // info length: each past the end of the packet. + refuse(9, 0xF0); + refuse(11, 0xF0); + let info = usize::from(u16::from_be_bytes([ok[8], ok[9]])); + refuse(12 + info + 3, 0xF0); + } +} diff --git a/src/mux/mpg/tests.rs b/src/mux/mpg/tests.rs index 92ec8c19..6987e927 100644 --- a/src/mux/mpg/tests.rs +++ b/src/mux/mpg/tests.rs @@ -2269,3 +2269,268 @@ fn native_16_bit_lpcm_frames_pack_as_16_bit() { assert_eq!(aus.len(), 1); assert_eq!((aus[0].lpcm_bits, aus[0].data.len()), (16, 16)); } + +// A multi-clip playlist's source PTS restarts at a join: the sink must place the next clip +// after the first (the timeline corrector the MKV muxer uses), never on top of it. +#[test] +fn a_clip_join_pts_reset_continues_the_timeline() { + let fx = fixture(&Opts { + lpcm: false, + ..Opts::default() + }); + let mut sink = MpgSink::create(Vec::new(), &fx.title).expect("create"); + for _clip in 0..2 { + for f in &fx.frames { + sink.write(f).expect("write"); + } + } + sink.finish().expect("finish"); + let out = sink.mux.take().expect("mux").into_writer(); + let parsed = replay::parse(&out).expect("parses"); + let pts: Vec = parsed + .pes + .iter() + .filter(|x| x.key == (0xE0, None)) + .filter_map(|x| x.pts) + .collect(); + let span = pts.iter().max().unwrap() - pts.iter().min().unwrap(); + // Two 6 s clips: the output spans ~12 s, not one clip's 6 s. + assert!(span >= 11 * 90_000, "span {span} ticks"); +} + +// A track first seen after the origin window with a timestamp below the origin is clamped +// to 0 and counted, never wrapped through a negative-to-unsigned cast. +#[test] +fn a_timestamp_below_the_origin_is_clamped_and_counted() { + let fx = fixture(&Opts { + lpcm: false, + spu_tracks: 0, + ..Opts::default() + }); + let mut sink = MpgSink::create(Vec::new(), &fx.title).unwrap(); + let mut f = PesFrame { + track: 1, + pts: VIDEO_START_NS, + keyframe: true, + data: vec![0x55; 1 << 20], + duration_ns: None, + discard_padding_ns: 0, + source: None, + coding: None, + }; + for _ in 0..(HOLD_CAP_BYTES >> 20) + 2 { + sink.write(&f).unwrap(); + } + assert_eq!(sink.counters().origin_saturated, 0); + f.pts = VIDEO_START_NS - 3_000 * MS; + f.data = vec![0x55; 64]; + sink.write(&f).unwrap(); + assert_eq!(sink.counters().origin_saturated, 1); + sink.finish().unwrap(); + let out = sink.mux.take().unwrap().into_writer(); + let parsed = replay::parse(&out).unwrap(); + assert!( + parsed + .pes + .iter() + .any(|x| x.key.0 == 0xC0 && x.pts == Some(0)), + "the clamped AU is stamped 0" + ); +} + +// MS-5/MS-6: the system header's audio and video bounds count the carried streams, and the +// video buffer bound is vbv_buffer_size + 8 KiB in 1024-byte units. +#[test] +fn the_system_header_states_the_bounds() { + let r = run(&fixture(&Opts::default())); + let sh = &r.parsed.system_headers[0].1; + // Six audio streams (MP2 base and extension, AC-3, DTS, two LPCM); subpictures are not audio. + assert_eq!(sh[9] >> 2, 6, "audio_bound"); + assert_eq!(sh[10] & 0x1F, 1, "video_bound"); + let video = sh[12..].chunks(3).find(|e| e[0] == 0xE0).unwrap(); + let size = u16::from(video[1] & 0x1F) << 8 | u16::from(video[2]); + assert_eq!(video[1] & 0x20, 0x20, "scale 1024"); + assert_eq!(u64::from(size), (112 * 2048 + 8192) / 1024); +} + +// The sequence_extension's vbv_buffer_size_extension carries the high bits of the buffer size. +#[test] +fn vbv_extension_bits_extend_the_size() { + let mut h = seq_header(720, 576, 3, 112, false); + let e = h.windows(4).position(|w| w == [0, 0, 1, 0xB5]).unwrap() + 4; + h[e + 4] = 2; + assert_eq!(vbv_bytes(&h), Some(((2u64 << 10) | 112) * 2048)); +} + +// MPG's first pack rate is R0: SD pictures at the SD base rate, HD at the HD one. +#[test] +fn the_base_rate_follows_the_picture_height() { + let first_rate = |res| { + let mut fx = fixture(&Opts { + secs: 2, + ..Opts::default() + }); + if let DiscStream::Video(v) = &mut fx.title.streams[0] { + v.resolution = res; + } + run(&fx).parsed.packs[0].rate + }; + assert_eq!(first_rate(Resolution::R576i), R0_SD); + assert_eq!(first_rate(Resolution::R1080i), R0_HD); +} + +// MS-22: a base layer is 11172 audio (0x03) when its header says MPEG-1, 13818-3 (0x04) when +// the ID bit marks a low-sampling-frequency stream. +#[test] +fn the_mp2_stream_type_follows_the_header_id_bit() { + let stream_type = |id_byte: u8| { + let mut fx = fixture(&Opts { + lpcm: false, + spu_tracks: 0, + secs: 2, + ..Opts::default() + }); + fx.title.streams.truncate(2); + for f in fx.frames.iter_mut().filter(|f| f.track == 1) { + f.data[1] = id_byte; + } + fx.frames.retain(|f| f.track <= 1); + let r = run(&fx); + let m = &r.parsed.psms[0].1; + let info = usize::from(u16::from_be_bytes([m[8], m[9]])); + let es = &m[12 + info..m.len() - 4]; + let n = usize::from(u16::from_be_bytes([es[2], es[3]])); + let audio = &es[4 + n..]; + assert_eq!(audio[1], 0xC0); + audio[0] + }; + assert_eq!(stream_type(0xFD), 0x03); + assert_eq!(stream_type(0xF5), 0x04); +} + +// EOF pads the last partial LPCM packing unit with silence rather than dropping its samples. +#[test] +fn a_partial_lpcm_unit_at_eof_is_padded_not_lost() { + let mut fx = fixture(&Opts { + secs: 2, + ..Opts::default() + }); + let last = fx.frames.iter().rfind(|f| f.track == 6).unwrap(); + // Three stereo 24-bit sample frames: under any packing unit. + let tail = PesFrame { + pts: last.pts + 10 * MS, + data: vec![0x21; 3 * 6], + ..last.clone() + }; + fx.frames.push(tail); + let r = run(&fx); + let want: usize = fx + .frames + .iter() + .filter(|f| f.track == 6) + .map(|f| f.data.len()) + .sum(); + let mut parser = crate::mux::codec::lpcm::LpcmParser::new_dvd(); + let mut got = 0; + for x in r.parsed.pes.iter().filter(|x| x.key == (0xBD, Some(0xA1))) { + let mut data = x.sub_hdr[4..7].to_vec(); + data.extend_from_slice(&x.es); + let pes = crate::mux::ts::PesPacket { + source: None, + pid: 0, + pts: x.pts.map(|p| p as i64), + dts: None, + data, + discontinuity: false, + }; + got += parser + .parse(&pes) + .iter() + .map(|f| f.data.len()) + .sum::(); + } + assert!(got >= want, "{got} bytes out of {want} in"); +} + +// Without a map the tracks sort video, MPEG audio, private audio by sub-stream, then +// subpictures, whatever the order the packets arrive in; 0x80-0x87 is AC-3, 0x88-0x8F DTS. +#[test] +fn an_unmapped_scan_orders_tracks_and_labels_private_audio() { + let private = |sub: u8| { + let mut d = vec![sub, 1, 0, 1]; + d.resize(16, 0); + (0xBD, d) + }; + let ps = scan_pack( + &[ + private(0x20), + private(0x88), + private(0x80), + (0xC0, vec![0xFF, 0xFD, 0xC4, 0x00, 0, 0]), + (0xE0, scan_video()), + ], + None, + ); + let s = scan::scan_streams(&ps).unwrap(); + let got: Vec = s + .iter() + .map(|x| match x { + DiscStream::Video(_) => "video".into(), + DiscStream::Audio(a) => format!("{:?}:{:#x}", a.codec, a.pid), + DiscStream::Subtitle(t) => format!("{:?}", t.codec), + }) + .collect(); + assert_eq!( + got, + ["video", "Mp2:0xc0", "Ac3:0xbd80", "Dts:0xbd88", "DvdSub"] + ); +} + +// An extension whose map pairs it with a base other than 0xC0|n is left out, and so is one +// whose base is itself an extension. +#[test] +fn a_map_pairing_an_extension_with_the_wrong_base_drops_it() { + let entry = |stream_type, stream_id, descriptors| pack::PsmEntry { + stream_type, + stream_id, + descriptors, + }; + let e = [ + entry(0x02, 0xE0, vec![]), + entry( + 0x04, + 0xC0, + pack::hierarchy_descriptor(pack::HIERARCHY_BASE, 0, 0).to_vec(), + ), + entry( + 0x04, + 0xC1, + pack::hierarchy_descriptor(pack::HIERARCHY_BASE, 2, 0).to_vec(), + ), + // 0xD1 embeds layer 0 (base 0xC0), but the IR pairs 0xD1 only with 0xC1. + entry( + 0x04, + 0xD1, + pack::hierarchy_descriptor(pack::HIERARCHY_EXTENSION, 3, 0).to_vec(), + ), + ]; + let m = pack::psm(&[], &e).unwrap(); + let ps = scan_pack( + &[ + (0xE0, scan_video()), + (0xC0, vec![0xFF, 0xFD, 0xC4, 0x00, 0, 0]), + (0xC1, vec![0xFF, 0xFD, 0xC4, 0x00, 0, 0]), + (0xD1, vec![0x7F, 0xF1, 0x23, 0x45]), + ], + Some(&m), + ); + let s = scan::scan_streams(&ps).unwrap(); + let pids: Vec = s + .iter() + .filter_map(|x| match x { + DiscStream::Audio(a) => Some(a.pid), + _ => None, + }) + .collect(); + assert_eq!(pids, vec![0xC0, 0xC1]); +} diff --git a/src/mux/network.rs b/src/mux/network.rs index 5341a233..67f1efcc 100644 --- a/src/mux/network.rs +++ b/src/mux/network.rs @@ -750,6 +750,59 @@ mod tests { assert_eq!(n, 1); } + // `finish_incomplete` arms the reset itself; a sender that never reaches its drop (or a + // receiver waiting on the close) still sees the abort mark. + #[test] + fn finish_incomplete_resets_the_connection_at_once() { + use crate::pes::PesSink as _; + let (addr, handle) = spawn_ending_reader(); + let mut writer = NetworkStream::connect_vetted(&addr.to_string(), false) + .unwrap() + .meta(&sample_title()); + writer.write(&one_frame()).unwrap(); + let linger = |w: &NetworkStream| match &w.mode { + Mode::Write { writer, .. } => { + socket2::SockRef::from(writer.get_ref()).linger().unwrap() + } + Mode::Read { .. } => unreachable!(), + }; + assert_eq!(linger(&writer), None); + writer.finish_incomplete().unwrap(); + assert_eq!(linger(&writer), Some(std::time::Duration::ZERO)); + drop(writer); + let _ = handle.join().unwrap(); + } + + // A timing set for a stream the title does not have is an error, not an out-of-range + // index; one set after the header is on the wire is refused, not silently dropped. + #[test] + fn track_timing_is_range_and_header_checked() { + use crate::pes::PesSink as _; + let timing = crate::pes::TrackTiming { + codec_delay_ns: 1, + seek_preroll_ns: 2, + }; + let mut timings = Vec::new(); + assert!(set_timing(&mut timings, 1, timing, 2).is_ok()); + assert_eq!(timings, vec![Default::default(), timing]); + assert!(set_timing(&mut timings, 2, timing, 2).is_err()); + + let (addr, handle) = spawn_ending_reader(); + let mut writer = NetworkStream::connect_vetted(&addr.to_string(), false) + .unwrap() + .meta(&sample_title()); + writer.write(&one_frame()).unwrap(); + let e = writer.set_track_timing(0, timing).unwrap_err(); + assert!( + e.to_string() + .contains(&crate::error::Error::StreamHeaderWritten.to_string()), + "{e}" + ); + writer.finish().unwrap(); + drop(writer); + let _ = handle.join().unwrap(); + } + // A connection cut inside a frame (even with a clean FIN) is an error. #[test] fn a_connection_cut_mid_frame_is_an_error_at_the_receiver() { diff --git a/src/mux/pipelined_stream.rs b/src/mux/pipelined_stream.rs index e51294fc..39d3beeb 100644 --- a/src/mux/pipelined_stream.rs +++ b/src/mux/pipelined_stream.rs @@ -872,6 +872,46 @@ mod tests { ); } + /// A parser that holds one inter frame, flagged discontinuous, until flush. + struct HeldDanglingParser; + impl CodecParser for HeldDanglingParser { + fn parse(&mut self, _: &PesPacket) -> Vec { + vec![] + } + fn flush(&mut self) -> Vec { + vec![super::super::codec::Frame { + coding: None, + source: None, + pts_ns: 0, + keyframe: false, + discontinuity: true, + data: b"P".to_vec(), + duration_ns: None, + }] + } + fn codec_private(&self) -> Option> { + None + } + } + + // B1 at EOF: a final inter frame held by the parser and flagged discontinuous goes + // through the same gate: it is dropped and counted, not written with dangling references. + #[test] + fn b1_gate_applies_to_the_frame_flushed_at_eof() { + let title = video_title(false); + let parsers: Vec<(u16, Box)> = + vec![(0x1011, Box::new(HeldDanglingParser))]; + let (mut stream, tx) = make_stream(title, parsers, vec![(0x1011u16, 0usize)]); + tx.send(DemuxBatch::Ts(vec![ts_pes(0x1011, vec![1])])) + .unwrap(); + tx.send(DemuxBatch::Eof).unwrap(); + assert!( + stream.read().unwrap().is_none(), + "the dangling frame is dropped" + ); + assert_eq!(stream.errors(), 1, "and counted"); + } + /// Counterpart to B1: a discontinuity on a NON-video track must NOT drop /// frames — audio/subtitle access units are independent, so the gate admits /// every frame the parser still produces. diff --git a/src/mux/ps.rs b/src/mux/ps.rs index 2a8063dc..a30ba1e2 100644 --- a/src/mux/ps.rs +++ b/src/mux/ps.rs @@ -41,7 +41,8 @@ pub struct PsPacket { /// PES stream ID (0xE0 for video, 0xC0 for audio, 0xBD for private, etc.). pub stream_id: u8, /// Sub-stream ID for private stream 1 (AC3: 0x80-0x87, DTS: 0x88-0x8F, - /// LPCM: 0xA0-0xA7, subtitles: 0x20-0x3F). + /// LPCM: 0xA0-0xA7, E-AC-3: 0xC0-0xCF, subtitles: 0x20-0x3F); for extended stream + /// id 0xFD, the PES `stream_id_extension`. pub sub_stream_id: Option, /// Presentation timestamp in 90kHz ticks. pub pts: Option, diff --git a/src/mux/resolve.rs b/src/mux/resolve.rs index 6d37a3a4..3d68a8de 100644 --- a/src/mux/resolve.rs +++ b/src/mux/resolve.rs @@ -455,7 +455,14 @@ fn open_container( opts.raw, )?)) } - StreamUrl::Stdio => Ok(Box::new(StdioStream::input_staged(opts.raw))), + StreamUrl::Stdio => { + let mut stdio = StdioStream::input_staged(opts.raw); + // A selection is resolved against the title at open: read the header first. + if !opts.selection.is_all() { + stdio.prime()?; + } + Ok(Box::new(stdio)) + } StreamUrl::Null => Err(crate::error::Error::StreamWriteOnly.into()), // `mp4://` as a source: demux a progressive MP4 back into PES frames, so // `mp4://` flows to every sink (mkv://, audio://, json://, …). @@ -536,18 +543,12 @@ where }; let title = disc.titles[idx].clone(); let format = disc.content_format; - // Pass `DecryptKeys::None` to the decrypt decorator when --raw is set — - // the read stack still flows through the same producer+demux+parse - // pipeline, just without the AACS/CSS step. One highway for ISO reads. - let effective_keys = if opts.raw { - crate::decrypt::DecryptKeys::None - } else { - keys - }; + // `--raw` keys are already `None`: the read stack still flows through the same + // producer+demux+parse pipeline, just without the AACS/CSS step. let stream = build_iso_pipeline( reader, title, - effective_keys, + keys, ISO_MUX_BATCH_SECTORS, format, opts.raw, @@ -556,7 +557,7 @@ where Ok(stream) } -// `image_input` over the rip's key set (KU §3.1, §3.5): the gate is `check_decryptable`, +// `image_input_scanned` over the rip's key set (KU §3.1, §3.5): the gate is `check_decryptable`, // and the mux reads through the set's reader. No lookup, no banking. fn keyed_image_input( reader: S, @@ -918,7 +919,7 @@ pub(crate) fn build_sector_pipeline( // on CSS/unencrypted (per-2048-byte-sector) would reject any extent // whose sector count isn't a multiple of 3. let unit_align: u16 = match &keys { - crate::decrypt::DecryptKeys::Aacs { .. } => 3, + crate::decrypt::DecryptKeys::Aacs { .. } => AACS_UNIT_SECTORS, _ => 1, }; let full_extents = extents.clone(); @@ -991,10 +992,12 @@ pub(crate) fn build_keyed_pipeline( ) .map_err(io::Error::from)?; let plan = IsoPlan { - extents: set.key_map().read_plan(&full_extents, 3), + extents: set + .key_map() + .read_plan(&full_extents, u32::from(AACS_UNIT_SECTORS)), full_extents, policy, - unit_align: 3, + unit_align: AACS_UNIT_SECTORS, format: set.content_format(), }; iso_pipeline_tail(decrypting, plan, title, ctx) @@ -1160,6 +1163,9 @@ fn build_ps_pipeline( iso_pipeline_tail(stage, plan, title, ctx).map(|p| p.with_video_stream_id(scan.video_id)) } +// Sectors in one AACS aligned unit: the `unit_align` of every keyed read plan. +const AACS_UNIT_SECTORS: u16 = crate::aacs::content::ALIGNED_UNIT_SECTORS as u16; + // Sectors per m2ts read: whole AACS units, ~1 MiB (the stage's byte-view refill). const M2TS_READ_SECTORS: u16 = 510; diff --git a/src/mux/select.rs b/src/mux/select.rs index 90e4cd51..0a5288cb 100644 --- a/src/mux/select.rs +++ b/src/mux/select.rs @@ -475,4 +475,40 @@ mod tests { "the added base is logged" ); } + + // The pairing is per index: extension 0xD1 belongs to base 0xC1, never to 0xC0. + #[test] + fn an_mp2_extension_pairs_with_the_base_of_its_own_index() { + let mp2 = |pid: u16, label: &str| { + Stream::Audio(AudioStream { + pid, + codec: Codec::Mp2, + channels: AudioChannels::Stereo, + language: "eng".into(), + sample_rate: SampleRate::S48, + secondary: false, + purpose: LabelPurpose::Normal, + label: label.into(), + }) + }; + let keep = |audio: Vec| { + let mut t = DiscTitle::empty(); + t.streams = vec![ + video(0x00E0), + mp2(0x00C0, ""), + mp2(0x00C1, ""), + mp2(0x00D1, crate::disc::MP2_EXTENSION_LABEL), + ]; + StreamSelection { + audio: PidFilter::Only(audio), + subtitle: PidFilter::All, + } + .apply(&mut t) + .unwrap(); + pids(&t) + }; + assert_eq!(keep(vec![0x00C1]), vec![0x00E0, 0x00C1, 0x00D1]); + assert_eq!(keep(vec![0x00C0]), vec![0x00E0, 0x00C0]); + assert_eq!(keep(vec![0x00D1]), vec![0x00E0, 0x00C1, 0x00D1]); + } } diff --git a/src/mux/selected.rs b/src/mux/selected.rs index 5917cf58..59ef8a73 100644 --- a/src/mux/selected.rs +++ b/src/mux/selected.rs @@ -156,6 +156,62 @@ mod tests { assert!(s.read().unwrap().is_none()); } + struct Rich(Frames); + + impl PesSource for Rich { + fn read(&mut self) -> std::io::Result> { + self.0.read() + } + fn info(&self) -> &DiscTitle { + self.0.info() + } + fn track_timing(&self, track: usize) -> TrackTiming { + TrackTiming { + codec_delay_ns: track as u64 * 10, + seek_preroll_ns: 0, + } + } + fn headers_ready(&self) -> bool { + false + } + fn config_changes(&self) -> Vec<(usize, u64)> { + vec![(1, 4), (2, 7)] + } + fn errors(&self) -> u64 { + 5 + } + fn lost_bytes(&self) -> u64 { + 9 + } + } + + // Per-track answers are asked of the inner source by its own index and come back under the + // kept numbering; the source-wide counters and readiness pass straight through. + #[test] + fn inner_answers_are_translated_and_counters_pass_through() { + let mut title = DiscTitle::empty(); + title.streams = vec![audio(1), audio(2), audio(3)]; + let src = Rich(Frames(title, Default::default())); + let sel = StreamSelection { + audio: PidFilter::Only(vec![3]), + subtitle: PidFilter::All, + }; + let s = SelectedSource::wrap(Box::new(src), &sel).unwrap(); + assert_eq!( + s.track_timing(0).codec_delay_ns, + 20, + "inner track 2's timing" + ); + assert_eq!(s.track_timing(1), TrackTiming::default()); + assert_eq!( + s.config_changes(), + vec![(0, 7)], + "renumbered, unkept dropped" + ); + assert_eq!((s.errors(), s.lost_bytes()), (5, 9)); + assert!(!s.headers_ready()); + } + // An unknown PID refuses at open, as on a disc title. #[test] fn an_unknown_pid_refuses() { diff --git a/src/mux/source.rs b/src/mux/source.rs index 66b6b216..22b61944 100644 --- a/src/mux/source.rs +++ b/src/mux/source.rs @@ -30,12 +30,13 @@ pub struct ScannedTitle { } impl ScannedTitle { - /// Title `idx` of `disc`, named by the disc. `None` when out of range. + /// Title `idx` of an image's `disc`, named as [`Source`] names an image's title (its + /// meta title, else the title's own playlist name). `None` when out of range. pub fn of(disc: &Disc, idx: usize) -> Option { Some(ScannedTitle { title: disc.titles.get(idx)?.clone(), format: disc.content_format, - disc_name: Some(disc_name(disc)), + disc_name: disc.meta_title.clone(), volume_id: disc.volume_id.clone(), }) } @@ -294,14 +295,74 @@ impl<'a> Source<'a> { } } - // The output title's name (SO14): every disc layout names it by the disc (its meta - // title, else its volume id); a container keeps its own. + // The output title's name (SO14): a drive's is the disc's meta title, else its volume id; + // an image's or folder's the meta title, else the playlist (a folder's volume id is its + // directory name, and an image and its folder must name a title alike); a container's own. pub(crate) fn title_name(&self) -> Option { match &self.origin { Origin::Prescanned { title, .. } | Origin::Live { title, .. } => { title.disc_name.clone() } + Origin::Image { disc, .. } => disc.meta_title.clone(), _ => self.layout().map(disc_name), } } } + +#[cfg(test)] +mod tests { + use super::*; + + fn disc(volume_id: &str, meta_title: Option<&str>) -> Disc { + let mut title = crate::disc::DiscTitle::empty(); + title.playlist = "VTS_11_26.VOB".into(); + Disc { + volume_id: volume_id.into(), + meta_title: meta_title.map(str::to_string), + format: crate::disc::DiscFormat::Dvd, + capacity_sectors: 1, + capacity_bytes: 2048, + layers: 1, + titles: vec![title], + region: crate::disc::DiscRegion::Free, + aacs: None, + css: None, + encrypted: false, + aacs_error: None, + css_error: None, + content_format: crate::disc::ContentFormat::MpegPs, + } + } + + fn image(d: Disc, folder: bool) -> Source<'static> { + Source { + origin: Origin::Image { + path: PathBuf::from("x"), + folder, + reader: Box::new(crate::test_util::MemSource::new(vec![0u8; 2048])), + disc: d, + }, + } + } + + // An image and its extracted folder name a title alike: the folder's volume id is its + // directory name, so with no meta title both keep the title's playlist name, and the + // engine's pre-scanned image title agrees. + #[test] + fn an_image_and_its_folder_name_a_title_alike() { + let iso = image(disc("GREENLAND", None), false); + let dir = image(disc("fmt_dir", None), true); + assert_eq!(iso.title_name(), None); + assert_eq!(dir.title_name(), None); + let scanned = ScannedTitle::of(&disc("GREENLAND", None), 0).unwrap(); + assert_eq!(scanned.disc_name, None); + assert_eq!(scanned.title.playlist, "VTS_11_26.VOB"); + + let iso = image(disc("GREENLAND", Some("Greenland")), false); + let dir = image(disc("fmt_dir", Some("Greenland")), true); + assert_eq!(iso.title_name().as_deref(), Some("Greenland")); + assert_eq!(dir.title_name().as_deref(), Some("Greenland")); + let scanned = ScannedTitle::of(&disc("GREENLAND", Some("Greenland")), 0).unwrap(); + assert_eq!(scanned.disc_name.as_deref(), Some("Greenland")); + } +} diff --git a/src/mux/stdio.rs b/src/mux/stdio.rs index bb14cb62..a2a2defc 100644 --- a/src/mux/stdio.rs +++ b/src/mux/stdio.rs @@ -136,6 +136,12 @@ impl StdioStream { pub fn info(&self) -> &crate::disc::DiscTitle { &self.disc_title } + + // Read the FMKV header now, so `info()` is the stream's title before the first frame + // (a stream selection is resolved against it at open). + pub(crate) fn prime(&mut self) -> io::Result<()> { + self.ensure_header_read() + } } impl crate::pes::PesSource for StdioStream { @@ -386,6 +392,72 @@ mod tests { assert!(r.read().unwrap().is_none()); } + // A zero-frame title still puts the header on the wire: the reader sees the title. + #[test] + fn finish_without_frames_still_writes_the_header() { + let out = Shared::default(); + let mut w = StdioStream::from_writer(&title_with_codec_privates(), Box::new(out.clone())); + w.finish().unwrap(); + let mut r = reader(out.0.lock().unwrap().clone()); + r.prime().unwrap(); + assert_eq!(r.info().playlist, "StdioTitle"); + assert!(r.read().unwrap().is_none()); + } + + // Frame framing follows the header: with no timing set (v1) frames carry no padding field + // and read back whole; with a timing set (v2) the DiscardPadding survives. + #[test] + fn frame_padding_follows_the_header_version() { + let round_trip = |timed: bool, padding: i64| { + let out = Shared::default(); + let mut w = + StdioStream::from_writer(&title_with_codec_privates(), Box::new(out.clone())); + if timed { + let timing = crate::pes::TrackTiming { + codec_delay_ns: 5, + seek_preroll_ns: 9, + }; + w.set_track_timing(0, timing).unwrap(); + } + let mut f = frame(&[1, 2, 3]); + f.discard_padding_ns = padding; + w.write(&f).unwrap(); + w.write(&frame(&[4, 5])).unwrap(); + w.finish().unwrap(); + let mut r = reader(out.0.lock().unwrap().clone()); + let a = r.read().unwrap().unwrap(); + let b = r.read().unwrap().unwrap(); + assert!(r.read().unwrap().is_none()); + (a.discard_padding_ns, a.data, b.data) + }; + assert_eq!(round_trip(false, 0), (0, vec![1, 2, 3], vec![4, 5])); + assert_eq!( + round_trip(true, -2_500_000), + (-2_500_000, vec![1, 2, 3], vec![4, 5]) + ); + } + + // prime() reads the header up front: info() is the stream's title before any frame, + // and the first read still returns the first frame. + #[test] + fn prime_reads_the_header_before_the_first_frame() { + let out = Shared::default(); + let mut w = StdioStream::from_writer(&title_with_codec_privates(), Box::new(out.clone())); + w.write(&frame(&[1, 2, 3])).unwrap(); + w.finish().unwrap(); + + let mut r = reader(out.0.lock().unwrap().clone()); + assert_eq!(r.info().streams.len(), 0); + r.prime().unwrap(); + assert_eq!(r.info().playlist, "StdioTitle"); + assert_eq!( + r.info().streams.len(), + title_with_codec_privates().streams.len() + ); + let f = r.read().unwrap().unwrap(); + assert_eq!(f.data, vec![1, 2, 3]); + } + // A zero-byte stdin is a clean headerless end, and never header-ready. #[test] fn empty_input_is_clean_eof_not_ready() { diff --git a/src/mux/ts.rs b/src/mux/ts.rs index 3444c678..d9cd8a5d 100644 --- a/src/mux/ts.rs +++ b/src/mux/ts.rs @@ -169,13 +169,7 @@ impl PesAssembler { bytes = self.buffer.len(), "PES buffer cap exceeded; dropping partial PES and resyncing on next PUSI", ); - self.buffer.clear(); - self.active = false; - self.header_remaining = 0; - // A dropped partial PES is a gap in the elementary stream — flag - // it so the NEXT completed PES carries a discontinuity, matching - // every other partial-drop path in this file (`drop_partial`). - self.pending_discontinuity = true; + self.drop_partial(); return; } // Grow by doubling but never past `cap`, so the allocation honours the share too. @@ -1255,6 +1249,66 @@ mod tests { assert!(out[1].data.starts_with(b"BBBB") && out[1].data.ends_with(&[b'b'; 184])); } + // A packet repeating the previous CC with other data means a whole CC cycle (16 + // packets) was lost: a continuation drops the open PES, a PUSI flags the new one. + #[test] + fn same_cc_with_other_data_is_a_gap() { + let pid = 0x1011; + let mut demux = TsDemuxer::new(&[pid]); + let mut out = demux.feed(&ts_payload_packet(pid, true, 0, &pes_start(b"AAAA"))); + out.extend(demux.feed(&ts_payload_packet(pid, false, 0, b"XXXX"))); + out.extend(demux.feed(&ts_payload_packet(pid, true, 1, &pes_start(b"CCCC")))); + out.extend(demux.flush()); + assert_eq!(out.len(), 1, "A is dropped, not spliced with X: {out:?}"); + assert!(out[0].data.starts_with(b"CCCC") && out[0].discontinuity); + + let mut demux = TsDemuxer::new(&[pid]); + let mut out = demux.feed(&ts_payload_packet(pid, true, 3, &pes_start(b"AAAA"))); + out.extend(demux.feed(&ts_payload_packet(pid, true, 3, &pes_start(b"BBBB")))); + out.extend(demux.flush()); + assert_eq!(out.len(), 2, "{out:?}"); + assert!(!out[0].discontinuity); + assert!(out[1].discontinuity, "the lost cycle flags the new PES"); + } + + // A header-only PES holds no ES bytes: no empty packet reaches the codec parsers. + #[test] + fn a_header_only_pes_is_not_emitted() { + let pid = 0x1011; + let exact = |pusi: bool, cc: u8, es: &[u8]| { + let mut p = es_packet_exact(pid, pusi, &pes_start(es)); + p[7] |= cc; + p + }; + let mut demux = TsDemuxer::new(&[pid]); + let mut out = demux.feed(&exact(true, 0, &[])); + out.extend(demux.feed(&exact(true, 1, b"BBBB"))); + out.extend(demux.flush()); + assert_eq!(out.len(), 1, "{out:?}"); + assert_eq!(out[0].data, b"BBBB"); + let mut demux = TsDemuxer::new(&[pid]); + let mut out = demux.feed(&exact(true, 0, &[])); + out.extend(demux.flush()); + assert!(out.is_empty(), "{out:?}"); + } + + // A zero-filled gap that ends mid-packet leaves its partial tail in the remainder, so the + // next feed completes it and the packet after the gap stays on the grid. + #[test] + fn a_gap_ending_mid_packet_keeps_the_next_feed_aligned() { + let pid = 0x1011; + let mut demux = TsDemuxer::new(&[pid]); + let mut first = ts_payload_packet(pid, true, 0, &pes_start(b"AAAA")); + first.extend(vec![0u8; 3 * BD_SOURCE_PACKET_BYTES + 50]); + let mut second = vec![0u8; BD_SOURCE_PACKET_BYTES - 50]; + second.extend(ts_payload_packet(pid, true, 1, &pes_start(b"BBBB"))); + let mut out = demux.feed(&first); + out.extend(demux.feed(&second)); + out.extend(demux.flush()); + assert_eq!(out.len(), 2, "{out:?}"); + assert!(out[1].data.starts_with(b"BBBB")); + } + // A PUSI whose payload is not a PES start leaves no PES open: the continuations after it // are not emitted as a headless frame, and the next PES is flagged. #[test] @@ -2672,6 +2726,34 @@ mod tests { assert_eq!(v.resolution, Resolution::R1080i, "MPEG-2 defaults to 1080i"); } + // Program-level descriptors (a non-zero program_info_length) are skipped, not read as ES + // entries. + #[test] + fn scan_streams_skips_program_descriptors() { + let pmt_pid = 0x0100u16; + let mut data = pat_packet(pmt_pid); + let mut body = [0xFFu8; 184]; + body[0] = 0x00; + let s = 1; + body[s] = 0x02; + // 9 fixed + 6 descriptor + 5 ES entry + 4 CRC. + let section_length: usize = 9 + 6 + 5 + 4; + body[s + 1] = 0xB0; + body[s + 2] = section_length as u8; + body[s + 3..s + 12].copy_from_slice(&[0, 1, 0xC1, 0, 0, 0xE0, 0, 0xF0, 6]); + // HDMV registration descriptor. + body[s + 12..s + 18].copy_from_slice(&[0x05, 0x04, b'H', b'D', b'M', b'V']); + let p = s + 18; + body[p..p + 5].copy_from_slice(&[0x1B, 0xE0 | 0x10, 0x11, 0xF0, 0x00]); + data.extend(bdts_packet(body, pmt_pid, true)); + data.extend(pat_packet(pmt_pid)); + let streams = scan_streams(&data).expect("the PMT parses"); + assert!( + matches!(&streams[..], [crate::disc::Stream::Video(v)] if v.pid == 0x1011 && v.codec == crate::disc::Codec::H264), + "{streams:?}" + ); + } + #[test] fn scan_streams_oversized_prog_info_len_does_not_panic() { // Regression: a PMT with prog_info_len larger than the section body must diff --git a/src/mux/tsmux.rs b/src/mux/tsmux.rs index f2cd7ef6..5c206d57 100644 --- a/src/mux/tsmux.rs +++ b/src/mux/tsmux.rs @@ -107,8 +107,8 @@ pub struct TsMuxer { /// the borrow of the converted bytes does not conflict with the `&mut self` /// the writer needs. annex_b: Vec, - /// Per-track arrival-time copy of `params_written` for the pre-keyframe drop guard, - /// which runs before the start-up hold (design §2.3 tick-domain rule 1). + /// Per-track: a video frame has arrived past the pre-keyframe drop guard, which runs + /// before the start-up hold (design §2.3 tick-domain rule 1). arrival_armed: Vec, /// Per video track: the online DTS deriver, built at its first frame. dts: Vec>, @@ -506,8 +506,8 @@ impl TsMuxer { self.write_psi_if_due(pts_90k)?; // NAL video (HEVC/H.264): convert length-prefixed NALUs to Annex B and prepend - // codec_private params on the first keyframe only (arm `params_written` even if - // data is empty, else later non-key frames fail the drop guard). Others pass through. + // codec_private params on the first keyframe only (`params_written` latches even if + // data is empty, so no later keyframe re-prepends them). Others pass through. let mut annex_b = std::mem::take(&mut self.annex_b); let convert = is_video && self.is_nal_video(track); if convert { @@ -715,8 +715,8 @@ impl TsMuxer { c } - /// Flush the underlying writer. BD-TS needs no stream trailer, so this - /// only drains buffering; the muxer remains usable afterwards. + /// End the stream: resolve the DTS start-up windows, drain the hold and flush the + /// writer. BD-TS needs no stream trailer. Call once, after the last frame. /// /// Returns [`Error::MuxEmpty`](crate::error::Error::MuxEmpty) when not a /// single frame was emitted: an `m2ts://` sink that wrote only the FMKV @@ -1047,10 +1047,10 @@ mod tests { } #[test] - fn empty_data_keyframe_arms_params_so_later_frames_survive() { - // An empty-data keyframe must still arm params_written; otherwise - // every subsequent non-key frame would be dropped by the - // pre-keyframe guard and the track would emit no real frames. + fn empty_data_keyframe_arms_the_guard_so_later_frames_survive() { + // An empty-data keyframe must still arm the pre-keyframe guard; otherwise + // every subsequent non-key frame would be dropped and the track would + // emit no real frames. let mut sink: Vec = Vec::new(); { let mut mux = TsMuxer::new(&mut sink, &[VIDEO_PID]); @@ -1350,11 +1350,11 @@ mod tests { ); } - // A non-NAL video track must still arm `params_written`, or every later - // non-keyframe fails the drop guard and silently vanishes — same class of - // bug `empty_data_keyframe_arms_params_so_later_frames_survive` guards. + // A non-NAL video keyframe must still arm the pre-keyframe guard, or every later + // non-keyframe silently vanishes — same class of bug + // `empty_data_keyframe_arms_the_guard_so_later_frames_survive` guards. #[test] - fn non_nal_video_keyframe_arms_params_so_later_frames_survive() { + fn non_nal_video_keyframe_arms_the_guard_so_later_frames_survive() { let key: Vec = vec![0x00, 0x00, 0x01, 0xB3, 0xAA, 0xBB]; let non_key: Vec = vec![0x00, 0x00, 0x01, 0xB6, 0xCC, 0xDD]; @@ -1438,6 +1438,34 @@ mod tests { assert_eq!(pats, 13); } + // The PES stream_id follows the declared stream_type: the MPEG audio types (13818-1 + // Table 2-22) take an audio id, everything else private_stream_1. + #[test] + fn mpeg_audio_stream_types_get_an_audio_stream_id() { + let types = [0x03u8, 0x04, 0x0F, 0x81, 0x06]; + let pids: Vec = (0..types.len() as u16).map(|k| AUDIO_PID + k).collect(); + let mut sink: Vec = Vec::new(); + { + let mut mux = TsMuxer::new(&mut sink, &pids); + mux.set_program(types.to_vec()).unwrap(); + for t in 0..types.len() { + mux.write_frame(t, 0, true, &[0xFF, 0xF1, 0x4C, 0x80]) + .unwrap(); + } + mux.finish().unwrap(); + } + let packets = parse_bd_ts(&sink); + let ids: Vec = pids + .iter() + .map(|&pid| { + let first = packets.iter().find(|p| p.pid == pid && p.pusi).unwrap(); + assert_eq!(first.payload[..3], [0, 0, 1]); + first.payload[3] + }) + .collect(); + assert_eq!(ids, [0xC0, 0xC0, 0xC0, 0xBD, 0xBD]); + } + // A PES header's 33-bit PTS and DTS (DTS = PTS when absent), 90 kHz. fn pes_pts_dts(payload: &[u8]) -> (u64, u64) { let ts = |b: &[u8]| { diff --git a/src/mux/videomap.rs b/src/mux/videomap.rs index 2cae9084..da7ab36e 100644 --- a/src/mux/videomap.rs +++ b/src/mux/videomap.rs @@ -404,6 +404,27 @@ mod tests { t } + // An explicit aspect wins; SD without one is unknown; HD falls back to the coded size. + #[test] + fn display_aspect_ratio_falls_back_by_resolution() { + let t = video_title( + Codec::Hevc, + Resolution::R1080p, + FrameRate::F24, + ColorSpace::Bt709, + ); + let DiscStream::Video(mut v) = t.streams[0].clone() else { + unreachable!() + }; + assert_eq!(display_aspect_ratio(&v, 1920, 1080), (1920, 1080)); + assert_eq!(display_aspect_ratio(&v, 0, 0), (0, 1)); + v.display_aspect = Some((16, 9)); + assert_eq!(display_aspect_ratio(&v, 1920, 1080), (16, 9)); + v.display_aspect = None; + v.resolution = Resolution::R576i; + assert_eq!(display_aspect_ratio(&v, 720, 576), (0, 1)); + } + fn src(medium: Medium, path: &str, title: usize) -> SourceInfo { SourceInfo { medium, diff --git a/src/pes.rs b/src/pes.rs index a0252c9a..0e503650 100644 --- a/src/pes.rs +++ b/src/pes.rs @@ -1,10 +1,10 @@ -//! Stream — read PES frames in, write PES frames out. +//! PES frames — the unit that moves between an input and an output. //! -//! A stream is a stream. You read() from it or write() to it. -//! The stream handles its own format internally. +//! A [`PesSource`] yields frames with `read()`; a [`PesSink`] takes them with `write()`. +//! Each handles its own format internally. //! -//! disc.read() → PES frame (sectors → decrypt → demux internally) -//! mkv.write(frame) → MKV file (mux internally) +//! source.read() → PES frame (sectors → decrypt → demux internally) +//! sink.write(frame) → output file or wire (mux internally) // Shared by `serialize`/`deserialize` so the wire format round-trips. // Oversized frames are rejected on write rather than hard-erroring diff --git a/src/scsi/linux.rs b/src/scsi/linux.rs index 03dcdd84..5a87dd33 100644 --- a/src/scsi/linux.rs +++ b/src/scsi/linux.rs @@ -145,9 +145,9 @@ impl SgIoTransport { fd } - // Map errno from a failed open(): permission-denied -> DevicePermission, - // else DeviceNotFound. Path carried in the error; no English text (app - // layer localizes). + // Map errno from a failed open(): EACCES -> DevicePermission, ENOENT/ENXIO/ + // ENODEV -> DeviceNotFound, else (EMFILE, EIO, EBUSY) IoError, so the real + // cause is not reported as an unplug. No English text (app layer localizes). fn open_error(device: &Path) -> Result { Err(Self::map_open_error( &std::io::Error::last_os_error(), @@ -160,10 +160,19 @@ impl SgIoTransport { Error::DevicePermission { path: device.display().to_string(), } - } else { + } else if err.kind() == std::io::ErrorKind::NotFound + || matches!(err.raw_os_error(), Some(libc::ENXIO | libc::ENODEV)) + { Error::DeviceNotFound { path: device.display().to_string(), } + } else { + Error::IoError { + source: match err.raw_os_error() { + Some(code) => std::io::Error::from_raw_os_error(code), + None => std::io::Error::new(err.kind(), err.to_string()), + }, + } } } @@ -208,7 +217,7 @@ impl SgIoTransport { return Err(Error::ScsiError { opcode: cdb[0], status: hdr.status, - sense: Some(super::parse_sense(&sense, hdr.sb_len_wr)), + sense: super::sense_for_status(hdr.status, &sense, hdr.sb_len_wr), }); } Ok(()) @@ -301,11 +310,11 @@ impl Drop for SgIoTransport { } impl ScsiTransport for SgIoTransport { - // Execute via one synchronous SG_IO ioctl; errors map to IoError/ScsiError. fn last_sense_progress(&self) -> Option { self.last_progress } + // Execute via one synchronous SG_IO ioctl; errors map to IoError/ScsiError. fn execute( &mut self, cdb: &[u8], @@ -666,8 +675,6 @@ pub(super) fn disc_presence(path: &Path) -> Result { r } -// After a transport failure the fd is reopened in the background and the next -// execute() adopts it. Needs a real device: FREEMKV_TEST_SG_DEVICE (default sg2). /// Open a block device read-only (no O_DIRECT). Negative on failure. pub(crate) fn open_block_ro(path: &str) -> i32 { let mut bytes = path.as_bytes().to_vec(); @@ -701,6 +708,8 @@ mod recovery_device_tests { use super::*; use std::time::Duration; + // After a transport failure the fd is reopened in the background and the next + // execute() adopts it. Needs a real device: FREEMKV_TEST_SG_DEVICE (default sg2). #[test] #[ignore] fn timeout_does_not_kill_transport() { @@ -741,6 +750,22 @@ mod recovery_device_tests { } // ── CDB guard wiring ─────────────────────────────────────────────────────── +#[cfg(test)] +mod open_error_tests { + use super::*; + + #[test] + fn open_errno_maps_to_its_real_cause() { + let p = Path::new("/dev/sg9"); + let map = |code| SgIoTransport::map_open_error(&std::io::Error::from_raw_os_error(code), p); + assert!(matches!(map(libc::EACCES), Error::DevicePermission { .. })); + assert!(matches!(map(libc::ENOENT), Error::DeviceNotFound { .. })); + assert!(matches!(map(libc::ENODEV), Error::DeviceNotFound { .. })); + assert!(matches!(map(libc::EMFILE), Error::IoError { .. })); + assert!(matches!(map(libc::EBUSY), Error::IoError { .. })); + } +} + #[cfg(test)] mod raw_command_cdb_guard_tests { use super::*; diff --git a/src/scsi/macos.rs b/src/scsi/macos.rs index 0e734f39..ee7adb3e 100644 --- a/src/scsi/macos.rs +++ b/src/scsi/macos.rs @@ -15,9 +15,9 @@ use std::sync::atomic::{AtomicBool, Ordering}; const K_SENSE_DATA_SIZE: usize = 32; -/// Max CDB length the SCSI commands this library issues ever use; also -/// the clamp Linux applies. Used to bound the `cdb_len` passed to the -/// shim so a pathological >255-byte slice can't wrap a `u8`. +/// Max CDB length the SCSI commands this library issues ever use. An +/// over-length CDB is rejected (never clamped) before the `cdb_len` reaches +/// the shim, so a pathological >255-byte slice can't wrap a `u8`. const K_MAX_CDB_SIZE: usize = 16; // The C shim uses a single global IOKit handle, so only one MacScsiTransport may exist at a @@ -493,6 +493,28 @@ mod tests { } } + /// `execute` itself rejects an over-length CDB; the fake device would accept + /// it if the guard were bypassed and the length truncated. + #[test] + fn execute_rejects_an_oversized_cdb_before_the_shim() { + let _globals = FakeDeviceGuard(shim_globals()); + assert!( + !OPEN.swap(true, Ordering::Acquire), + "OPEN held outside SHIM_GLOBALS" + ); + assert_eq!(unsafe { shim_selftest_install_fake_device() }, 0); + let mut transport = MacScsiTransport { + last_progress: None, + }; + for len in [K_MAX_CDB_SIZE + 1, 256 + 6] { + let r = transport.execute(&vec![0u8; len], DataDirection::None, &mut [], 1_000); + assert!( + matches!(r, Err(Error::InvalidCdbLength { .. })), + "{len}-byte CDB: {r:?}" + ); + } + } + /// A CDB exactly at the limit must not trigger the guard, and its length /// reaches the shim verbatim. #[test] diff --git a/src/scsi/macos_shim.c b/src/scsi/macos_shim.c index d4871dbd..f19daa3f 100644 --- a/src/scsi/macos_shim.c +++ b/src/scsi/macos_shim.c @@ -208,15 +208,11 @@ static int conforms_to_any(io_object_t obj, const char *const *classes) { return 0; } -// IOMainPort is macOS 12+; IOMasterPort is its deprecated pre-12 name. +// The default IOKit port is MACH_PORT_NULL (kIOMainPortDefault), on every macOS: no +// IOMainPort/IOMasterPort call, so no availability check the Rust link cannot resolve. static kern_return_t shim_main_port(mach_port_t *mp) { - if (__builtin_available(macOS 12.0, *)) { - return IOMainPort(0, mp); - } -#pragma clang diagnostic push -#pragma clang diagnostic ignored "-Wdeprecated-declarations" - return IOMasterPort(0, mp); -#pragma clang diagnostic pop + *mp = MACH_PORT_NULL; + return kIOReturnSuccess; } // Visitor result: 0 = continue (svc released), 1 = stop and return svc diff --git a/src/scsi/mod.rs b/src/scsi/mod.rs index 1c36373c..8189c361 100644 --- a/src/scsi/mod.rs +++ b/src/scsi/mod.rs @@ -1192,6 +1192,11 @@ mod transport_helper_tests { let v = get_config_010c(&mut t).unwrap(); assert_eq!(v.len(), 28, "header + full 010Ch descriptor"); assert_eq!(&v[12..24], b"202101311259", "date through the minute"); + // MMC-6 §6.6: RT=10b (one feature), Starting Feature 010Ch, Allocation Length 28. + assert_eq!( + t.0, + [SCSI_GET_CONFIGURATION, 0x02, 0x01, 0x0C, 0, 0, 0, 0, 28, 0] + ); } } diff --git a/src/scsi/windows.rs b/src/scsi/windows.rs index 3d400823..b3545ca1 100644 --- a/src/scsi/windows.rs +++ b/src/scsi/windows.rs @@ -254,8 +254,8 @@ pub(super) fn list_drives() -> Vec { drives } -/// TEST UNIT READY probe on Windows. No in-library recovery — see the -/// Linux `disc_presence` doc block for the rationale. +/// TEST UNIT READY probe on Windows. No in-library recovery: the answer is +/// classified by [`super::tur_disc_presence`]. pub(super) fn disc_presence(path: &Path) -> Result { let mut transport = SptiTransport::open(path)?; let cdb = [crate::scsi::SCSI_TEST_UNIT_READY, 0, 0, 0, 0, 0]; diff --git a/src/sector/bus_removal.rs b/src/sector/bus_removal.rs index 020fdf6f..8a1423a6 100644 --- a/src/sector/bus_removal.rs +++ b/src/sector/bus_removal.rs @@ -330,12 +330,18 @@ impl BusGate { Some(h) => head_byte0(h), None => None, }; - // An unreadable or unmapped head is cached as encrypted: never re-read. + // An unreadable head is a guess, not a measurement: it is not + // remembered, so a later read that has the head decides for real. + // An unmapped head can never be measured, so that guess is stable. let e = byte0.is_none_or(|b| b & CPI_MASK != 0); - if self.cpi.len() >= CPI_CACHE_MAX { - self.cpi.clear(); + if byte0.is_some() || head.is_none() { + if self.cpi.len() >= CPI_CACHE_MAX { + self.cpi.clear(); + } + self.cpi.insert(key, e); + } else { + return e; } - self.cpi.insert(key, e); e } }; @@ -658,6 +664,39 @@ mod tests { assert_eq!(s.inner().fua, [true, false]); } + // The wrapper is transparent to the inner source's capabilities and speed control. + #[test] + fn bus_removal_forwards_random_access_and_set_speed() { + struct Seq { + speed: u16, + } + impl SectorSource for Seq { + fn capacity_sectors(&self) -> u32 { + 0 + } + fn read_sectors(&mut self, _: u32, _: u16, _: &mut [u8], _: bool) -> Result { + Ok(0) + } + fn random_access(&self) -> bool { + false + } + fn set_speed(&mut self, kbs: u16) { + self.speed = kbs; + } + } + let mut s = BusRemovalSectorSource::new(Seq { speed: 0 }, BusStage::Passthrough); + assert!(!s.random_access()); + s.set_speed(4500); + assert_eq!(s.inner().speed, 4500); + } + + // An extent running past the 32-bit LBA space is clipped at it, never wrapped. + #[test] + fn bus_map_extent_past_the_lba_space_is_clipped() { + let map = BusMap::from_files(vec![vec![(0xFFFF_FFF0, 100)]]); + assert_eq!(map.covered_ranges(), vec![(0xFFFF_FFF0, 16)]); + } + // A partly overlapping second file keeps its file offset across the clip. #[test] fn bus_map_partial_overlap_clip_keeps_the_file_offset() { @@ -987,10 +1026,10 @@ mod tests { assert_eq!(buf, clear, "unknown head CPI must de-bus"); } - // A unit head that fails to read is fetched once, fast-path (recovery=false), - // and cached as encrypted: neighbours never re-read it. + // A unit head that fails to read is fetched fast-path (recovery=false) and the + // unit guessed encrypted; the guess is not remembered, so each read retries. #[test] - fn host_key_failed_head_fetch_is_cached_as_encrypted() { + fn host_key_failed_head_fetch_guesses_encrypted_without_caching() { struct BadHead { bytes: Vec, reads: Vec<(u32, bool)>, @@ -1036,11 +1075,91 @@ mod tests { assert_eq!(got, clear); assert_eq!( s.inner().reads, - vec![(301, true), (300, false), (302, true)], - "one fast head fetch, then cached" + vec![(301, true), (300, false), (302, true), (300, false)], + "the unreadable head is retried, never cached" ); } + // A head that was unreadable once and is clear when later read: the guess made + // meanwhile must not decide the whole unit. + #[test] + fn host_key_head_guess_is_replaced_by_the_measured_head() { + let rdk = [0x62u8; 16]; + let mut clear = clear_content(3); // LBAs 300..303 + for l in 0..3 { + clear[l * SECTOR_BYTES] &= !0xC0; + } + let mut wire = clear.clone(); + encrypt_bus(&mut wire, &rdk); + let map = Arc::new(BusMap::from_files(vec![vec![(300, 3)]])); + let mut g = BusGate::new(map); + // First pass: sector 301 alone, head 300 unreadable -> guessed encrypted. + let mut b = sector(&wire, 300, 301).to_vec(); + g.debus(&mut b, &rdk, 301, &mut |_| None); + // Second pass: the whole unit with its real (clear) head. + let mut all = clear.clone(); + g.debus(&mut all, &rdk, 300, &mut |_| { + unreachable!("head is in the buffer") + }); + assert_eq!(all, clear, "a clear unit must pass through untouched"); + } + + // Any copy-permission bit set means encrypted (CPI 01/10/11), not just the top bit. + #[test] + fn host_key_any_cpi_bit_marks_the_unit_encrypted() { + let rdk = [0x63u8; 16]; + for (head, encrypted) in [(0x00u8, false), (0x40, true), (0x80, true), (0xC0, true)] { + let mut clear = clear_content(3); + clear[0] = (clear[0] & !0xC0) | head; + let mut wire = clear.clone(); + if encrypted { + encrypt_bus(&mut wire, &rdk); + } + let mut g = BusGate::new(Arc::new(BusMap::from_files(vec![vec![(300, 3)]]))); + g.debus(&mut wire, &rdk, 300, &mut |_| None); + assert_eq!(wire, clear, "head byte0 {head:#x}"); + } + } + + // The decision is kept per (file, unit): interleaved reads of two files' mid-unit + // sectors fetch each head once, and unit 0 of one file does not decide the other's. + #[test] + fn host_key_decisions_are_keyed_by_file_and_unit() { + let rdk = [0x64u8; 16]; + let mut clear = clear_content(6); // file 0 = 300..303 (CPI set), file 1 = 303..306 (clear) + clear[3 * SECTOR_BYTES] &= !0xC0; + let mut wire = clear.clone(); + encrypt_bus(&mut wire[..3 * SECTOR_BYTES], &rdk); + let map = Arc::new(BusMap::from_files(vec![vec![(300, 3)], vec![(303, 3)]])); + let mut g = BusGate::new(map); + let mut heads = Vec::new(); + for lba in [301u32, 304, 302, 305] { + let mut b = sector(&wire, 300, lba).to_vec(); + g.debus(&mut b, &rdk, lba, &mut |h| { + heads.push(h); + Some(sector(&clear, 300, h)[0]) + }); + assert_eq!(b, sector(&clear, 300, lba), "lba {lba}"); + } + assert_eq!(heads, vec![300, 303], "each head fetched once"); + } + + // The per-unit cache is bounded: it is cleared rather than growing with the disc. + #[test] + fn host_key_cpi_cache_stays_bounded() { + let units = CPI_CACHE_MAX as u32 + 8; + let map = Arc::new(BusMap::from_files(vec![vec![(0, units * 3)]])); + let mut g = BusGate::new(map); + let rdk = [0x65u8; 16]; + let mut buf = clear_content(1); + buf[0] &= !0xC0; + for u in 0..units { + g.debus(&mut buf, &rdk, u * 3 + 1, &mut |_| Some(0)); + assert!(g.cpi.len() <= CPI_CACHE_MAX); + } + assert!(g.cpi.len() < units as usize); + } + // Unrecorded extents keep file offsets: [500,1) + hole(1) + [700,4) puts // 700 in unit 0 (head 500) and 701 at the head of unit 1. #[test] diff --git a/src/sector/decrypting.rs b/src/sector/decrypting.rs index 487ec322..9c06423e 100644 --- a/src/sector/decrypting.rs +++ b/src/sector/decrypting.rs @@ -291,8 +291,6 @@ impl DecryptingSectorSource { } // Decrypt `buf` with the active keys, gated by content ranges when set. - // Shared by the first read and post-fetch retry so both agree on which - // units are content and the unit-key try order. fn decrypt_buf( buf: &mut [u8], keys: &mut DecryptKeys, @@ -1396,6 +1394,32 @@ mod tests { assert_eq!(dec.blanked_units(), 1); } + /// A blanked unit reaches the run: its loss counter and one `UnitBlanked` at the read's + /// LBA; a read with no damage reports nothing. + #[test] + fn blanked_units_are_reported_to_the_run() { + let seen = Arc::new(std::sync::Mutex::new(Vec::new())); + let sink = seen.clone(); + let ctx = + crate::ctx::Ctx::default().with_events(Arc::new(move |e: &crate::event::Event<'_>| { + if let crate::event::Event::UnitBlanked { lba, units } = e { + sink.lock().unwrap().push((*lba, *units)); + } + })); + let mut dec = damaged_file_source(4, &[1, 2]); + dec.observe(&ctx); + read_units(&mut dec, 0, 1).expect("intact unit"); + assert!(seen.lock().unwrap().is_empty(), "no damage, no event"); + assert_eq!(ctx.stats.snapshot().units_blanked, 0); + read_units(&mut dec, 1, 2).expect("a damage cluster is blanked"); + assert_eq!( + *seen.lock().unwrap(), + vec![(u64::from(FILE_LBA) + 3, 2)], + "one event: the read's LBA and the unit count" + ); + assert_eq!(ctx.stats.snapshot().units_blanked, 2); + } + /// A cluster of damaged units is blanked and counted however it is read: alone, first, /// with no intact unit around it. KS-2 [BD] §3.10.1: "Each MPEG source packet consists of /// the TP_extra_header (4 bytes) and an MPEG Transport packet". diff --git a/src/sector/mod.rs b/src/sector/mod.rs index 7228edaa..8bfa03c4 100644 --- a/src/sector/mod.rs +++ b/src/sector/mod.rs @@ -36,7 +36,8 @@ pub trait SectorSource: Send { /// /// # Panics /// - /// Implementations may panic if `buf` is undersized. + /// Implementations may panic if `buf` is undersized; `FileSectorSource` + /// returns `DiscRead` instead. fn read_sectors( &mut self, lba: u32, @@ -44,7 +45,6 @@ pub trait SectorSource: Send { buf: &mut [u8], recovery: bool, ) -> Result; - // `FileSectorSource` checks this and returns `DiscRead` instead of panicking. /// Like [`read_sectors`], but with an explicit Force Unit Access request. /// @@ -267,6 +267,50 @@ mod tests { s.set_unit_base(base); } + // FUA and random-access answers must cross the forwarding impls, or Pass-N recovery + // reads lose Force Unit Access and a prefetcher claims random access. + struct FuaSpy(Arc>>); + impl SectorSource for FuaSpy { + fn capacity_sectors(&self) -> u32 { + 0 + } + fn read_sectors(&mut self, _: u32, _: u16, _: &mut [u8], _: bool) -> Result { + self.0.lock().unwrap().push(false); + Ok(0) + } + fn read_sectors_fua( + &mut self, + _: u32, + _: u16, + _: &mut [u8], + _: bool, + fua: bool, + ) -> Result { + self.0.lock().unwrap().push(fua); + Ok(0) + } + fn random_access(&self) -> bool { + false + } + } + + fn fua_through(mut s: S) -> bool { + s.read_sectors_fua(0, 1, &mut [0u8; 2048], false, true) + .unwrap(); + s.random_access() + } + + #[test] + fn forwarding_impls_keep_fua_and_random_access() { + let seen = Arc::new(Mutex::new(Vec::new())); + let boxed: Box = Box::new(FuaSpy(seen.clone())); + assert!(!fua_through(boxed)); + let mut spy = FuaSpy(seen.clone()); + let by_ref: &mut dyn SectorSource = &mut spy; + assert!(!fua_through(by_ref)); + assert_eq!(*seen.lock().unwrap(), vec![true, true]); + } + // Leaf sources with no bus stage: image bytes on file are already bus-clear. const UNMAPPED_LEAVES: &[&str] = &["FileSectorSource", "DirImage"]; diff --git a/src/sector/prefetched.rs b/src/sector/prefetched.rs index 0ca4ee15..044b3f96 100644 --- a/src/sector/prefetched.rs +++ b/src/sector/prefetched.rs @@ -177,10 +177,7 @@ impl PrefetchedSectorSource { crate::sector::read_stage::ExtentWalk::new(extents, policy, unit_align, ctx)?; let loss = walk.loss(); let total_sectors = walk.total_sectors(); - let batch_sectors = match policy { - crate::sector::read_stage::ReadPolicy::Image { batch } - | crate::sector::read_stage::ReadPolicy::Live { batch, .. } => batch, - }; + let batch_sectors = policy.batch(); let unmapped = reader.unmapped_stream_files().to_vec(); let (tx, rx) = bounded::(PREFETCH_CHANNEL_DEPTH); let (recycle_tx, recycle_rx) = bounded::>(PREFETCH_CHANNEL_DEPTH + 1); @@ -1349,6 +1346,86 @@ mod tests { }); } + // After the producer reports an error, the closed channel that follows is a dead source, + // never a clean end of stream. + #[test] + fn a_producer_error_latches_the_source_as_terminated() { + let _serial = serial(); + with_watchdog(Duration::from_secs(10), || { + let extents = vec![Extent { + start_lba: 0, + sector_count: 3, + }]; + let mut pf = PrefetchedSectorSource::new(ErrorSource, extents, 3, &Ctx::default()) + .expect("spawn"); + let mut buf = vec![0u8; 3 * 2048]; + pf.read_sectors(0, 3, &mut buf, false) + .expect_err("the reader's error"); + let again = pf.read_sectors(0, 3, &mut buf, false); + assert!( + matches!(&again, Err(e) if e.is_source_terminated()), + "second read after a producer error: {again:?}" + ); + }); + } + + // A reader that panics is an error to the consumer, not a dropped channel read as EOF. + #[test] + fn a_panicking_reader_is_an_error_not_eof() { + struct Panics; + impl SectorSource for Panics { + fn read_sectors(&mut self, _: u32, _: u16, _: &mut [u8], _: bool) -> Result { + panic!("reader panic (expected by the test)"); + } + fn capacity_sectors(&self) -> u32 { + 3 + } + } + let _serial = serial(); + with_watchdog(Duration::from_secs(10), || { + let extents = vec![Extent { + start_lba: 0, + sector_count: 3, + }]; + let mut pf = + PrefetchedSectorSource::new(Panics, extents, 3, &Ctx::default()).expect("spawn"); + let mut buf = vec![0u8; 3 * 2048]; + let r = pf.read_sectors(0, 3, &mut buf, false); + assert!( + matches!(r, Err(crate::error::Error::DemuxThreadPanicked)), + "{r:?}" + ); + }); + } + + // A byte view ends at the file's real length: the zero-padded tail of the last sector + // is cut, and nothing past it is read. + #[test] + fn a_byte_view_is_clipped_to_the_real_length() { + for len in [3 * 2048 + 700u64, 4 * 2048, 1000] { + let extents = vec![Extent { + start_lba: 0, + sector_count: 8, + }]; + let policy = crate::sector::read_stage::ReadPolicy::Image { batch: 2 }; + let view = ByteView { + prefix: Vec::new(), + len, + }; + let (mut pf, _loss) = PrefetchedSectorSource::file_bytes( + PatternSource { capacity: 16 }, + extents, + policy, + &Ctx::default(), + view, + ) + .expect("spawn"); + let (got, last) = drain_direct(&mut pf, 2, 16); + assert!(matches!(last, Ok(0)), "{last:?}"); + assert_eq!(got.len() as u64, len, "len {len}"); + } + } + // An inner source that answers a mid-extent read with `Ok(0)` has quit early: the producer // must say so, not drop `tx` (which reads as clean EOF and lets `fill_extents` fabricate // zeros for the rest). diff --git a/src/sector/read_stage.rs b/src/sector/read_stage.rs index 197797c3..2ee86a93 100644 --- a/src/sector/read_stage.rs +++ b/src/sector/read_stage.rs @@ -20,7 +20,7 @@ use std::sync::atomic::{AtomicU64, Ordering}; // Ramp back to the preferred batch size after this many clean sectors (100 // MiB = 51,200 sectors) — long enough that noisy zones can't trigger a // premature probe, short enough an isolated failure doesn't lock size 1. -const PROBE_THRESHOLD_SECTORS: u32 = 100 * 1024 * 1024 / 2048; +const PROBE_THRESHOLD_SECTORS: u32 = (100 * 1024 * 1024 / SECTOR) as u32; const SECTOR: usize = crate::consts::SECTOR_BYTES; diff --git a/src/sector/stage.rs b/src/sector/stage.rs index 73b4a56a..78f0ccaa 100644 --- a/src/sector/stage.rs +++ b/src/sector/stage.rs @@ -448,6 +448,35 @@ mod tests { ); } + // A stream shorter than one unit is BD-TS only if every source packet is synced at byte 4. + #[test] + fn a_short_stream_is_bd_ts_only_when_every_packet_syncs() { + let unit = ts_unit(false); + let short = unit[..3 * BD_SOURCE_PACKET_BYTES].to_vec(); + assert_eq!(classify(&short), Kind::BdTs); + let mut one_bad = short.clone(); + one_bad[BD_SOURCE_PACKET_BYTES + 4] = 0x00; + assert_eq!(classify(&one_bad), Kind::Opaque, "one packet off sync"); + let mut sync_at_0 = short; + for p in sync_at_0.chunks_mut(BD_SOURCE_PACKET_BYTES) { + p[0] = 0x47; + p[4] = 0x00; + } + assert_eq!(classify(&sync_at_0), Kind::Opaque, "sync belongs at byte 4"); + } + + // KS-5: a decrypter that leaves CPI set on units it has already decrypted still streams: + // clear TS is never a missing-key refusal, wherever its CPI flag says otherwise. + #[test] + fn clear_ts_with_a_cpi_flag_is_not_refused() { + let bytes: Vec = (0..4).flat_map(|_| ts_unit(true)).collect(); + let mut stage = Stage::lazy(Cursor::new(bytes.clone()), false); + let mut out = Vec::new(); + stage.read_to_end(&mut out).expect("clear TS is not E7022"); + assert_eq!(out, bytes); + assert_eq!(stage.kind(), Some(Kind::BdTs)); + } + // An Opaque stream is handed through byte-identical, even if a later sector looks like a // scrambled pack (a VOB muxed into an MKV attachment stays untouched). #[test] diff --git a/src/session.rs b/src/session.rs index 226cffb6..a1a4f7e0 100644 --- a/src/session.rs +++ b/src/session.rs @@ -276,8 +276,7 @@ impl DiscSession { return Err(e); } }; - self.disc = Some(disc); - Ok(self.disc.as_ref().expect("disc just stored")) + Ok(&*self.disc.insert(disc)) } /// [`Self::scan`] under the session's op token (stop design §2.2, §6 ST-L3): the @@ -306,8 +305,7 @@ impl DiscSession { return Err(e); } }; - self.disc = Some(disc); - Ok(self.disc.as_ref().expect("disc just stored")) + Ok(&*self.disc.insert(disc)) } /// End the session on the handle it holds (stop design §2.5), never re-opening the diff --git a/src/test_util_golden.rs b/src/test_util_golden.rs index 5ed59be7..b3b1bb53 100644 --- a/src/test_util_golden.rs +++ b/src/test_util_golden.rs @@ -37,7 +37,7 @@ impl Golden { self } - /// Record the length and SHA-256 of `bytes` under `key`. + /// Record the length and the first 16 bytes (hex) of the SHA-256 of `bytes` under `key`. pub fn bytes(&mut self, key: &str, bytes: &[u8]) -> &mut Self { use sha2::{Digest, Sha256}; let digest: String = Sha256::digest(bytes) diff --git a/src/udf.rs b/src/udf.rs index d56a0c20..a97d9d64 100644 --- a/src/udf.rs +++ b/src/udf.rs @@ -139,7 +139,8 @@ pub(crate) struct AbsExtent { pub struct DirEntry { pub name: String, pub is_dir: bool, - /// LBA within the metadata partition (add metadata_start for absolute) + /// Block within the metadata partition. Not `metadata_start() + meta_lba` when the + /// Metadata File is fragmented: later extents live elsewhere on disc. pub meta_lba: u32, /// File size in bytes (from ICB info_length) pub size: u64, @@ -320,9 +321,15 @@ impl UdfFs { } // Trim to the real file size, or to the requested prefix — whichever is - // smaller. If extents under-covered the file (e.g. sparse), leave what - // we have rather than over-reporting. + // smaller. Extents covering less than that mean a damaged File Entry. let trim_to = (entry.size as usize).min(limit); + if data.len() < trim_to { + return Err(Error::DiscRead { + sector: self.partition_start as u64, + status: None, + sense: None, + }); + } if data.len() > trim_to { data.truncate(trim_to); } @@ -397,14 +404,16 @@ impl UdfFs { if !seen.insert(e.meta_lba) { continue; } - match self.extents_abs_at(reader, e.meta_lba) { + match self.data_extents_abs(reader, e) { Ok(exts) => ranges.extend( exts.iter() .filter(|x| x.recorded && x.len > 0) .map(|x| (x.lba, (x.len as u64).div_ceil(SECTOR_BYTES_U64) as u32)), ), - Err(Error::Halted) => return Err(Error::Halted), - Err(_) => {} // embedded data, or unreadable: nothing to stage + // Embedded data lives in the File Entry already staged above. + Err(Error::UdfEmbeddedData) => {} + // An unread File Entry would stage a tree missing that file's data. + Err(e) => return Err(e), } compact_ranges(&mut ranges, self.meta.start())?; } @@ -412,6 +421,27 @@ impl UdfFs { Ok(merge_ranges(&ranges)) } + // A directory's FID data is metadata-partition-relative, a file's physical-partition-relative. + fn data_extents_abs( + &self, + reader: &mut dyn SectorSource, + e: &DirEntry, + ) -> Result> { + if !e.is_dir { + return self.extents_abs_at(reader, e.meta_lba); + } + self.read_icb_extents(reader, e.meta_lba)? + .into_iter() + .map(|x| { + Ok(AbsExtent { + lba: self.meta_to_abs(x.lba)?, + len: x.len, + recorded: x.recorded, + }) + }) + .collect() + } + fn collect_file_ranges( &self, reader: &mut dyn SectorSource, @@ -1532,22 +1562,31 @@ fn read_file_size(reader: &mut dyn SectorSource, meta: &MetaMap, meta_lba: u32) } // Decode an OSTA CS0 string: first byte is a compression ID (8 = one byte per code point, -// i.e. Latin-1; 16 = UTF-16BE). NULs are dropped. +// i.e. Latin-1; 16 = UTF-16BE, surrogate pairs combined). Control characters (NUL included) +// are dropped. fn decode_cs0(data: &[u8]) -> String { let Some((&comp, rest)) = data.split_first() else { return String::new(); }; let s: String = match comp { - 16 => rest - .as_chunks::<2>() - .0 - .iter() - .filter_map(|&c| char::from_u32(u16::from_be_bytes(c) as u32)) - .collect(), + 16 => char::decode_utf16( + rest.as_chunks::<2>() + .0 + .iter() + .map(|&c| u16::from_be_bytes(c)), + ) + .filter_map(|c| c.ok()) + .collect(), 8 => rest.iter().map(|&b| b as char).collect(), _ => String::from_utf8_lossy(rest).into_owned(), }; - s.replace('\0', "").trim().to_string() + // Names reach error text and logs: no control characters. A '/' is kept so the + // structure-file filter still rejects the name. + s.chars() + .filter(|c| !c.is_control()) + .collect::() + .trim() + .to_string() } // Parse a UDF filename (an OSTA CS0 string). @@ -1574,7 +1613,8 @@ fn compact_ranges(ranges: &mut Vec<(u32, u32)>, meta_start: u32) -> Result<()> { } /// Merge overlapping or adjacent (start, count) ranges. Caller sorts by start -/// first. Shared range utility — also used to build the disc's encrypted-content +/// first; zero-length ranges are kept (unlike `whole_disc::merge_ranges`, which sorts and +/// drops them). Shared range utility — also used to build the disc's encrypted-content /// extent map (see `Disc::encrypted_content_ranges`). pub(crate) fn merge_ranges(ranges: &[(u32, u32)]) -> Vec<(u32, u32)> { let mut result: Vec<(u32, u32)> = Vec::new(); @@ -6361,6 +6401,39 @@ mod audit_tests { assert_eq!(r.reads[&5], 1, "shared ICB read once (non_stream_ranges)"); } + // A File Entry that fails to read must fail the staging plan (else the staged image + // silently lacks that file's data); an embedded-data entry is skipped, its data + // already being in the staged File Entry. + #[test] + fn non_stream_ranges_fail_on_an_unread_entry_but_skip_embedded_data() { + struct FailAt(GenReader, u32); + impl SectorSource for FailAt { + fn read_sectors(&mut self, lba: u32, n: u16, buf: &mut [u8], r: bool) -> Result { + if lba == self.1 { + return Err(Error::DiscRead { + sector: lba as u64, + status: None, + sense: None, + }); + } + self.0.read_sectors(lba, n, buf, r) + } + } + let fs = fs_of(0, 1, vec![file("A", 5), file("B", 6)]); + let err = fs.non_stream_ranges(&mut FailAt(GenReader::new(1), 6)); + assert!( + matches!(err, Err(Error::DiscRead { sector: 6, .. })), + "{err:?}" + ); + + let mut r = GenReader::new(1); + let mut embedded = efe(0); + embedded[34..36].copy_from_slice(&3u16.to_le_bytes()); + r.over.insert(6, embedded); + fs.non_stream_ranges(&mut r) + .expect("embedded data is not an error"); + } + #[test] fn metadata_ranges_propagate_a_stop() { let fs = fs_of(0, 1, vec![file("F", 7)]); @@ -6532,6 +6605,139 @@ mod audit_tests { ); } + // Marks sector data by which entry point served it; forwards the trait's extras. + struct Probe { + unmapped: Vec, + } + + impl SectorSource for Probe { + fn read_sectors(&mut self, _: u32, count: u16, buf: &mut [u8], _: bool) -> Result { + buf[..count as usize * 2048].fill(0x11); + Ok(count as usize * 2048) + } + fn read_sectors_fua( + &mut self, + lba: u32, + count: u16, + buf: &mut [u8], + recovery: bool, + fua: bool, + ) -> Result { + if fua { + buf[..count as usize * 2048].fill(0x22); + return Ok(count as usize * 2048); + } + self.read_sectors(lba, count, buf, recovery) + } + fn unmapped_stream_files(&self) -> &[crate::sector::bus_removal::UnmappedStreamFile] { + &self.unmapped + } + fn random_access(&self) -> bool { + false + } + } + + #[test] + fn buffered_reader_sends_a_fua_read_to_the_medium_and_forwards_the_source_traits() { + let mut inner = Probe { + unmapped: vec![crate::sector::bus_removal::UnmappedStreamFile::new( + "/BDMV/STREAM/00001.m2ts".into(), + 7, + &Error::Halted, + )], + }; + let mut br = BufferedSectorReader::new(&mut inner, 8); + let mut buf = [0u8; 2048]; + br.read_sectors(100, 1, &mut buf, true).unwrap(); + assert_eq!(buf[0], 0x11); + br.read_sectors_fua(100, 1, &mut buf, true, true).unwrap(); + assert_eq!( + buf[0], 0x22, + "a FUA read must not be answered from the cache" + ); + assert!(!br.random_access()); + assert_eq!(br.unmapped_stream_files().len(), 1); + } + + #[test] + fn non_stream_ranges_cover_structures_and_non_stream_data_only() { + let mut r = GenReader::new(0); + let mut a = efe(16); + put_ad(&mut a, 0, 2048, 20); + put_ad(&mut a, 1, 2048 | (1 << 30), 700); + r.over.insert(300, a); + let mut s = efe(8); + put_ad(&mut s, 0, 10 * 2048, 9000); + r.over.insert(103, s); + let dir = |name: &str, meta_lba, entries| DirEntry { + name: name.into(), + is_dir: true, + meta_lba, + size: 0, + entries, + }; + let stream = dir("STREAM", 2, vec![file("00000.m2ts", 3)]); + let bdmv = dir("BDMV", 1, vec![stream, file("a.bin", 4)]); + let mut fs = fs_of(100, 8, vec![bdmv]); + fs.meta = MetaMap(vec![(100, 4), (300, 4)]); + fs.partition_start = 50; + assert_eq!( + fs.non_stream_ranges(&mut r).expect("ranges"), + vec![(0, 50), (70, 1), (100, 8), (300, 4)] + ); + } + + #[test] + fn utf16_cs0_combines_surrogate_pairs() { + // U+1F600 is D83D DE00. + assert_eq!( + parse_udf_name(&[16, 0, b'A', 0xD8, 0x3D, 0xDE, 0x00]), + "A\u{1F600}" + ); + } + + #[test] + fn cs0_names_carry_no_control_characters() { + assert_eq!( + parse_udf_name(&[8, b'a', 0x1B, b'[', b'2', b'J', b'b']), + "a[2Jb" + ); + } + + #[test] + fn read_file_fails_when_extents_cover_less_than_the_declared_size() { + let mut r = GenReader::new(0); + let mut icb = efe(8); + put_ad(&mut icb, 0, 2048, 10); + r.over.insert(5, icb); + let mut f = file("F", 5); + f.size = 8192; + let fs = fs_of(0, 1, vec![f]); + assert!(matches!( + fs.read_file(&mut r, "/F"), + Err(Error::DiscRead { .. }) + )); + } + + #[test] + fn non_stream_ranges_map_directory_data_through_the_metadata_partition() { + let mut r = GenReader::new(0); + let mut icb = efe(8); + put_ad(&mut icb, 0, 2048, 3); + r.over.insert(105, icb); + let mut dir = file("D", 5); + dir.is_dir = true; + let mut fs = fs_of(100, 1, vec![dir]); + fs.partition_start = 5000; + let ranges = fs.non_stream_ranges(&mut r).expect("ranges"); + let covers = |lba: u32| ranges.iter().any(|&(s, n)| (s..s + n).contains(&lba)); + assert!( + covers(103), + "directory data sits at metadata block 3: {ranges:?}" + ); + assert!(!covers(5003), "not at partition_start + 3: {ranges:?}"); + } + #[test] fn eight_bit_cs0_is_latin1() { assert_eq!(parse_udf_name(&[8, 0xE9, b'A']), "\u{e9}A"); diff --git a/src/unlock_bridge.rs b/src/unlock_bridge.rs index bb8572ed..812a4f13 100644 --- a/src/unlock_bridge.rs +++ b/src/unlock_bridge.rs @@ -353,6 +353,41 @@ mod tests { assert!(fu::scsi::ScsiSense::from_buf(&sense).is_illegal_request()); } + /// A `DiscRead` carrying a real status + sense is a drive answer like any CHECK CONDITION: + /// it must not collapse to a dead bus. + #[test] + fn disc_read_preserves_status_and_sense() { + let err = adapt(|| crate::error::Error::DiscRead { + sector: 7, + status: Some(0x02), + sense: Some(crate::scsi::ScsiSense { + sense_key: 0x05, + asc: 0x24, + ascq: 0x00, + }), + }); + assert_eq!(err.status, 0x02); + let sense = err.sense.expect("sense preserved"); + assert!(fu::scsi::ScsiSense::from_buf(&sense).is_illegal_request()); + } + + /// Each host-cert field lands in its own slot: v1 and v2 material never swap. + #[test] + fn host_certs_map_field_for_field() { + let cert = crate::aacs::types::HostCert { + private_key: [1; 20], + certificate: vec![2; 92], + private_key_v2: Some([3; 32]), + certificate_v2: Some(vec![4; 8]), + }; + let mapped = map_host_certs(std::slice::from_ref(&cert)); + assert_eq!(mapped.len(), 1); + assert_eq!(mapped[0].private_key, [1; 20]); + assert_eq!(mapped[0].certificate, vec![2; 92]); + assert_eq!(mapped[0].private_key_v2, Some([3; 32])); + assert_eq!(mapped[0].certificate_v2, Some(vec![4; 8])); + } + /// A drive-tagged transport fault (status 0xFF) crosses unchanged. #[test] fn scsi_transport_fault_maps_unchanged() { diff --git a/src/whole_disc_tests.rs b/src/whole_disc_tests.rs index 428a2b20..7ef7c3f9 100644 --- a/src/whole_disc_tests.rs +++ b/src/whole_disc_tests.rs @@ -173,6 +173,16 @@ fn unit_aligned_widens_reads_onto_each_files_grid() { assert_eq!(r.inner.reads, vec![(305, 3)]); } +// A partial last unit is read to the span's end, never onto the sectors beyond it. +#[test] +fn unit_aligned_does_not_widen_a_read_past_the_span_end() { + let mut r = reader(vec![(100, 4, 100)]); + let mut buf = vec![0u8; 2048]; + r.read_sectors(103, 1, &mut buf, false).unwrap(); + assert_eq!(r.inner.reads, vec![(103, 1)]); + assert_eq!(buf[0], 103); +} + #[test] fn unit_aligned_propagates_inner_read_errors_on_both_paths() { let mut r = reader(vec![(100, 30, 100)]); @@ -208,6 +218,9 @@ fn unit_block_end_pulls_back_to_the_straddling_units_head() { assert_eq!(r.unit_block_end(90, 103), 103); assert_eq!(r.unit_block_end(103, 104), 104); assert_eq!(r.unit_block_end(0, 50), 50); + // The straddled unit's head lies before the span: no pull-back into the previous region. + let r = reader(vec![(300, 9, 299)]); + assert_eq!(r.unit_block_end(250, 300), 300); } // Spec guards (keys-upfront-design §7.8) on the per-file unit grid. @@ -338,6 +351,33 @@ mod spec_guards { } } +// The inner source's access pattern and speed control reach the mux/driver through the wrapper. +#[test] +fn unit_aligned_forwards_random_access_and_speed() { + #[derive(Default)] + struct Sequential { + speed: Option, + } + impl SectorSource for Sequential { + fn capacity_sectors(&self) -> u32 { + 0 + } + fn read_sectors(&mut self, _: u32, _: u16, _: &mut [u8], _: bool) -> Result { + Ok(0) + } + fn random_access(&self) -> bool { + false + } + fn set_speed(&mut self, kbs: u16) { + self.speed = Some(kbs); + } + } + let mut r = UnitAligned::new(Sequential::default(), Vec::new()); + assert!(!r.random_access()); + r.set_speed(1234); + assert_eq!(r.inner.speed, Some(1234)); +} + // UnitAligned wraps the decrypting reader for whole-disc/image copies; it must // relay the inner source's unmapped list (an image built on it would else drop it). #[test] diff --git a/tests/goldens/parity_image_bd_bad_sectors.golden b/tests/goldens/parity_image_bd_bad_sectors.golden index f0468f72..2903f713 100644 --- a/tests/goldens/parity_image_bd_bad_sectors.golden +++ b/tests/goldens/parity_image_bd_bad_sectors.golden @@ -18,7 +18,7 @@ iso-null outcome = completed=true opened=true bytes=26416 errors=0 lost=0 stream iso-mkv-raw refused = E6008 iso-mkv-selection outcome = completed=true opened=true bytes=20016 errors=0 lost=0 streams=2 undelivered=[] iso-mkv-selection out o.mkv = 21563 bytes sha256:cda2cf35944d0511ffd8de92d4a542fd -live-mkv refused = E6008 +live-mkv refused = E6000 live-mkv out o.mkv = 493 bytes sha256:454f3eb165010aa1646faed1279583cf live-mkv-raw refused = E6000 live-mkv-skip outcome = completed=true opened=true bytes=17910 errors=7 lost=6144 streams=3 undelivered=[] diff --git a/tests/goldens/parity_live_bd_bad_sectors.golden b/tests/goldens/parity_live_bd_bad_sectors.golden index 91c3a7d2..409c9d72 100644 --- a/tests/goldens/parity_live_bd_bad_sectors.golden +++ b/tests/goldens/parity_live_bd_bad_sectors.golden @@ -1,4 +1,4 @@ -strict refused = E6008 +strict refused = E6000 strict out o.mkv = 489 bytes sha256:8628b7bc94f1815e2948a4ce9f32c10f skip outcome = completed=true opened=true bytes=17910 errors=7 lost=6144 streams=3 undelivered=[] skip out o.mkv = 19705 bytes sha256:e8bf5f82342443c240a95385c881d3e2 diff --git a/tests/lint_stop.rs b/tests/lint_stop.rs index a6bae9a3..58ad9c7a 100644 --- a/tests/lint_stop.rs +++ b/tests/lint_stop.rs @@ -398,19 +398,29 @@ fn no_thread_sleep_on_op_paths() { let mut hits = Vec::new(); for s in srcs.iter().filter(|s| in_dirs(s, OP_DIRS)) { hits.extend(s.hits("thread::sleep").into_iter().map(|i| (s, i))); - for i in s.hits("thread::{") { - let group = &s.code[i..close_of(s.code.as_bytes(), i + 8)]; - if group - .split(|c: char| !is_ident(c as u8)) - .any(|w| w == "sleep") - { - hits.push((s, i)); - } - } + hits.extend(sleep_imports(&s.code).into_iter().map(|i| (s, i))); } check("thread::sleep on an op path", hits, ALLOW); } +// Offsets of `thread::{...}` import groups that bring `sleep` into scope. +fn sleep_imports(code: &str) -> Vec { + code.match_indices("thread::{") + .map(|(i, _)| i) + .filter(|&i| { + code[i..close_of(code.as_bytes(), i + 8)] + .split(|c: char| !is_ident(c as u8)) + .any(|w| w == "sleep") + }) + .collect() +} + +#[test] +fn sleep_imports_finds_a_grouped_sleep_only() { + let code = "use std::thread::{self, sleep};\nuse std::thread::{self, spawn};\nuse a::thread::{sleeper};"; + assert_eq!(sleep_imports(code), vec![code.find("thread::{").unwrap()]); +} + /// §2.2: "**Every** raw `self.scsi.as_mut().execute` in `drive/mod.rs` and `identity.rs` /// goes through [`Drive::exec`]"; §5.8 widens the grep to all of `drive/`. #[test] @@ -583,11 +593,6 @@ const ALLOW_BASELINE: &[(&str, &str, usize)] = &[ r#"expect(dead_code,reason="payloadsarefaultdetailforDebug;callersmatchthevariant")"#, 1, ), - ( - "src/labels/ctrm.rs", - r#"allow(clippy::items_after_test_module)"#, - 1, - ), ("src/mpls.rs", r#"allow(dead_code)"#, 2), ("src/mux/demux_sink.rs", r#"allow(dead_code)"#, 2), ("src/mux/demux_thread.rs", r#"allow(dead_code)"#, 1), diff --git a/tests/spec_guards.rs b/tests/spec_guards.rs index a878c663..b425d488 100644 --- a/tests/spec_guards.rs +++ b/tests/spec_guards.rs @@ -264,6 +264,26 @@ fn v20_stride_64_evidence() { ); } +/// KS-27: HD DVD `.evo` units carry the flag in PES_scrambling_control (bits 5-4 of byte +/// 20). Literal offsets here, so a drifted constant in the library fails this guard. +#[test] +fn hddvd_flag_is_bits_5_4_of_byte_20() { + let unit_with = |at: usize, b: u8| { + let mut u = vec![0u8; ALIGNED_UNIT_LEN]; + u[at] = b; + aacs_unit_encrypted(&u, ContentFormat::MpegPs) + }; + for b in [0x10, 0x20, 0x30, 0xFF] { + assert!(unit_with(20, b), "byte 20 = {b:#04x}"); + } + for b in [0x00, 0x0F, 0x40, 0x80, 0xC0] { + assert!(!unit_with(20, b), "byte 20 = {b:#04x}"); + } + for at in [0, 19, 21] { + assert!(!unit_with(at, 0x30), "byte {at}"); + } +} + /// per evidence (no public spec); do not change without evidence proving otherwise — /// KS-27: the HD DVD book was "removed from this AACS website due to inactivity". #[test]