libfreemkv/docs/udf.md

UDF 2.50 Filesystem Parser

What is UDF?

UDF (Universal Disc Format) is the filesystem standard for optical media. It is defined by ECMA-167 with extensions from the OSTA (Optical Storage Technology Association). BD-ROM discs use UDF revision 2.50, which introduces the metadata partition -- a critical feature that separates file metadata from file content on disc.

Why UDF 2.50 for Blu-ray?

Older UDF revisions (1.02, 1.50) scatter ICBs (file metadata) and file data across the same partition. On a high-capacity Blu-ray disc (25-100 GB), this creates excessive seeking when the drive needs to read a directory listing or locate a file. UDF 2.50 solves this by placing all metadata into a contiguous region near the beginning of the disc. The drive reads metadata from one compact area and streams file data from another -- no interleaved seeks.

BD-ROM Part 3 of the Blu-ray specification mandates UDF 2.50.

Metadata Partitions

A UDF 2.50 BD-ROM has two logical partitions:

• Partition 0 (Type 1) -- the physical partition. Contains actual file data (m2ts streams, playlist files, etc.). Mapped directly to disc sectors starting at the Partition Descriptor's partitionStartingLocation.

• Partition 1 (Type 2) -- the metadata partition. Contains all ICBs (Inode-like structures), directory data, and the File Set Descriptor. The metadata partition is itself stored as a file within the physical partition. Its location is found by reading an Extended File Entry at LBA 0 of the physical partition.

The key rule: ICBs and directory data live in the metadata partition. File content lives in the physical partition. When an ICB's allocation descriptor gives an LBA, the partition it refers to depends on what the LBA describes -- metadata-relative for directory entries, physical-partition-relative for file data extents.

Pointer Chain

Reading a UDF 2.50 filesystem follows a fixed chain of pointers. Each step reads one or two 2048-byte sectors:

Sector 256: AVDP (Anchor Volume Descriptor Pointer, tag 2)
  |
  v
Sectors 32-63: VDS (Volume Descriptor Sequence)
  |-- Partition Descriptor (tag 5) --> partition_start (physical sector)
  |-- Logical Volume Descriptor (tag 6) --> partition maps, FSD location
  |
  v
Partition Maps in LVD (offset 440):
  |-- Map 0: Type 1 (physical partition)
  |-- Map 1: Type 2 (metadata partition, identified by "*UDF Metadata Partition")
  |
  v
Metadata file ICB at partition_start + 0 (Extended File Entry, tag 266)
  |-- Allocation descriptor --> metadata content location
  |
  v
metadata_start = partition_start + allocation_position
  |
  v
FSD at metadata_start + 0 (File Set Descriptor, tag 256)
  |-- Root Directory ICB: long_ad at offset 400 --> root_lba (metadata-relative)
  |
  v
Root Directory ICB at metadata_start + root_lba (Extended File Entry, tag 266)
  |-- Allocation descriptor --> directory data location (metadata-relative)
  |
  v
Directory data: File Identifier Descriptors (tag 257)
  |-- Each FID names a file/subdirectory and points to its ICB
  |-- Recurse into subdirectories to build the full file tree

AVDP (Sector 256)

The Anchor Volume Descriptor Pointer is always at sector 256 (ECMA-167 section 10.2). It points to the Main Volume Descriptor Sequence. Tag identifier = 2.

VDS (Sectors 32+)

The Volume Descriptor Sequence contains:

• Partition Descriptor (tag 5): byte offset 188 holds partitionStartingLocation -- the absolute sector where the physical partition begins.
• Logical Volume Descriptor (tag 6): byte offset 268 holds the number of partition maps. The partition maps themselves start at offset 440. For BD-ROM, map 0 is Type 1 (physical) and map 1 is Type 2 (metadata).
• Terminating Descriptor (tag 8): signals the end of the VDS.

Metadata File

When two partition maps exist and the second is Type 2, the metadata partition content is located by reading the Extended File Entry at the first sector of the physical partition (partition_start + 0). This ICB's allocation descriptor gives the offset and length of the metadata content within the physical partition.

File Set Descriptor

The FSD (tag 256) sits at metadata-relative LBA 0 (the first sector of the metadata content). It contains a long allocation descriptor at offset 400 pointing to the root directory ICB. The LBA in this long_ad is at bytes 404-407.

Directory Traversal

Each directory is an ICB (Extended File Entry, tag 266, or File Entry, tag 261) whose allocation extent points to directory data. The directory data is a sequence of File Identifier Descriptors (FIDs, tag 257):

FID Field	Offset	Size	Description
Tag	0	2	Always 257
File characteristics	18	1	Bit 1 = directory, bit 3 = parent
L_FI (name length)	19	1	Length of filename
ICB (long_ad)	20	16	Points to the entry's ICB
L_IU	36	2	Implementation use length
Filename	38 + L_IU	L_FI	UDF-encoded filename

FIDs are 4-byte aligned. The parser advances by (38 + L_IU + L_FI + 3) & !3 bytes per entry.

ICB Layout

Both File Entry (tag 261) and Extended File Entry (tag 266) share the same info_length field:

Field	Tag 261 Offset	Tag 266 Offset
info_length (u64)	56	56
L_EA (u32)	168	208
L_AD (u32)	172	212
Allocation descriptors	176 + L_EA	216 + L_EA

Allocation descriptors use the Short Allocation Descriptor format: 4 bytes extent length (upper 2 bits = type), 4 bytes extent position (LBA).

How read_filesystem() Works

The read_filesystem() function in src/udf.rs follows the pointer chain above:

1. Reads sector 256, validates AVDP (tag 2).
2. Scans sectors 32-63 for the Partition Descriptor and Logical Volume Descriptor.
3. If two partition maps exist and the second is Type 2, reads the metadata file ICB at partition_start to find metadata_start.
4. Reads the FSD at metadata_start, extracts the root directory ICB LBA.
5. Calls read_directory() recursively (max depth MAX_DIR_DEPTH = 8) to build the full file tree.

Each directory read involves two sector reads: one for the ICB, then one or more for the directory data. File sizes are read from info_length in each file's ICB.

read_file() reads a file by navigating the directory tree, reading the file's ICB to get its data extent, then reading the data sector by sector from the physical partition (partition_start + LBA, not metadata_start).

Buffered Sector Reads

USB optical drives have ~500ms round-trip latency per SCSI command. Since read_filesystem() and read_file() issue one SCSI READ per sector, a full disc scan can require hundreds of commands -- taking 10+ minutes on USB.

Disc::scan() wraps the drive in a BufferedSectorReader before reading. On a single-sector read, the buffer prefetches a batch of sectors (sized from the kernel's max_hw_sectors_kb for the device) and caches them. Subsequent reads to nearby LBAs return from cache with zero SCSI overhead. After parsing the UDF directory structure, the entire metadata partition is pre-read into the cache, so all ICB lookups during title scanning and encryption resolution are instant.

prefetch_ranges pre-reads multiple sector ranges into the permanent cache, in batch-sized chunks, stored per-sector in a HashMap. This is used to bulk-load all small files (AACS, MPLS, CLPI, META) before scanning. The permanent cache holds one ~2 KB Vec per sector in the HashMap, so the total sector count bounds RAM; a crafted UDF (a bogus metadata-file size in metadata_sector_ranges) could otherwise drive that count to billions. The cumulative prefetched sectors are capped at MAX_PREFETCH_SECTORS (~1 GiB of cache); once exceeded, prefetching stops seeding the cache. The sliding-window read path still serves any LBA on demand, so this only forgoes the bulk speed-up — it never loses data.

The buffer is transparent -- read_filesystem(), read_file(), and all downstream code still call read_sectors(lba, 1, buf) as before. The batching happens inside the SectorSource implementation.

UDF Filename Encoding

UDF filenames use a compression ID as the first byte:

• 8 = 8-bit characters (ASCII)
• 16 = 16-bit big-endian Unicode (UTF-16BE)

The parser handles both encodings. All path lookups are case-insensitive.

BD-ROM Directory Structure

A typical Blu-ray disc has this directory layout:

/
+-- BDMV/
|   +-- index.bdmv          Disc index (title list, first play)
|   +-- MovieObject.bdmv    Movie objects (navigation commands)
|   +-- PLAYLIST/
|   |   +-- 00000.mpls      Main movie playlist
|   |   +-- 00001.mpls      Director's commentary
|   |   +-- ...
|   +-- CLIPINF/
|   |   +-- 00001.clpi      Clip info for 00001.m2ts
|   |   +-- 00002.clpi
|   |   +-- ...
|   +-- STREAM/
|   |   +-- 00001.m2ts      Transport stream (video/audio/subtitle data)
|   |   +-- 00002.m2ts
|   |   +-- ...
|   +-- BACKUP/              Duplicate of index, MovieObject, playlists, clip info
|
+-- AACS/                    AACS encryption data (encrypted discs only)
|   +-- Unit_Key_RO.inf      Unit key file (encrypted)
|   +-- MKB_RW.inf           Media Key Block
|   +-- Content000.cer       Content certificate
|   +-- DUPLICATE/           Backup copies
|
+-- CERTIFICATE/             BD+ certificate data (some discs)

The parser reads from BDMV/PLAYLIST/ and BDMV/CLIPINF/ to discover titles and their sector layouts. The BDMV/STREAM/ directory contains the actual transport streams but is not parsed by the UDF layer -- stream data is read by LBA directly using extents computed from CLPI EP maps.

metadata_sector_ranges() skip policy

metadata_sector_ranges() skips directories named STREAM (case-insensitive) and individual files larger than 50 MB. Nothing else is filtered by name — BACKUP/DUPLICATE are traversed, and MKB_RO.inf is excluded only because it exceeds the 50 MB cap.

metadata_file_location() and read_directory() internals

metadata_file_location returns None when the map does not fit wholly inside the descriptor, or when its partition type identifier is not "*UDF Metadata Partition" — a Virtual (UDF 2.50 2.2.8) or Sparable (2.2.9) partition map is also ECMA-167 3/10.7.3 Type 2 and records unrelated fields at that offset, so its bytes must never be read as a location.

read_directory walks each directory's ICB (Extended File Entry) to find directory data containing File Identifier Descriptors (FIDs); each FID names a file/subdir and points to its ICB. Directories deeper than MAX_DIR_DEPTH are recorded as entries but not descended into. budget tracks total FID entries consumed across the whole tree walk and the walk aborts with Error::DiscRead once it exceeds MAX_TOTAL_DIR_ENTRIES. visited is the set of metadata-relative ICB LBAs already opened as directories; a repeated LBA is a cycle and is skipped. The function's wide argument list (reader, partition/meta offsets, depth, the global entry budget, and the cycle-detection visited-set) is inherent to the walk, not a refactor smell.

extents_abs_at(), file_extents(), and file_extents_addressing()

extents_abs_at resolves multi-extent / Long-AD / continuation ICBs to absolute disc extents. Unrecorded (ECMA-167 4/14.14.1.1 type-1) extents are included: their space is allocated to the file and occupies its byte space, so dropping them would slide every later extent's bytes down by the hole's length in a sequential extraction. They are returned FLAGGED (AbsExtent::recorded == false), because keeping the extent while losing the flag is the same bug the other way round: the caller would then read sectors the file never wrote and ship whatever the media holds there. read_file_limited emits zeros for such an extent; every consumer of this list has to be able to do the same. It is pub(crate) because it returns AbsExtent, itself internal.

file_extents refuses a file that contains an unrecorded extent because a (lba, sector_count) pair cannot say "this range is a hole": reading it splices undefined sectors into the stream as content, and dropping it slides every later extent's byte space. Until an extent can carry the flag end-to-end, refusing is the only truthful answer at this signature — and a title that is not offered is not a title that was silently mis-ripped.

file_extents_addressing is the byte-space-only counterpart: it includes unrecorded extents UNFLAGGED, in a shape identical to file_extents's safe return, because omitting one would misaddress the rest of the file and nothing on this path treats the returned sectors as stream bytes. It stays pub(crate) rather than public: an external consumer reaching for the more general-sounding name would silently obtain a read plan over a hole, and a doc comment alone is not a guard against that — scoping makes the misuse a compile error.

read_icb_extent(): first RECORDED extent, not extents.first()

read_icb_extents retains ECMA-167 4/14.14.1.1 type-1 (allocated, not recorded) descriptors, because dropping one would slide every later extent's data down by the hole's length. But a type-1 extent's lba is where SPACE is allocated, not where bytes live, and read_icb_extent's one caller — file_start_lba — hands its result out as "the absolute starting LBA of a file's first data extent". ifo.rs then uses that as the base for every VTS VOB extent (file_start_lba(IFO) + vtstt_vobs + cell.first_sector), so a file whose FIRST descriptor is type-1 would put the entire video title set at the wrong place on disc, with no error anywhere. read_icb_extent therefore returns the first extent with recorded == true, not extents.first().

read_inline_data() and the AACS .inf regression

Tiny files such as the AACS *.inf key files are routinely embedded directly in the ICB (ICB Tag flags low 3 bits == 3) rather than via out-of-line extents. read_icb_extents finds no real extents for them (it would misparse the embedded payload as allocation descriptors), so read_file must read the inline payload via read_inline_data instead. A 0.31.0 regression added a per-extent MAX_FILE_BYTES cap that turned the misparsed-embedded case into a hard error, which surfaced as autorip "could not read this disc's key files" on discs whose AACS .inf files are ICB-embedded — the keyserver was then never called.

MAX_FILE_BYTES and the AACS MKB

read_file() caps a single unbounded read at MAX_FILE_BYTES (64 MiB) to bound the allocation a crafted ICB info_length/extent length can force. The one legitimately huge file on disc, the AACS MKB_RO.inf, is allocated to a fixed ~128 MiB and zero-padded, so it is NOT read through this unbounded path: read_aacs_inputs_from_reader instead reads a bounded prefix via read_file_prefix and trims to the real record length, so it never trips the cap (and never reads 100+ MiB of padding). A 0.31.0 regression read the MKB through the unbounded path instead, so the cap rejected it, read_aacs_inputs failed, and autorip reported "could not read this disc's key files" without ever contacting the keyserver.

merge_ranges() half-open semantics

merge_ranges takes HALF-OPEN (start, count) ranges, so start + count is the EXCLUSIVE end and two ranges touch exactly when the next start equals the previous end. A range that starts one sector LATER than that has a genuine, untouched sector between them, and merging across it claims coverage of a sector neither input ever described. That matters because the merged output is a boundary, not a hint: disc::merged_extents feeds Disc::encrypted_content_ranges, which decides which sectors the decrypting source treats as content. A sector in a real gap belongs to no title extent — it is nav/UDF/padding — and must stay outside the content gate, not be folded into it.

build_dir_icb_tagged() offsets

build_dir_icb_tagged builds a directory ICB with an explicit descriptor tag, extended attribute length, RAW (unmasked) allocation-descriptor length field and allocation position. ECMA-167 4/14.17 Extended File Entry (tag 266): L_EA at byte 208, L_AD at 212, allocation descriptors start at 216 + L_EA. ECMA-167 4/14.9 File Entry (tag 261): L_EA at 168, L_AD at 172, allocation descriptors start at 176 + L_EA. The extended-attribute area is filled with a recognisable non-zero pattern (0xA5) so that reading the allocation descriptor from the wrong offset cannot silently produce a usable value.

Test fixture builders

build_efe_long builds a tag-266 ICB whose allocation descriptors are LONG ADs (16 bytes: len(4) | lba(4) | part_ref(2) | impl_use(6)); the trailing part_ref/impl_use bytes are left zero, which is what trips the old 8-byte-stride parser into reading a bogus zero-length terminator.

build_entry_ads builds a tag-261/266 entry with short ADs and every disc-controlled field of the descriptor area exposed: l_ea (extended attribute length — descriptors begin at the entry type's own base, 176 for 261 or 216 for 266, plus this; the attribute area is filled with 0xA5 so a descriptor read from the wrong offset cannot silently produce a usable value), l_ad (the DECLARED descriptor-area length, independent of how many descriptors are actually written), and extra (descriptors written immediately after the declared area, i.e. bytes the entry does not claim are descriptors at all).

VDS anchor shape-vs-content fallback tests

An anchor whose recorded VDS extent passes every SHAPE check (length >= 16 sectors, non-zero location, no address wrap) but holds no sequence must still mount, by sweeping the customary location. Shape is a property of the FIELD, not of what is there: a mastering tool that wrote the reserve location, a stale anchor on a rewritten volume, or deliberate corruption all produce this. Selecting the fallback on shape alone would leave the branch a damaged disc actually takes with no recovery path at all — the sibling Metadata File Location chain in the same function already treats its recorded value as a CANDIDATE and falls back on outcome; this is that principle applied one level up.

The recorded extent and the customary fallback are two SWEEPS, not two locations: an anchor may point at the customary LBA and still declare a window narrower than the fallback's. De-duplicating them on start LBA alone then throws the wider sweep away and the sequence is never reached, even though it is exactly where the fallback would have looked. The regression test's fixture records (LBA 32, 16 sectors) — the minimum ECMA-167 3/10.2.1 permits, so it passes every shape check — while the Partition and Logical Volume Descriptors sit at 50/51, inside the customary 32-sector window but past the declared 16. No Terminating Descriptor lies in 32..48, or both sweeps would stop at the same sector and the wider one would buy nothing.

conformant_meta_vol's extent length and position are given distinct byte patterns, and distinct patterns from one another, so a descriptor byte sourced from the wrong offset — or the two fields transposed — changes the reported metadata partition rather than landing on a value that happens to work. The length is an exact multiple of the 2048-byte logical sector, as a real metadata partition is: its only observable is a sector COUNT, which rounds up, so an off-by-one byte in the low half is visible only when the true value sits exactly on the boundary.

u32::MAX overflow guards in BufferedSectorReader

prefetch advances the read LBA with start_lba + offset. A UDF whose metadata descriptor declares a partition near the top of the LBA space (ECMA-167 §14.1: extent_location is an unconstrained Uint32) drove that add past u32::MAX — an 'attempt to add with overflow' panic in debug inside the public Disc::scan, and in release a wrap to a low LBA that filled the sliding cache with a completely different region while cache_start still claimed the high one.

prefetch_ranges walks each disc-derived (start_lba, sector_count) range with start + offset + i. collect_file_ranges permits any LBA up to u32::MAX (ECMA-167 §14.14.1 allocation descriptors are unconstrained Uint32s) and nothing bounds start + count, so a range within one batch of the top of the space overflowed: debug panic inside Disc::scan, or in release a wrap that seeded the PERMANENT cache with this file's bytes keyed at LBA 0 — every later single-sector read of those low LBAs (the AVDP/VDS re-reads) then returned the wrong sector.

The sliding-cache hit test computed cache_start + cache_sectors. Once cache_start is a disc-controlled LBA near u32::MAX (set by a batch read) that add overflowed: debug panic inside SectorSource::read_sectors, release wrap to a small value that silently disabled the cache.

file_extents() refusal and continuation-cycle test rationale

file_extents builds the extent list a Blu-ray / HD-DVD TITLE is actually ripped through (disc/bluray.rs, disc/hddvd.rs, mux/resolve.rs). Its (lba, sector_count) tuple cannot say "this range is a hole", and both ways of pretending otherwise are wrong: folding an unrecorded extent in makes the mux read undefined sectors and splice whatever the media holds there into the stream as content, while dropping it slides every later extent's byte space (that list IS a byte-space map, aacs::segment::clip_byte_to_lba). So at this signature the only truthful answer is refusal.

icb_extents_exhausted_continuation_budget_is_an_error_not_a_short_list covers hostile input: a type-3 continuation descriptor whose continuation block points back at itself (a cycle). The MAX_AD_BLOCKS bound must make this TERMINATE rather than loop forever, and terminating by exhausting the budget is not the same as reaching the end of the chain. It used to fall through to Ok(extents) with the list silently cut short at whatever the 256th hop had collected; disc/extract.rs then zero-pads the missing tail out to the entry's declared size, sets complete = true with bytes_unreadable = 0, and renames off .partial — a mostly-zero file delivered as a verified complete extraction. Exhausting the budget means "I do not know the rest of this file", and the only honest answer is an error.

Embedded-data and zero-length-hole test fixtures

AD type 3 is EMBEDDED data, not a descriptor list. RED BEFORE GREEN: with the old _ => 8 fallback, read_icb_extents returned Ok([IcbExtent { lba: 999, len: 2048 }]) — an extent decoded out of the file's own CONTENT bytes. A rip would then read sector 999, which holds something else entirely, and emit it as this title's stream at rc=0. That is the failure the refusal exists to prevent, so the icb_extents_embedded_data_is_refused_not_decoded_as_descriptors fixture deliberately makes the content decode as a PLAUSIBLE extent rather than as garbage: garbage would have been caught by the terminator check anyway, and would prove nothing.

file_extents refuses a file whose extent list contains an unrecorded hole, because reading one splices undefined sectors into the rip and dropping one slides every later extent's byte space. Neither is true of a hole with LENGTH ZERO: it displaces nothing, every later extent sits at the same offset with or without it, and the callers' own sectors > 0 && lba > 0 filter already discards it. Refusing on !recorded alone therefore dropped whole titles off discs that ripped correctly before — the reverse of the defect the refusal exists for, and just as silent, since the title simply vanishes. file_extents_accepts_a_zero_length_unrecorded_extent covers this.

build_efe_ext() extended allocation descriptor layout

build_efe_ext builds an Extended File Entry (tag 266) ICB whose allocation descriptors are EXTENDED ADs (ECMA-167 §14.14.3, 20 bytes each): ExtentLength(4) | RecordedLength(4) | InformationLength(4) | ExtentLocation lb_addr { logicalBlockNumber(4) | partitionRef(2) } | impl_use(2). The 30-bit length + 2-bit type live in ExtentLength (offset +0); the logical block number lives in ExtentLocation at offset +12. It sets ICB Tag flags (abs offset 34) low bits to 2 = Extended AD so the parser must select the 20-byte stride and read the LBA from off+12, not off+4.

References

• ECMA-167: Volume and File Structure of Write-Once and Rewritable Media
• UDF 2.50 (OSTA): Universal Disk Format Specification
• BD-ROM Part 3: Blu-ray Disc Read-Only Format, File System Specifications