pq/libfreemkv
Watch
1
Fork
You've already forked libfreemkv
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Compare commits

base: pq:v0.26.1
Branches Tags
pq:dev
pq:main
pq:qa
..
compare: pq:v0.3.0
Branches Tags
pq:dev
pq:main
pq:qa
171 changed files with 5103 additions and 62768 deletions
Download patch file Download diff file Expand all files Collapse all files

31
.github/workflows/ci.yml vendored
View file

@ -6,37 +6,16 @@ on:
pull_request:
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
with:
components: clippy, rustfmt
- run: cargo fmt --check
# libfreemkv is a library — Cargo.lock is gitignored. --locked
# would always fail on a fresh runner because there's no committed
# lockfile to lock against. The binary crates (freemkv, autorip,
# bdemu) track Cargo.lock and DO use --locked.
- run: cargo clippy -- -D warnings
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- run: cargo test --tests
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo test
check-macos:
runs-on: macos-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- run: cargo check
check-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo check

14
.github/workflows/release.yml vendored
View file

@ -12,7 +12,7 @@ jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
- name: Verify Cargo.toml version matches tag
run: |
CARGO_VER="v$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')"
@ -26,18 +26,16 @@ jobs:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
# libfreemkv is a library — Cargo.lock isn't tracked, so --locked
# would always fail (no lockfile to lock against on a fresh runner).
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo test
publish:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@1.86.0
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- name: Publish to crates.io
run: cargo publish
env:
@ -47,7 +45,7 @@ jobs:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:

3
.github/workflows/update-readme.yml vendored
View file

@ -11,10 +11,9 @@ jobs:
update:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
with:
ref: main
token: ${{ secrets.ORG_DISPATCH_TOKEN }}
- name: Update version in README
run: |

2
.gitignore vendored
View file

@ -3,5 +3,3 @@ Cargo.lock
*.swp
*.swo
.DS_Store
.cargo/
.claude/worktrees/

2417
CHANGELOG.md
View file

File diff suppressed because it is too large Load diff

49
Cargo.toml
View file

@ -1,8 +1,7 @@
[package]
name = "libfreemkv"
version = "0.26.1"
edition = "2024"
rust-version = "1.86"
version = "0.3.0"
edition = "2021"
license = "AGPL-3.0-only"
description = "Open source raw disc access library for optical drives"
repository = "https://github.com/freemkv/libfreemkv"
@ -13,49 +12,17 @@ categories = ["hardware-support", "multimedia"]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha1 = "0.10"
sha2 = "0.10"
aes = "0.8"
cbc = "0.1"
flate2 = "1"
num-bigint = "0.4"
num-traits = "0.2"
num-integer = "0.1"
rand = "0.8"
cmac = "0.7"
zip = { version = "2", default-features = false, features = ["deflate"] }
base64 = "0.22.1"
# Trace-level instrumentation for Disc::copy + SgIoTransport::execute. Permitted
# under project docs ("Acceptable strings: debug/trace logging"). Consumers (autorip)
# wire a tracing subscriber and pipe events into the JSONL debug log.
tracing = "0.1"
# Bounded MPSC channel with kernel-wakeup send_timeout. Used by `io::pipeline`
# so the halt-aware send/finish loops can BLOCK on consumer drain instead of
# polling — the 50 ms poll cadence of the previous mpsc-based impl capped mux
# throughput at ~1 MB/s (see (internal)/memory/
# feedback_send_with_halt_poll_throttle.md, 0.21.7).
crossbeam-channel = "0.5"
# Persistent work-stealing thread pool for parallel AACS unit
# decryption. Per-call std::thread::scope spawned fresh OS threads
# and that overhead dominated for typical batch sizes (60 units).
# rayon's global pool initialises once on first use.
rayon = "1"
# SIMD-accelerated bytestring search. Drives the HEVC/H.264 start-code
# scan in `mux::codec::h264::find_start_code` — naive byte-by-byte
# walk is ~500 MB/s single-thread on x86_64; memchr's vectorised
# `memmem::find` for the 3-byte `00 00 01` needle hits ~5 GB/s on
# AVX2-capable hosts.
memchr = "2"
[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
[target.'cfg(target_os = "macos")'.dependencies]
libc = "0.2"
[dev-dependencies]
tempfile = "3"
[[bench]]
name = "sgio_read"
harness = false
[[bin]]
name = "freemkv-info"
path = "src/bin/freemkv_info.rs"
[[bin]]
name = "freemkv-test"
path = "src/bin/freemkv_test.rs"

20
Makefile
View file

@ -1,20 +0,0 @@
# libfreemkv — local dev helper.
# Mirrors the cross-crate scripts in (internal)/scripts/test-all.sh
# but scoped to this single crate.
.PHONY: test build check ci clean
test:
cargo test --tests
build:
cargo build --release
check:
cargo fmt --check
cargo clippy --all-targets -- -D warnings
ci: check build test
clean:
cargo clean

224
README.md
View file

@ -1,18 +1,10 @@
[![Crates.io](https://img.shields.io/crates/v/libfreemkv)](https://crates.io/crates/libfreemkv)
[![docs.rs](https://img.shields.io/docsrs/libfreemkv)](https://docs.rs/libfreemkv)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE)
[![Drives: 206](https://img.shields.io/badge/drives-206-brightgreen)]()
# libfreemkv
Rust library for 4K UHD / Blu-ray / DVD optical drives. Drive access, disc scanning, stream labels, AACS decryption, CSS decryption, KEYDB updates, and content reading in one crate. Bundled drive profiles — no external files needed.
Built-in keys cover DVDs and Blu-rays (AACS 1.0). For UHD (AACS 2.0 / 2.1) discs, an optional `keydb.cfg` supplies disc-specific volume unique keys.
**12+ MB/s** sustained read speeds on BD. Full init: unlock, firmware upload, speed calibration — all from pure Rust.
Multi-lingual by design — the library outputs structured data and numeric error codes, never English text. Build any UI or localization on top.
**[API Documentation](https://docs.rs/libfreemkv)** · **[Technical Docs](docs/)**
Rust library for raw sector access on optical drives. Identifies drives using standard SCSI commands, matches them against 206 bundled profiles, and unlocks raw read mode. No external files, no configuration.
Part of the [freemkv](https://github.com/freemkv) project.
@ -20,173 +12,113 @@ Part of the [freemkv](https://github.com/freemkv) project.
```toml
[dependencies]
libfreemkv = "0.25"
libfreemkv = "0.2"
```
## Quick Start
```rust
use libfreemkv::{Drive, Disc, ScanOptions};
use libfreemkv::DriveSession;
use std::path::Path;
// Open drive — profiles are bundled, auto-identified
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // wait for disc
drive.init()?; // unlock + firmware upload
drive.probe_disc()?; // probe disc surface for optimal speeds
let mut session = DriveSession::open(Path::new("/dev/sr0"))?;
// Scan disc — UDF, playlists, streams, AACS (all automatic)
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
session.unlock()?; // activate raw read mode
session.calibrate()?; // optimize read speed
for title in &disc.titles {
println!("{} — {} streams", title.duration_display(), title.streams.len());
}
// Stream pipeline — read PES frames from any source, write to any output
let opts = libfreemkv::InputOptions::default();
let mut input = libfreemkv::input("iso://Disc.iso", &opts)?;
let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
}
output.finish()?;
let mut buf = vec![0u8; 2048];
let n = session.read_sectors(0, 1, &mut buf)?;
```
### Multi-pass recovery rip
## How It Works
For damaged discs the library exposes two flat verbs — `Disc::sweep` for the
forward Pass 1 and `Disc::patch` for retrying bad ranges. The library never
loops; the multipass policy is the caller's job. See
[`docs/rip-recovery.md`](docs/rip-recovery.md).
1. **INQUIRY** (SPC-4 §6.4) — reads vendor_id, product_id, product_revision, vendor_specific
2. **GET CONFIGURATION 010C** (MMC-6 §5.3.10) — reads firmware_date
3. **Profile match** — five fields uniquely identify the drive against 206 bundled profiles
4. **READ BUFFER** — single command with per-drive mode/buffer_id activates raw read mode
5. **Signature verify** — drive responds with 4-byte signature + "MMkv" confirmation
No fingerprints. No encrypted lookups. All identification uses standard SCSI fields.
## API
```rust
use libfreemkv::{SweepOptions, PatchOptions};
use libfreemkv::disc::{mapfile, mapfile_path_for};
use std::path::Path;
let mut session = DriveSession::open(device_path)?;
let iso = Path::new("disc.iso");
// Drive identity
session.drive_id.vendor_id // "HL-DT-ST"
session.drive_id.product_id // "BD-RE BU40N"
session.drive_id.product_revision // "1.03"
session.drive_id.vendor_specific // "NM00000"
session.drive_id.firmware_date // "211810241934"
// Pass 1: disc → ISO. Skip-on-error, zero-fill, write the sidecar mapfile.
disc.sweep(&mut drive, iso, &SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true,
progress: None,
halt: None,
})?;
// Profile data
session.profile.chipset // Chipset::MediaTek
session.profile.unlock_mode // 0x01
session.profile.unlock_buf_id // 0x44
session.profile.signature // [0x99, 0x9e, 0xc3, 0x75]
// Pass 2..N: retry every non-finished range. Idempotent.
loop {
let map = mapfile::Mapfile::load(&mapfile_path_for(iso))?;
let stats = map.stats();
if stats.bytes_pending + stats.bytes_unreadable == 0 { break; }
// Operations
session.unlock()?;
session.calibrate()?;
session.read_sectors(lba, count, &mut buf)?;
session.status()?;
session.read_config()?;
session.read_register(index)?;
```
let outcome = disc.patch(&mut drive, iso, &PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true,
wedged_threshold: 50,
progress: None,
halt: None,
})?;
if outcome.bytes_recovered_this_pass == 0 { break; }
## Chipset Support
| Chipset | Status | Drives | Brands |
|---------|--------|--------|--------|
| MediaTek MT1959 | Supported | 206 | LG, ASUS, HP |
| Renesas | Planned | -- | Pioneer |
## Drive Profile
Each bundled profile (compiled into the binary):
```json
{
"vendor_id": "HL-DT-ST",
"product_id": "BD-RE BU40N ",
"product_revision": "1.03",
"vendor_specific": "NM00000",
"firmware_date": "211810241934",
"chipset": "mediatek",
"unlock_mode": 1,
"unlock_buf_id": 68,
"signature": "999ec375",
"register_offsets": ["10e291", "11ab1c"]
}
// Mux from the ISO via the normal stream pipeline (no drive involvement).
```
## What It Does
- **Drive access** — open, identify, unlock, firmware upload, speed calibration, eject
- **12+ MB/s reads** — auto-detects kernel transfer limits, sustained full speed
- **Disc scanning** — UDF 2.50 filesystem, MPLS playlists, CLPI clip info
- **Stream labels** — 5 BD-J format parsers (Paramount, Criterion, Pixelogic, CTRM, Deluxe)
- **AACS decryption** — transparent key resolution and content decrypt (1.0 + 2.0 bus decryption)
- **KEYDB updates** — download, verify, save from any HTTP URL (zero deps, raw TCP)
- **Content reading** — adaptive batch reads with automatic decryption
- **Stream I/O** — unified stream pipeline for reading and writing any format
### Streams
| Stream | Input | Output | Transport |
|--------|-------|--------|-----------|
| DiscStream | Yes | -- | Optical drive via SCSI |
| IsoStream | Yes | -- | Blu-ray ISO image file (read via stream pipeline; written via `Disc::sweep()`) |
| MkvStream | Yes | Yes | Matroska container |
| M2tsStream | Yes | Yes | BD transport stream with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | Raw byte pipe |
| NullStream | -- | Yes | Discard sink (byte counter for benchmarks) |
Streams implement `FrameSource` (read) and/or `FrameSink` (write); direction is type-checked. `input()` / `output()` resolve URL strings to PES stream instances. All URLs use the `scheme://path` format — bare paths are rejected.
### Keys
DVDs (CSS) decrypt out of the box — the 1999-era public player keys are compiled into the library.
Blu-rays and UHD (AACS) require a `keydb.cfg` at `~/.config/freemkv/keydb.cfg` (or passed via `ScanOptions`). The file holds all DKs, PKs, host certs, and per-disc VUKs. No AACS key material is compiled into the binary.
## Architecture
```text
Drive — open, identify, init, unlock, single-shot read
├── ScsiTransport — SG_IO (Linux), IOKit (macOS), SPTI (Windows)
├── DriveProfile — per-drive unlock parameters (bundled)
└── PlatformDriver — MediaTek (supported), Renesas (planned)
Disc — scan titles, streams, AACS/CSS state
├── UDF reader — Blu-ray UDF 2.50 with metadata partitions
├── MPLS parser — playlists → titles + clips + streams
├── CLPI parser — clip info → EP map → sector extents
├── IFO parser — DVD title sets, PGC chains, cell addresses
├── Labels — 5 BD-J format parsers (detect + parse)
├── AACS — key resolution + content decryption
├── CSS — DVD CSS cipher (table-driven, no keys needed)
└── KEYDB — download + verify + save
Streams — unified PES pipeline
├── FrameSource — read() PES frames (direction-typed)
├── FrameSink — write() PES frames (direction-typed)
├── DiscStream — sectors → decrypt → TS demux → PES
├── IsoStream — ISO file → decrypt → TS demux → PES
├── MkvStream — MKV mux/demux
├── M2tsStream — BD transport stream
├── NetworkStream — TCP with FMKV metadata header
├── StdioStream — stdin/stdout pipe
└── NullStream — discard sink
```
See [docs/](docs/) for detailed technical documentation on each module.
## Error Codes
All errors are structured with numeric codes. No user-facing English text — applications format their own messages.
| Code | Error | Meaning |
|------|-------|---------|
| E1000 | DeviceNotFound | Device path doesn't exist |
| E1001 | DevicePermission | No access (try sudo or cdrom group) |
| E2000 | UnsupportedDrive | No matching profile |
| E3000 | UnlockFailed | Unlock command rejected |
| E3001 | SignatureMismatch | Response signature wrong |
| E3002 | NotUnlocked | Operation requires unlock first |
| E4000 | ScsiError | SCSI command failed |
| E5000 | IoError | System I/O error |
| Range | Category |
|-------|----------|
| E1xxx | Device errors (not found, permission) |
| E2xxx | Profile errors (unsupported drive) |
| E3xxx | Unlock errors (failed, signature) |
| E4xxx | SCSI errors (command failed, timeout) |
| E5xxx | I/O errors |
| E6xxx | Disc format errors |
| E7xxx | AACS errors |
| E8xxx | KEYDB update errors |
## Platform Support
## Platform
| Platform | Status | Backend |
|----------|--------|---------|
| Linux | Supported | SG_IO ioctl |
| macOS | Supported | IOKit SCSITask |
| Windows | Supported | SPTI |
| macOS | Planned | IOKit |
| Windows | Planned | SPTI |
The `ScsiTransport` trait abstracts the platform. Adding a backend is one file behind a `cfg` gate.
## Contributing
Run `freemkv info disc:// --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to contribute your drive's profile.
Run `freemkv info --share` with the [freemkv CLI](https://github.com/freemkv/freemkv) to submit your drive's profile.
## License

312
TROUBLESHOOTING.md
View file

@ -1,312 +0,0 @@
# Troubleshooting Guide
Common problems and solutions for optical drive ripping with freemkv.
---
## 1. USB-SATA Bridge Issues
This is the single most common source of problems when ripping discs over USB.
### Symptoms
- The drive disappears mid-rip. The ripping tool reports the device is gone, and `ls /dev/sg*` no longer shows it.
- The device re-enumerates under a different name: `sg4` becomes `sg5`, then `sg7`, then `sg11` after each USB port reset.
- `dmesg` shows USB port resets: `usb X-Y: reset high-speed USB device`, `xhci_hcd 0000:00:14.0: Cannot enable. Maybe the USB cable is bad?`, or `usb-storage: device reset failed`.
- The SCSI layer reports `host_status=7` (Linux USB transport error) in sense data.
- The drive works fine for reading data discs or burning, but crashes when hitting damaged sectors during a rip.
- After the crash, the drive is completely invisible until physically unplugged and reconnected.
### Root Cause
USB-SATA bridges translate between the USB Mass Storage protocol (BOT or UAS) and the drive's native SATA interface. When the optical drive encounters an unreadable sector, it returns a SCSI CHECK CONDITION with sense key 0x03 (MEDIUM ERROR). Some bridge chipsets -- particularly the Initio INIC-36xx family -- have buggy firmware that mishandles this error response.
Specific failure modes:
- **Incorrect residue reporting.** The bridge claims a different number of bytes transferred than what actually occurred. The Linux USB storage driver sees this discrepancy as a protocol violation and resets the port to recover. The `US_FL_IGNORE_RESIDUE` quirk exists specifically for this class of bug (see `drivers/usb/storage/transport.c` in the Linux kernel).
- **Bridge firmware crash.** On some Initio bridges, a malformed SCSI error response from the drive causes the bridge MCU to hang entirely. The USB controller sees the device stop responding and initiates a port reset. The bridge recovers (it re-enumerates), but the rip is dead -- all state is lost.
- **Sense data corruption.** The bridge forwards garbled or truncated sense data to the host, which the SCSI midlayer cannot parse, leading to a transport reset.
This is a hardware + firmware problem, not a software bug. The same drive connected via direct SATA does not exhibit these symptoms.
### Known Affected Bridges
| Chipset | USB IDs | Notes |
|---------|---------|-------|
| Initio INIC-3609 | `13fd:3609` | Very common in cheap SATA-to-USB enclosures. Highly problematic. |
| Initio INIC-3619 | `13fd:3940` | Same firmware family as INIC-3609. |
| Initio INIC-3069 | `13fd:0840` | Older variant, same residue bug. |
| ASMedia ASM1051 | `174c:5106` | Early ASM SATA bridge. Residue issues on error paths. |
| JMicron JMB36x | `152d:0561` | Some firmware versions. Not all JMicroon chips are affected. |
If your drive came in a pre-built external enclosure (Vantec, Sabrent, OWC, etc.), it almost certainly uses one of these bridge chips internally.
### The Fix: USB Storage Quirk
Apply the `US_FL_IGNORE_RESIDUE` kernel quirk for your bridge. This tells the Linux USB storage driver to ignore the residue field in SCSI response frames, preventing the port reset on mismatched byte counts.
**Step 1: Identify your bridge's vendor:product ID.**
```bash
lsusb
```
Look for your drive's entry. Example output:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation INIC-3609
```
Here the vendor ID is `13fd` and the product ID is `0840`.
**Step 2: Apply the quirk at runtime.**
```bash
echo "13fd:0840:i" > /sys/module/usb_storage/parameters/quirks
```
Replace `13fd:0840` with your device's actual IDs. The `:i` flag means `US_FL_IGNORE_RESIDUE`.
You can combine multiple flags. Common additions:
- `:i` -- ignore residue (`US_FL_IGNORE_RESIDUE`)
- `:u` -- force BOT mode instead of UAS, for bridges with UAS bugs
**Step 3: Reconnect the drive.** Unplug and replug the USB cable, or bind/unbind the device. The quirk is applied per-module-load, so existing sessions may need the drive reconnected.
### Making It Persistent
Add the quirk to your kernel boot parameters so it survives reboots.
Edit `/etc/default/grub` (GRUB) and add to `GRUB_CMDLINE_LINUX_DEFAULT`:
```
GRUB_CMDLINE_LINUX_DEFAULT="quiet usb_storage.quirks=13fd:0840:i"
```
Then rebuild the GRUB config:
```bash
sudo update-grub
```
For systemd-boot, add to your loader entry or `/etc/kernel/cmdline`:
```
usb_storage.quirks=13fd:0840:i
```
Multiple devices can be separated by commas:
```
usb_storage.quirks=13fd:0840:i,174c:5106:u
```
### Recommended Bridges
If you are buying a USB-SATA adapter or enclosure for optical drive use:
| Bridge | USB IDs | Notes |
|--------|---------|-------|
| ASMedia ASM1153 | `174c:1153` | Reliable. Widely available in SATA-USB 3.0 cables. |
| JMicron JMS578 | `152d:0578` | Good firmware. Supports UASP. |
| Icy Box IB-AC640-C3 | N/A | Uses a known-good bridge internally. Plug-and-play. |
Avoid any enclosure or adapter listing an Initio chipset.
### Best Solution: Direct SATA
Connect your optical drive directly to a motherboard SATA port. This eliminates the USB-SATA bridge entirely and is the most reliable configuration:
- No USB protocol overhead or translation errors.
- No bridge firmware bugs.
- No port resets or re-enumeration.
- Full SATA error recovery handled natively by the kernel's libata driver.
- Sustained read speeds are limited only by the drive, not the USB bus.
If your machine has a free SATA port, use it.
---
## 2. Damaged Disc Handling
### Symptoms
- SCSI MEDIUM ERROR (sense key 0x03) at specific LBAs. `dmesg` shows `sr X:0:0:0: [srY] Unrecoverable read error` or similar.
- Read speed drops to near zero when approaching a damaged area.
- The drive makes audible retrying noises (laser repositioning, spindle speed changes).
- On USB-connected drives: the bridge crashes (see section 1 above) when the drive returns the error.
### How freemkv Handles This
freemkv uses a three-layer recovery model. See [`docs/rip-recovery.md`](docs/rip-recovery.md) for full details.
- **Pass 1 (Disc::copy):** Fast sweep with 64 KB reads. On failure, zero-fills the block and skips forward. Writes a ddrescue-format mapfile for later retry.
- **Pass 2+ (Disc::patch):** Targeted re-reads of bad ranges with a long 30-second timeout per CDB. The drive firmware performs its own ECC and laser power retries within that window.
- **In-stream (DiscStream):** Adaptive batch halving -- reduces request size on failure to isolate bad sectors within a larger block.
This means a disc with some bad sectors will still produce a usable ISO. The damaged areas are zero-filled in pass 1 and retried in subsequent passes. Structure-protected sectors (deliberate unreadable regions from copy protection) will never yield, which is expected.
### The Drive Taint Issue (LG BU40N)
Some drives, notably the LG BU40N, exhibit a "taint" behavior after encountering MEDIUM ERRORs:
1. The drive hits a damaged sector and returns a MEDIUM ERROR.
2. From that point forward, **all subsequent reads fail** -- even reads to sectors that were previously successful.
3. The only recovery is to physically unplug and reconnect the drive (or power-cycle it).
This is not a freemkv bug. It is a drive firmware behavior triggered by the interaction between the drive's internal error recovery and the USB-SATA bridge's handling of the error response. The drive firmware enters a degraded state that it does not recover from without a power cycle.
Workarounds:
- **Use a direct SATA connection.** This eliminates the bridge interaction that triggers the taint.
- **Use a different bridge.** The ASM1153 and JMS578 are less likely to trigger this behavior.
- **Accept the partial ISO.** freemkv's skip-forward recovery will zero-fill the unreadable blocks and continue. The resulting ISO may be playable with minor glitches in the affected areas.
- **Physical replug between retry passes.** If running multi-pass patch, replug the drive between passes to clear the taint state.
freemkv deliberately does not attempt inline SCSI resets or eject cycles to recover from this state, because those operations were found to make the problem worse on affected hardware (see the design rationale in [`docs/rip-recovery.md`](docs/rip-recovery.md)).
---
## 3. Drive Not Detected
### Check Hardware Visibility
```bash
lsusb
```
Verify the drive appears in the USB device list. If it does not show up, the drive is not visible to the host at all -- check cables, power, and USB port.
```bash
ls /dev/sg*
```
On Linux, optical drives appear as `/dev/sg*` devices (the SCSI Generic interface). freemkv uses `/dev/sg*`, not `/dev/sr*`. If `lsusb` shows the device but no `/dev/sg*` entry exists, the `sg` kernel module may not be loaded:
```bash
sudo modprobe sg
```
### Check Kernel Messages
```bash
dmesg | grep -i usb | tail -30
dmesg | grep -i sg | tail -10
```
Look for:
- USB enumeration errors or failed port resets.
- `sg_add` messages confirming the sg device was registered.
- Permission denied or access errors.
### Permission Issues
On most Linux distributions, `/dev/sg*` devices are owned by `root:disk` or `root:cdrom` with restricted permissions. Running freemkv as an unprivileged user will fail with permission errors.
Options:
- Add your user to the appropriate group:
```bash
sudo usermod -aG disk $USER
```
Then log out and back in for the change to take effect. On some distributions the group is `cdrom` or `optical` instead of `disk`.
- Run with elevated privileges:
```bash
sudo freemkv ...
```
- Install a udev rule for persistent per-device permissions. Create `/etc/udev/rules.d/99-sg-optical.rules`:
```
SUBSYSTEM=="scsi_generic", ATTRS{type}=="5", MODE="0666"
```
Then reload udev rules:
```bash
sudo udevadm control --reload-rules && sudo udevadm trigger
```
### Spin-Up Delay
Optical drives take 30-60 seconds to spin up and become ready after hot-plug or disc insertion. During this window, SCSI commands may return NOT READY or timeout.
freemkv's `Drive::wait_ready()` handles this automatically by polling with TEST UNIT READY until the drive responds. If you are writing your own code using the library, always call `wait_ready()` before `init()`:
```rust
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?; // blocks until disc is ready, up to 30s
drive.init()?;
```
If the drive was just plugged in, wait a full minute before concluding it is not detected.
---
## 4. How to Identify Your USB-SATA Bridge
If you are experiencing the issues described in section 1, you need to know which bridge chipset your adapter or enclosure uses.
### Step 1: Find the Device
```bash
lsusb
```
Look for entries matching your drive or enclosure. Bridges may appear under their own manufacturer name or as a generic SATA device. Common examples:
```
Bus 002 Device 005: ID 13fd:0840 Initio Corporation
Bus 002 Device 006: ID 174c:1153 ASMedia Technology Inc. ASM1153
Bus 002 Device 007: ID 152d:0578 JMicron Technology Corp. JMS578
```
### Step 2: Match the IDs
| Vendor | Product ID | Chipset | Status |
|--------|-----------|---------|--------|
| `13fd` | `3609` | Initio INIC-3609 | Affected. Apply quirk. |
| `13fd` | `3940` | Initio INIC-3619 | Affected. Apply quirk. |
| `13fd` | `0840` | Initio INIC-3069 | Affected. Apply quirk. |
| `174c` | `5106` | ASMedia ASM1051 | Affected (early firmware). Apply quirk. |
| `174c` | `1153` | ASMedia ASM1153 | Good. No quirk needed. |
| `152d` | `0561` | JMicron JMB36x | Affected (some firmware). Apply quirk if issues occur. |
| `152d` | `0578` | JMicron JMS578 | Good. No quirk needed. |
### Step 3: Check dmesg for the Bridge Name
```bash
dmesg | grep -i "usb-storage\|uas\|initio\|asmedia\|jmicron"
```
This often reveals the bridge chipset even when `lsusb` shows a generic name.
### Step 4: If the Enclosure Is Sealed
Many external drive enclosures (Vantec NexStar, Sabrent, OWC, etc.) do not advertise the bridge chipset on the packaging. In this case:
1. Check `lsusb` while the enclosure is connected.
2. Search the vendor:product ID online -- there are community-maintained lists of which chipsets popular enclosures use.
3. If you cannot determine the chipset and are experiencing bridge crashes, assume it is an Initio and apply the quirk with its IDs.
4. The definitive test: connect the bare drive to a motherboard SATA port. If the problems disappear, the bridge was the cause.
---
## 5. General Debugging Checklist
When something goes wrong during a rip, gather this information before filing an issue:
1. **freemkv version:** `freemkv --version` or the crate version in `Cargo.toml`.
2. **Drive model:** from the drive label, or from `freemkv info`.
3. **Connection type:** USB (with bridge chipset if known) or direct SATA.
4. **Operating system and kernel:** `uname -a`.
5. **Kernel messages during the failure:** `dmesg | tail -50` immediately after the crash.
6. **SCSI device:** which `/dev/sg*` the drive was on, and whether it changed after the failure.
7. **The disc:** title, format (BD/DVD/UHD), condition.
Include all of the above in bug reports. SCSI transport errors that resolve with the `US_FL_IGNORE_RESIDUE` quirk or by switching to direct SATA are bridge firmware bugs, not freemkv bugs.

87
benches/sgio_read.rs
View file

@ -1,87 +0,0 @@
// Mimics ISO dump exactly — read + write + progress
use libfreemkv::Drive;
use std::io::Write;
use std::path::Path;
use std::time::Instant;
fn main() {
let device = std::env::args()
.skip(1)
.find(|a| !a.starts_with('-'))
.unwrap_or_else(|| match libfreemkv::find_drive() {
Some(d) => d.device_path().to_string(),
None => {
eprintln!("No drives found");
std::process::exit(1);
}
});
let mut drive = Drive::open(Path::new(&device)).unwrap_or_else(|e| {
eprintln!("Cannot open {}: {}", device, e);
std::process::exit(1);
});
eprintln!("wait_ready...");
let _ = drive.wait_ready();
eprintln!("read_capacity...");
let cap = drive.read_capacity().unwrap();
eprintln!("capacity: {} sectors", cap);
let batch = libfreemkv::disc::detect_max_batch_sectors(drive.device_path());
let mut buf = vec![0u8; batch as usize * 2048];
// Open /dev/null writer like ISO dump does
let file = std::fs::File::create("/dev/null").unwrap();
let mut writer = std::io::BufWriter::with_capacity(4 * 1024 * 1024, file);
eprintln!(
"Reading 1000 batches ({:.1} MB) with write + progress...",
1000.0 * batch as f64 * 2048.0 / 1_048_576.0
);
let start = Instant::now();
let mut ok = 0u32;
let mut fail = 0u32;
let mut bytes: u64 = 0;
// Recovery flag: true matches pre-0.11.13 bench behavior — full SCSI
// ECC retry loop on errors (slower, what the rip path used before the
// adaptive batch sizer landed). Flip to `false` for the fast-fail path
// that current rips use; benches are configurable via this constant.
const READ_WITH_RECOVERY: bool = true;
for i in 0..1000u32 {
let lba = i * batch as u32;
match drive.read(lba, batch, &mut buf, READ_WITH_RECOVERY) {
Ok(_) => {
writer.write_all(&buf).unwrap();
ok += 1;
}
Err(e) => {
fail += 1;
if fail <= 5 {
eprintln!(" FAIL LBA {}: {}", lba, e);
}
buf.fill(0);
writer.write_all(&buf).unwrap();
}
}
bytes += buf.len() as u64;
if i % 50 == 0 && i > 0 {
let elapsed = start.elapsed().as_secs_f64();
let mb = bytes as f64 / 1_048_576.0;
eprint!("\r {:.1} MB | {:.1} MB/s ", mb, mb / elapsed);
}
}
let elapsed = start.elapsed().as_secs_f64();
let mb = ok as f64 * batch as f64 * 2048.0 / 1_048_576.0;
eprintln!(
"\n{} ok, {} fail, {:.1} MB in {:.1}s = {:.1} MB/s",
ok,
fail,
mb,
elapsed,
mb / elapsed
);
}

36
build.rs
View file

@ -1,36 +0,0 @@
fn main() {
let target = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
if target == "macos" {
println!("cargo:rustc-link-lib=framework=IOKit");
println!("cargo:rustc-link-lib=framework=CoreFoundation");
let out_dir = std::env::var("OUT_DIR").unwrap();
let obj = format!("{out_dir}/macos_shim.o");
let lib = format!("{out_dir}/libmacos_scsi.a");
std::process::Command::new("cc")
.args([
"-c",
"src/scsi/macos_shim.c",
"-o",
&obj,
"-framework",
"IOKit",
"-framework",
"CoreFoundation",
"-Wall",
"-O2",
])
.status()
.expect("failed to compile macos_shim.c");
std::process::Command::new("ar")
.args(["rcs", &lib, &obj])
.status()
.expect("failed to create static lib");
println!("cargo:rustc-link-search=native={out_dir}");
println!("cargo:rustc-link-lib=static=macos_scsi");
println!("cargo:rerun-if-changed=src/scsi/macos_shim.c");
}
}

34
docs/README.md
View file

@ -1,34 +0,0 @@
# libfreemkv Documentation
Technical documentation for [libfreemkv](https://github.com/freemkv/libfreemkv), the open source optical drive library.
## Start Here
**[Disc to Rip: End-to-End Flow](disc-to-rip.md)** — How the library goes from a disc in the drive to decrypted content. Read this first.
## Reference
| Document | What it covers |
|----------|---------------|
| [Architecture](architecture.md) | Module map, design principles, error codes, platform support |
| [Drive Access](drive-access.md) | Drive, SCSI transport, profiles, unlock, why raw mode is needed |
| [Rip Recovery](rip-recovery.md) | Three-layer recovery model: Disc::patch, single-shot Drive::read, DiscStream batch halving |
| [AACS Encryption](aacs.md) | Key resolution (4 paths), content decryption, bus encryption, SCSI handshake |
| [UDF Filesystem](udf.md) | UDF 2.50 with metadata partitions, pointer chain, how files are read from disc |
| [MPLS Playlists](mpls.md) | Playlist format, play items, STN stream table, coding types |
| [CLPI Clip Info](clpi.md) | EP map (coarse + fine entries), timestamp-to-sector mapping, extent calculation |
| [API Design](api-design.md) | Stream API design, PES pipeline, input/output resolution |
## Reading Order
If you want to understand the whole library:
1. **[Disc to Rip](disc-to-rip.md)** — the big picture
2. **[Architecture](architecture.md)** — how modules fit together
3. **[Drive Access](drive-access.md)** — how we talk to hardware
4. **[UDF](udf.md)** → **[MPLS](mpls.md)** → **[CLPI](clpi.md)** — how disc content is structured
5. **[AACS](aacs.md)** — how encryption works and how we break it
## API Documentation
Generated API docs are on [docs.rs/libfreemkv](https://docs.rs/libfreemkv).

322
docs/aacs.md
View file

@ -1,322 +0,0 @@
# AACS Encryption Support
## Overview
AACS (Advanced Access Content System) is the encryption layer used by Blu-ray and UHD 4K discs to protect content. libfreemkv implements AACS decryption to enable transparent disc access.
There are two major versions:
- **AACS 1.0** -- Used by standard Blu-ray discs. Relies on a custom 160-bit elliptic curve for bus authentication and AES-128 for content encryption. Processing keys and device keys can derive the media key from the disc's Media Key Block (MKB).
- **AACS 2.0** -- Used by UHD 4K Blu-ray discs. Adds a per-sector bus encryption layer (read_data_key) on top of the standard content encryption. Uses P-256/SHA-256 for its native handshake, though drives accept AACS 1.0 host certificates for backward compatibility.
Both versions use AES-128-CBC for content decryption with a fixed initialization vector. The fundamental key hierarchy is the same: a Volume Unique Key (VUK) decrypts per-title unit keys, which in turn decrypt the content stream.
## Architecture
AACS support is split across two modules:
### `aacs.rs` -- Keys and Decryption
Handles everything related to key resolution and content decryption:
- KEYDB.cfg parsing (device keys, processing keys, host certificates, per-disc entries)
- Disc hash computation (SHA-1 of `Unit_Key_RO.inf`)
- VUK resolution chain (4 paths, described below)
- MKB record parsing and media key derivation
- Subset-difference tree traversal (AACS-G3 key derivation)
- Unit_Key_RO.inf parsing and unit key decryption
- Content Certificate parsing (AACS version detection)
- Aligned unit decryption (AES-128-CBC)
- Bus decryption (AACS 2.0 read_data_key layer)
### `aacs_handshake.rs` -- SCSI Authentication
Handles the drive-level SCSI authentication protocol:
- ECDH key agreement on the AACS 160-bit curve
- ECDSA signing and verification
- Bus key derivation
- AGID management (allocate/invalidate)
- Volume ID retrieval (encrypted with bus key, verified by AES-CMAC)
- Read Data Key retrieval (for AACS 2.0 bus decryption)
- AACS LA public key certificate verification
## Key Resolution Chain
When a disc is scanned, `resolve_keys()` attempts four paths in priority order. The first path that succeeds is used.
### Path 1: KEYDB VUK Lookup (fastest)
```
Unit_Key_RO.inf --> SHA-1 --> disc_hash --> KEYDB lookup --> VUK
```
The disc hash is computed as the SHA-1 digest of the raw `Unit_Key_RO.inf` file from the disc's `/AACS/` directory. This hash is used as the lookup key in `KEYDB.cfg`. If a matching entry contains a VUK (`V` field), it is used directly.
This is the fast path and resolves the vast majority of discs in a well-maintained KEYDB.
### Path 2: KEYDB Media Key + Volume ID
```
KEYDB media_key + Volume ID (from SCSI handshake) --> VUK derivation
```
If the disc hash is not in the KEYDB but a KEYDB entry has a matching Volume ID (`I` field) and a media key (`M` field), the VUK is derived:
```
VUK = AES-128-ECB-DECRYPT(media_key, volume_id) XOR volume_id
```
Requires a successful SCSI handshake to obtain the Volume ID.
### Path 3: MKB + Processing Keys
```
MKB (from disc) + processing_keys (from KEYDB) --> media_key --> VUK
```
Processing keys are pre-computed keys that work against specific MKB versions. For each processing key, the library:
1. Parses the MKB to extract the Verify Media Key Record (`mk_dv`), subset-difference index, and conditional values (cvalues).
2. Tries each processing key against each UV/cvalue pair: `mk = AES-DEC(pk, cvalue) XOR cvalue`.
3. Validates the derived media key: `AES-ECB(mk, mk_dv)` must produce 12 leading zero bytes.
4. Derives VUK from the validated media key and Volume ID.
### Path 4: MKB + Device Keys (Subset-Difference Tree)
```
MKB + device_keys --> subset-difference tree traversal --> processing_key --> media_key --> VUK
```
The most complex path. Each device key has an associated node number, UV value, and mask parameters that position it in the AACS subset-difference tree. The library:
1. Finds the subset-difference entry in the MKB that applies to the device key's node.
2. Traverses the tree using AACS-G3 key derivation: `aesg3(key, inc) = AES-DEC(key, seed) XOR seed`, where `seed[15]` is incremented by `inc`. Each tree node produces a left child (inc=0), a processing key (inc=1), and a right child (inc=2).
3. At each level, selects left or right based on the UV bit at the current position.
4. The resulting processing key is validated against the MKB cvalue to derive the media key.
5. VUK is derived from the media key and Volume ID.
## Content Decryption
### Aligned Units
AACS encrypts content in aligned units of 6144 bytes (3 sectors of 2048 bytes each). The encryption flag is signaled by the copy_permission_indicator bits in byte 0 of the unit (`unit[0] & 0xC0 != 0`).
### Per-Unit Key Derivation
Each aligned unit has its own decryption key derived from the CPS unit key:
1. **Derive**: AES-128-ECB encrypt the first 16 bytes of the unit (plaintext TP_extra_header) with the unit key.
2. **XOR**: XOR the encrypted result with the original 16 bytes to produce the per-unit decryption key.
3. **Decrypt**: AES-128-CBC decrypt bytes 16 through 6143 using the per-unit key and the fixed AACS IV.
4. **Clear flag**: Clear the encryption indicator bits (`unit[0] &= !0xC0`).
### Fixed IV
All AES-CBC operations in AACS use the same fixed initialization vector, defined in the AACS specification.
### Verification
After decryption, the library verifies correctness by checking for MPEG-TS sync bytes (0x47) at the expected 192-byte packet boundaries within the unit. Blu-ray transport stream packets are 192 bytes: 4-byte TP_extra_header followed by a 188-byte TS packet.
## Bus Encryption
### AACS 1.0
Standard Blu-ray discs do not use bus encryption. Content is read directly from the disc and decrypted using the unit key.
### AACS 2.0
UHD 4K discs add a per-sector bus encryption layer. The drive encrypts data as it is read from the disc, and the host must decrypt it before applying AACS content decryption.
Bus encryption uses a **read_data_key** obtained during the SCSI handshake. For each 2048-byte sector within an aligned unit, bytes 16 through 2047 are AES-128-CBC encrypted with the read_data_key and the fixed AACS IV. The first 16 bytes of each sector remain plaintext.
The full decryption pipeline for AACS 2.0:
1. **Bus decrypt**: For each sector, AES-128-CBC decrypt bytes 16..2047 with the read_data_key.
2. **Content decrypt**: Standard per-unit key derivation and AES-128-CBC decryption as described above.
## SCSI Handshake
The AACS SCSI authentication handshake establishes a shared bus key between host and drive, then uses it to securely transfer the Volume ID and read data keys.
### Protocol Flow
1. **Invalidate AGIDs**: Send REPORT KEY with format 0x3F for AGIDs 0-3 to clear stale sessions.
2. **Allocate AGID**: REPORT KEY format 0x00 returns a fresh Authentication Grant ID.
3. **Send host credentials**: SEND KEY format 0x01 transmits the host nonce (20 random bytes) and host certificate (92 bytes).
4. **Receive drive credentials**: REPORT KEY format 0x01 returns the drive nonce and drive certificate.
5. **Receive drive key**: REPORT KEY format 0x02 returns the drive's ephemeral EC key point and ECDSA signature over `host_nonce || drive_key_point`.
6. **Verify drive key**: The signature is verified against the drive's public key (extracted from its certificate). AACS 1.0 certificates are verified against the AACS LA public key.
7. **Send host key**: The host generates an ephemeral key pair, signs `drive_nonce || host_key_point` with the host private key, and sends via SEND KEY format 0x02.
8. **Compute bus key**: ECDH shared secret = `host_private_key * drive_key_point`. The bus key is the low 128 bits of the shared point's x-coordinate.
### Post-Authentication Reads
- **Volume ID**: REPORT DISC STRUCTURE format 0x80. Returns 16-byte VID encrypted with the bus key, plus an AES-CMAC MAC for integrity verification.
- **Read Data Keys**: REPORT DISC STRUCTURE format 0x84. Returns the read_data_key and write_data_key, each AES-ECB encrypted with the bus key.
### Elliptic Curve
AACS 1.0 uses a custom 160-bit Weierstrass curve (`y^2 = x^3 + ax + b mod p`) with 20-byte field elements. The library implements full EC arithmetic: point addition, doubling, scalar multiplication, modular inverse, ECDSA sign/verify, and ECDH key agreement.
## AACS 2.0 Status
AACS 2.0 discs are detected via the Content Certificate file (`Content000.cer` or `Content001.cer`). A certificate type byte of 0x01 indicates AACS 2.0.
AACS 2.0 drives are identified by their drive certificate type (0x11). These drives natively use P-256/SHA-256, but accept AACS 1.0 host certificates for backward compatibility.
Current implementation status:
- AACS 2.0 detection: **implemented** (Content Certificate parsing, drive cert type check)
- AACS 1.0 handshake with AACS 2.0 drives: **implemented** (backward compatibility mode)
- Full P-256 AACS 2.0 handshake: **not yet implemented** (prepared but rarely needed since drives accept AACS 1.0 host certs)
- Bus decryption with read_data_key: **implemented**
- Content decryption: **implemented** (same as AACS 1.0)
In practice, AACS 2.0 UHD discs work through the backward-compatible AACS 1.0 handshake path, with the addition of read_data_key bus decryption.
## API Usage
AACS decryption is transparent to the application. The `Disc::scan()` method handles everything automatically:
```rust
use libfreemkv::{Drive, Disc};
use libfreemkv::disc::ScanOptions;
use std::path::Path;
let mut drive = Drive::open(Path::new("/dev/sg4")).unwrap();
drive.wait_ready().unwrap();
drive.init().unwrap();
let disc = Disc::scan(&mut drive, &ScanOptions::default()).unwrap();
// Check encryption state
if disc.encrypted {
if let Some(ref aacs) = disc.aacs {
println!("AACS {}.0", aacs.version);
println!("Key source: {}", aacs.key_source.name());
println!("Disc hash: {}", aacs.disc_hash);
if let Some(mkb_ver) = aacs.mkb_version {
println!("MKB version: {}", mkb_ver);
}
} else {
println!("Encrypted but keys not available");
}
}
// Read content -- decryption is automatic
let mut reader = disc.open_title(&mut session, 0).unwrap();
while let Some(unit) = reader.read_unit().unwrap() {
// unit is 6144 bytes of decrypted content
}
```
The application never touches keys, never calls decryption functions, and never manages handshakes. All of that is internal to `Disc::scan()` and `ContentReader::read_unit()`.
### KEYDB Location
`ScanOptions` controls where the KEYDB is loaded from. If no explicit path is set, the library checks:
1. `~/.config/aacs/KEYDB.cfg`
2. `/etc/aacs/KEYDB.cfg`
To specify an explicit path:
```rust
let opts = ScanOptions::with_keydb("/path/to/KEYDB.cfg");
let disc = Disc::scan(&mut session, &opts).unwrap();
```
### AacsState
After a successful scan, `disc.aacs` contains an `AacsState` with:
| Field | Type | Description |
|-------|------|-------------|
| `version` | `u8` | AACS version (1 or 2) |
| `bus_encryption` | `bool` | Whether bus encryption is active |
| `mkb_version` | `Option<u32>` | MKB version from disc |
| `disc_hash` | `String` | SHA-1 of Unit_Key_RO.inf (hex with 0x prefix) |
| `key_source` | `KeySource` | How keys were resolved |
| `vuk` | `[u8; 16]` | Volume Unique Key |
| `unit_keys` | `Vec<(u32, [u8; 16])>` | Decrypted unit keys (CPS unit number, key) |
| `read_data_key` | `Option<[u8; 16]>` | AACS 2.0 bus decryption key |
| `volume_id` | `[u8; 16]` | Volume ID from SCSI handshake |
### KeySource
| Variant | Description |
|---------|-------------|
| `KeyDb` | VUK found directly in KEYDB by disc hash |
| `KeyDbDerived` | Media key + Volume ID from KEYDB, VUK derived |
| `ProcessingKey` | MKB + processing keys from KEYDB |
| `DeviceKey` | MKB + device keys, subset-difference tree traversal |
## KEYDB.cfg Format Reference
The KEYDB.cfg file contains all cryptographic material needed for AACS decryption. Lines starting with `;` or `#` are comments.
### Device Keys
```
| DK | DEVICE_KEY 0x<key> | DEVICE_NODE 0x<node> | KEY_UV 0x<uv> | KEY_U_MASK_SHIFT 0x<shift>
```
- `key`: 16-byte AES device key (hex)
- `node`: Device node number in the subset-difference tree (hex)
- `uv`: UV value for tree positioning (hex)
- `shift`: U mask shift value (hex)
### Processing Keys
```
| PK | 0x<key>
```
- `key`: 16-byte pre-computed processing key (hex)
### Host Certificate
```
| HC | HOST_PRIV_KEY 0x<privkey> | HOST_CERT 0x<cert>
```
- `privkey`: 20-byte ECDSA private key (hex)
- `cert`: 92-byte AACS host certificate (hex)
The host certificate is used for SCSI authentication. It contains the host's public key and is signed by the AACS Licensing Administrator.
### Disc Entries
```
0x<disc_hash> = <title> | D | <date> | M | 0x<media_key> | I | 0x<disc_id> | V | 0x<vuk> | U | <unit_keys>
```
- `disc_hash`: 20-byte SHA-1 of Unit_Key_RO.inf (hex)
- `title`: Human-readable disc title
- `D`: Date tag, followed by release/rip date
- `M`: Media key tag, followed by 16-byte media key (hex)
- `I`: Disc ID tag, followed by 16-byte Volume ID (hex)
- `V`: VUK tag, followed by 16-byte Volume Unique Key (hex)
- `U`: Unit keys tag, followed by space-separated `<unit_num>-0x<key>` pairs
All fields after the title are optional. A minimal entry needs only the disc hash and VUK:
```
0x<disc_hash> = <title> | V | 0x<vuk>
```
Inline comments are supported with `;`:
```
0x<disc_hash> = <title> | V | 0x<vuk> ; MKBv77
```

235
docs/api-design.md
View file

@ -1,235 +0,0 @@
# libfreemkv API Design
## Principles
1. Lib provides building blocks. App composes them.
2. No English text in lib. Error codes only. App handles i18n.
3. No display logic in lib. App decides what to show.
4. Streams are the pipeline. Each stage wraps the next.
5. Lib fires events. App listens.
## Core API
```rust
// Open drive — explicit steps, app prints between them
let mut drive = Drive::open(path)?;
drive.wait_ready()?;
drive.init()?;
drive.probe_disc()?;
// Scan disc
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Browse
disc.titles // Vec<DiscTitle>
disc.format // BD / UHD / DVD
disc.capacity_gb()
```
## PES Pipeline (primary API)
The PES pipeline is the main way to move content. All streams produce/consume
PES frames. The pipeline just reads frames and writes frames.
```rust
// URL-based — any source to any destination
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
}
output.finish()?;
```
The `FrameSource` and `FrameSink` traits — direction is type-checked, so
calling `read()` on a write-only sink (or `write()` on a read-only source)
is a compile error rather than a runtime fault:
```rust
pub trait FrameSource: Send {
fn read(&mut self) -> Result<Option<PesFrame>, Error>;
fn info(&self) -> &DiscTitle;
fn codec_private(&self, track: usize) -> Option<Vec<u8>> { None }
fn headers_ready(&self) -> bool { true }
}
pub trait FrameSink: Send {
fn write(&mut self, frame: &PesFrame) -> Result<(), Error>;
fn finish(self: Box<Self>) -> Result<(), Error>;
fn info(&self) -> &DiscTitle;
}
```
## Streams
All streams implement `FrameSource` (read) and/or `FrameSink` (write); the
directional split prevents runtime "wrong-direction" errors. URL-based
resolvers open any stream by string.
| Stream | Input | Output | URL | Transport |
|--------|-------|--------|-----|-----------|
| DiscStream | Yes | -- | `disc://` `disc:///dev/sg4` | Optical drive via SCSI |
| IsoStream | Yes | Yes | `iso://path.iso` | Blu-ray ISO image |
| MkvStream | Yes | Yes | `mkv://path` | Matroska container |
| M2tsStream | Yes | Yes | `m2ts://path` | BD-TS with FMKV metadata header |
| NetworkStream | Yes (listen) | Yes (connect) | `network://host:port` | TCP with FMKV metadata header |
| StdioStream | Yes (stdin) | Yes (stdout) | `stdio://` | Raw byte pipe |
| NullStream | -- | Yes | `null://` | Discard sink (byte counter) |
All URLs require a `scheme://path` format. Bare paths are rejected.
```rust
// PES pipeline (frame-level) — input() returns Box<dyn FrameSource>,
// output() returns Box<dyn FrameSink>.
let input = libfreemkv::input("disc:///dev/sg4", &opts)?; // DiscStream
let input = libfreemkv::input("iso://Dune.iso", &opts)?; // IsoStream
let output = libfreemkv::output("mkv://Dune.mkv", &title)?; // MkvOutputStream
let output = libfreemkv::output("m2ts://Dune.m2ts", &title)?; // M2tsOutputStream
let output = libfreemkv::output("network://10.0.0.1:9000", &title)?; // NetworkOutputStream
let output = libfreemkv::output("null://", &title)?; // NullOutputStream
```
### FMKV Metadata Header
M2tsStream and NetworkStream embed a JSON metadata header before the BD-TS data:
```
[8B magic "FMKV\0\0\0\0"][4B JSON length][JSON metadata][padding to 192B boundary][BD-TS data...]
```
The header carries title name, duration, codec_privates, and full stream layout
(PIDs, codecs, languages, labels). This allows the receiving end to set up
demuxing and track metadata without scanning the TS.
## Events
Lib fires events during operations. App provides a callback. No display, no text.
```rust
pub struct Event {
pub kind: EventKind,
}
pub enum EventKind {
// Init / scan
DriveOpened { device: String },
DriveReady,
InitComplete { success: bool },
ProbeComplete { success: bool },
ScanComplete { titles: usize },
// Read pipeline
BytesRead { bytes: u64, total: u64 },
ReadError { sector: u64, error: Error },
SpeedChange { speed_kbs: u16 },
ExtentStart { index: usize, start_sector: u64, sector_count: u64 },
SectorSkipped { sector: u64 },
BatchSizeChanged { new_size: u16, reason: BatchSizeReason },
Complete { bytes: u64, errors: u32 },
// Kept for forward-compat; not emitted in 0.13.6+
Retry { attempt: u32 },
SectorRecovered { sector: u64 },
}
```
Emission notes:
- `BytesRead { bytes, total }` is emitted from `DiscStream::fill_extents`
after each successful sector read. `bytes` is the cumulative running
total; `total` is the precomputed extent sum (0 if unknown).
- `SpeedChange` is emitted from the public `Drive::set_speed` API path.
It is no longer emitted from a recovery hot loop (recovery loop removed
in 0.13.6).
- `BatchSizeChanged` fires from the `DiscStream` adaptive sizer on shrink
(read failed at a larger size) and on probe-up (clean-read streak hit
the threshold). Consumers use it to display a "recovering" state
distinct from "ripping normally".
- `Retry` and `SectorRecovered` are NOT emitted in 0.13.6+. They were
tied to the inline `Drive::read` recovery phases that were removed; the
variants are kept for forward compatibility so consumers' match arms
don't need conditional compilation.
Events report what happened. App decides what to do. GUI shows a dialog. CLI
prints a line. Server logs to file.
## File Layout
```
libfreemkv/src/
├── lib.rs Public exports
├── error.rs Error codes (no English)
├── event.rs Event types for callbacks
├── halt.rs Halt cancellation token (Arc<AtomicBool> wrapper)
├── io/ Pipeline + WritebackFile primitives
│ ├── mod.rs Re-exports WritebackFile, Pipeline, Sink, Flow
│ ├── pipeline.rs Generic Pipeline<I, R> + Sink trait
│ ├── writeback_file.rs WritebackFile (was crate::io::Writer)
│ └── writeback.rs sync_file_range pipeline
├── drive/ Drive (open, init, single-shot read)
│ ├── mod.rs Drive struct, init, read (single-shot), reset, eject
│ ├── capture.rs Drive profile capture for contribution
│ ├── linux.rs Linux drive discovery
│ ├── macos.rs macOS drive discovery
│ └── windows.rs Windows drive discovery
├── disc/ Disc (scan, titles, AACS setup, sweep, patch)
│ ├── mod.rs Disc struct, scan, titles, formats
│ ├── sweep.rs Disc::sweep (Pass 1 forward sweep)
│ ├── patch.rs Disc::patch (Pass N retry over mapfile)
│ ├── mapfile.rs ddrescue-format mapfile
│ └── read_error.rs ReadCtx / ReadAction state machine
├── scsi/ SCSI transport (Linux SG_IO, macOS IOKit, Windows SPTI)
├── platform/ Drive unlock (MT1959 A/B)
├── aacs/ AACS decryption (handshake, keys, keydb, decrypt)
├── css/ DVD CSS cipher
├── decrypt.rs Unified decrypt dispatcher (AACS/CSS/None)
├── pes.rs PES frame types, FrameSource / FrameSink traits
├── sector/ Sector I/O (was sector.rs in 0.17)
│ ├── mod.rs SectorSource, SectorSink traits
│ ├── file.rs FileSectorSource, FileSectorSink (ISO-backed)
│ └── decrypting.rs DecryptingSectorSource decorator
├── udf.rs UDF 2.50 filesystem parser
├── mpls.rs MPLS playlist parser
├── clpi.rs CLPI clip info parser
├── ifo.rs DVD IFO parser
├── labels/ BD-J label extraction (5 format parsers)
├── keydb.rs KEYDB download, parse, save
├── identity.rs DriveId from INQUIRY
├── profile.rs Bundled drive profiles
├── speed.rs DriveSpeed enum
├── mux/
│ ├── mod.rs Public mux exports
│ ├── resolve.rs URL parser + input/output (Box<dyn FrameSource/Sink>)
│ ├── meta.rs FMKV header format
│ ├── disc.rs DiscStream (optical drive → PES)
│ ├── iso.rs IsoStream (ISO image read)
│ ├── isowriter.rs ISO image writer (UDF, AVDP, multi-extent)
│ ├── mkvstream.rs MkvStream (bidirectional Matroska)
│ ├── mkvout.rs MkvOutputStream (PES → MKV)
│ ├── m2ts.rs M2tsStream (BD-TS)
│ ├── pesout.rs PES output streams (M2ts, Network, Stdio, Null)
│ ├── network.rs NetworkStream (TCP + FMKV header)
│ ├── stdio.rs StdioStream (stdin/stdout pipe)
│ ├── null.rs NullStream (discard + byte counter)
│ ├── lookahead.rs LookaheadBuffer (codec header scanning)
│ ├── ts.rs BD-TS demuxer + PAT/PMT scanner
│ ├── tsreader.rs TS reader utilities
│ ├── tsmux.rs TS muxer (PES → BD-TS packets)
│ ├── ps.rs MPEG-2 PS demuxer (DVD)
│ ├── ebml.rs EBML read/write primitives
│ ├── mkv.rs MKV muxer (tracks, clusters, cues)
│ └── codec/ Frame parsers (H.264, HEVC, MPEG-2, VC-1, AC3, EAC3, DTS, TrueHD, LPCM, PGS)
└── ...
freemkv/src/
├── main.rs CLI dispatcher (URL routing)
├── pipe.rs PES pipeline — source → dest copy
├── disc_info.rs Disc/file info display
├── info.rs Drive info + profile submission
├── strings.rs i18n string table
├── output.rs Verbosity-filtered output
└── build.rs Bundled locale code generation
```

219
docs/architecture.md
View file

@ -1,219 +0,0 @@
# libfreemkv Architecture
Open source optical drive access library for 4K UHD Blu-ray, Blu-ray, and DVD.
Rust library with no external dependencies at runtime -- profiles are bundled,
AACS keys are derived internally, and all SCSI communication is handled in-process.
**Repository:** <https://github.com/freemkv/libfreemkv>
**License:** AGPL-3.0-only
---
## Design Principles
1. **CLI is dumb.** All drive communication, disc parsing, AACS decryption, and
format handling live in the library. CLI binaries are thin wrappers that call
`Drive::open()` and `Disc::scan()`.
2. **No external files.** Bundled drive profiles are compiled into the binary via
`include_str!`. No configuration directory, no runtime file lookups for drive
support.
3. **Transparent AACS.** The `ContentReader` decrypts on the fly when keys are
available. Callers read cleartext sectors without knowing whether the disc
was encrypted.
4. **Structured errors, no English.** Every error has a numeric code (E1000-E8000).
The library never formats user-facing messages -- applications do that.
5. **Library-agnostic.** No concept of "supported" vs "unsupported" drives at a
policy level. If a profile exists, the library uses it.
6. **Streams are dumb pipes.** Streams read/write PES frames. They don't know
about encryption, transport format, or source type. Decrypt is a stream-internal
concern; the pipeline just moves frames.
---
## Module Map
```
libfreemkv (lib.rs)
│
├── Drive Access
│ ├── drive Drive — open, identify, init, unlock, single-shot read
│ ├── scsi ScsiTransport trait + platform backends (sg async, IOKit, SPTI)
│ ├── platform/ Platform trait — per-chipset command handlers
│ │ └── mt1959 MediaTek MT1959 driver (LG, ASUS, HP)
│ ├── profile DriveProfile loading, matching, bundled JSON
│ ├── identity DriveId from INQUIRY + GET_CONFIG 010C
│ ├── speed DriveSpeed enum, SET CD SPEED CDB builder
│ └── event Event system for drive status callbacks
│
├── Disc Scanning
│ ├── disc Disc::scan() — titles, streams, extents, AACS setup
│ ├── udf UDF 2.50 filesystem reader (metadata partitions)
│ ├── mpls MPLS playlist parser — clips, streams, STN table
│ ├── clpi CLPI clip info parser — EP map, sector extents
│ ├── ifo DVD IFO parser — title sets, PGC chains, cell addresses
│ └── labels/ BD-J label extraction (5 formats: Paramount, Criterion, Pixelogic, CTRM, Deluxe)
│
├── Encryption
│ ├── aacs/ AACS handshake, KEYDB, VUK lookup, MKB, unit decryption
│ ├── css DVD CSS cipher — table-driven, no external keys needed
│ └── decrypt decrypt_sectors() — unified AACS/CSS/None dispatcher
│
├── Streaming
│ ├── mux/ Stream implementations (Disc, ISO, MKV, M2TS, Network, Stdio, Null)
│ ├── pes PES frame types; FrameSource / FrameSink direction-typed traits
│ └── sector/ SectorSource / SectorSink traits, FileSector{Source,Sink}, DecryptingSectorSource
│
├── I/O Primitives
│ ├── halt Halt cancellation token (one Arc<AtomicBool>, cloneable)
│ └── io/ Pipeline<I, R> + Sink trait + WritebackFile (bounded-cache writer)
│
├── Support
│ ├── keydb KEYDB.cfg download, parse, verify, save
│ ├── error Error enum with numeric codes E1000-E8000
│ └── profile Bundled drive profiles
│
└── lib.rs Public API re-exports
```
---
## Drive Access Flow
```
Drive::open(Path::new("/dev/sg4"))
│
├─ scsi::open() Open /dev/sg4 (async write/poll/read)
├─ DriveId::from_drive() INQUIRY + GET_CONFIG 010C
├─ profile::find_by_drive_id() Match against bundled profiles
├─ Platform::new() Instantiate chipset driver (Mt1959)
└─ Drive ready for init/unlock/read
```
After open:
- `init()` -- unlock + firmware upload + speed calibration
- `probe_disc()` -- probe disc surface for optimal speeds
- `read(lba, count, buf, recovery)` -- single-shot read; `recovery` only selects the per-CDB timeout (1.5 s vs. 30 s)
- `wait_ready()` -- wait for disc insertion
- `eject()` -- eject tray
Recovery is layered above `Drive::read`, not inside it. Layer 1
(`Disc::patch`) handles bad-range retry by replaying the ddrescue mapfile.
Layer 3 (`DiscStream::fill_extents` adaptive batch sizer) handles in-loop
request-size adaptation. Inline recovery (gentle retry → SCSI reset → retry)
was removed in 0.13.6 — see [`rip-recovery.md`](rip-recovery.md) and
the stop-wedge postmortem (2026-04-25).
---
## Disc Scanning Flow
```
Disc::scan(&mut drive, &ScanOptions)
│
├─ READ CAPACITY Get disc size in sectors
├─ udf::read_filesystem() Parse UDF 2.50 (AVDP → VDS → metadata → FSD → root)
├─ For each BDMV/PLAYLIST/*.mpls:
│ ├─ mpls::parse() Extract play items, STN streams
│ └─ For each clip:
│ └─ clpi::parse() EP map → sector extents for the clip's time range
├─ labels::detect() Parse BD-J JARs for stream labels
├─ Detect AACS Check for /AACS directory on disc
└─ Disc::setup_aacs() Handshake + KEYDB → VUK → unit keys (if encrypted)
```
For DVD:
```
Disc::scan_dvd(&mut drive, &ScanOptions)
│
├─ ifo::parse() Parse VIDEO_TS.IFO — title sets, PGC chains
├─ CSS detection Check disc structure flag
└─ CSS key cracking Table-driven, no KEYDB needed
```
The result is a `Disc` with:
- `titles: Vec<DiscTitle>` -- sorted by duration, each with streams, sector extents, codec_privates
- `decrypt_keys()` -- DecryptKeys for content decryption
- `encrypted: bool` -- whether the disc uses AACS/CSS
---
## AACS Decryption
Four key resolution paths, tried in order:
| Path | Method | Speed |
|------|--------|-------|
| 1 | VUK lookup by disc hash in KEYDB.cfg | Instant |
| 2 | Media Key + Volume ID from KEYDB → derive VUK | Fast |
| 3 | Processing Keys + MKB → Media Key → VUK | Medium |
| 4 | Device Keys + MKB subset-difference tree → VUK | Slow |
The AACS handshake (`aacs/handshake`) performs ECDH key agreement over the
AACS 1.0 160-bit elliptic curve to obtain:
- **Volume ID** -- needed for VUK derivation (paths 2-4)
- **Read Data Key** -- needed for AACS 2.0 (UHD) bus decryption
Content decryption uses AES-128-CBC on 6144-byte aligned units. The
`ContentReader` handles this transparently. Streams that read sectors
(DiscStream, IsoStream) decrypt internally — the pipeline sees clean bytes.
---
## Error Codes
All errors carry a numeric code for programmatic handling. No user-facing text
is baked into the library.
| Range | Category | Examples |
|-------|----------|----------|
| E1xxx | Device errors | `DeviceNotFound`, `DevicePermission` |
| E2xxx | Profile errors | `UnsupportedDrive`, `ProfileNotFound`, `ProfileParse` |
| E3xxx | Unlock errors | `UnlockFailed`, `SignatureMismatch`, `NotUnlocked`, `NotCalibrated` |
| E4xxx | SCSI errors | `ScsiError`, `ScsiTimeout` |
| E5xxx | I/O errors | `IoError` (wraps `std::io::Error`) |
| E6xxx | Disc format errors | `DiscError` (UDF, MPLS, CLPI parse failures) |
| E7xxx | AACS errors | `AacsError` (key resolution, handshake, decryption) |
| E8xxx | KEYDB errors | `KeydbError` (download, parse, save) |
---
## Platform Support
| Platform | Transport | Status |
|----------|-----------|--------|
| Linux | async sg write/poll/read on `/dev/sg*` | Supported |
| macOS | IOKit SCSITask | Supported |
| Windows | SPTI (`IOCTL_SCSI_PASS_THROUGH_DIRECT`) | Supported |
The `ScsiTransport` trait abstracts the platform. Adding a new platform requires
implementing `execute()` for that OS and wiring it into `scsi::open()`.
---
## Chipset Support
| Chipset | Drives | Status |
|---------|--------|--------|
| MediaTek MT1959 | LG, ASUS, HP | Supported (bundled profiles) |
| Renesas RS8xxx/RS9xxx | Pioneer, some HL-DT-ST | Planned |
The `Platform` trait abstracts chipset-specific commands. Each chipset implements
handlers (unlock, config, register, calibrate, keepalive, status, probe,
read_sectors, timing). All handlers are accessed via SCSI READ BUFFER with
chipset-specific mode and buffer ID bytes.
---
## Build
```
cargo build --release
```
Produces a Rust library crate. The `libc` dependency is unix-only (gated).
All three platforms build and pass CI.

197
docs/clpi.md
View file

@ -1,197 +0,0 @@
# CLPI Clip Information Format
## What is CLPI?
CLPI (Clip Information) files describe the structure of individual M2TS transport streams on a Blu-ray disc. Each `.clpi` file in `BDMV/CLIPINF/` corresponds to one `.m2ts` file in `BDMV/STREAM/` with the same numeric name (e.g. `00001.clpi` describes `00001.m2ts`).
The most important data in a CLPI file is the **EP (Entry Point) map**, which provides timestamp-to-sector mapping. This is essential for seeking and for extracting the specific sector ranges that correspond to a playlist's in/out time window.
## File Structure
CLPI files use big-endian byte order. The header:
```
Offset Size Field
------ ---- -----
0 4 Magic: "HDMV"
4 4 Version: "0200" (BD) or "0300" (UHD BD)
8 4 SequenceInfo start offset
12 4 ProgramInfo start offset
16 4 CPI (Characteristic Point Information) start offset
20 4 ClipMark start offset
24 4 ExtensionData start offset
```
### ClipInfo Section (offset 40)
Contains basic clip metadata. The source packet count (total number of 192-byte source packets in the M2TS file) is at offset 56:
```
Offset Size Field
------ ---- -----
40 4 ClipInfo length
44 2 Reserved
46 1 Stream type
47 1 Application type
48 4 Reserved
52 4 TS recording rate
56 4 Source packet count
```
The source packet count multiplied by 192 gives the total byte size of the M2TS file.
## EP Map
The EP map lives inside the CPI section and provides random access into the transport stream. It maps PTS timestamps to Source Packet Numbers (SPN), enabling precise seeking without scanning the stream.
### CPI Section Layout
```
Offset Size Field
------ ---- -----
0 4 CPI length
4 2 Reserved / CPI type
6 ... EP map
```
### EP Map Header
```
Offset Size Field
------ ---- -----
0 1 Reserved
1 1 Number of stream PID entries
2 ... Stream PID entry headers (one per stream)
```
Each stream PID entry header (14 bytes):
```
Offset Size Field
------ ---- -----
0 2 Stream PID
2 2 Reserved + EP stream type
4 2 Number of coarse entries
6 4 Number of fine entries (note: 32-bit, can be large)
10 4 EP map start offset (relative to EP map start)
```
libfreemkv parses only the first stream (primary video), which is sufficient for sector-level seeking.
### Two-Level Index
The EP map uses a two-level structure to compress what would otherwise be a very large lookup table:
- **Coarse entries**: low-resolution index covering large time/sector ranges
- **Fine entries**: high-resolution entries within each coarse range
Each coarse entry points to a range of fine entries via `ref_to_fine_id`.
### Coarse Entries (8 bytes each)
Located immediately after the fine table start offset (4 bytes) in the per-stream EP map:
```
Bits Field
------ -----
[31:14] ref_to_fine_id (18 bits) -- index of first fine entry in this range
[13:0] pts_coarse (14 bits) -- upper bits of PTS
```
Second dword:
```
Bits Field
------ -----
[31:0] spn_coarse (32 bits) -- upper bits of SPN
```
### Fine Entries (4 bytes each)
Located at the fine table start offset within the per-stream EP map:
```
Bits Field
------ -----
[31] is_angle_change_point (1 bit)
[30:28] I_end_position_offset (3 bits)
[27:17] pts_fine (11 bits) -- lower bits of PTS
[16:0] spn_fine (17 bits) -- lower bits of SPN
```
## Reconstructing Full PTS and SPN
The full timestamp and packet number are assembled by combining the coarse and fine components:
### Full PTS
```
full_pts = (pts_coarse << 19) + (pts_fine << 8)
```
The coarse component provides the upper bits, shifted left by 19. The fine component provides mid-range bits, shifted left by 8. The resulting PTS is in 45kHz ticks, matching the MPLS timestamp format.
### Full SPN
```
full_spn = (spn_coarse & 0xFFFE0000) + spn_fine
```
The coarse SPN provides the upper 15 bits (masked to clear the lower 17). The fine SPN provides the lower 17 bits. The result is a Source Packet Number -- each source packet is 192 bytes in the M2TS file.
### Resolved EP Map
The `resolved_ep_map()` method iterates through all coarse entries and their associated fine entries to produce a flat list of `(PTS, SPN)` pairs. For each coarse entry at index `ci`:
- Fine entries range from `coarse[ci].ref_to_fine_id` to `coarse[ci+1].ref_to_fine_id` (or end of fine table for the last coarse entry).
- Each fine entry is combined with its parent coarse entry to produce one full `(PTS, SPN)` pair.
## Deriving Sector Extents for Ripping
Given a playlist's in_time and out_time (from MPLS), the CLPI EP map provides the sector ranges to read from disc. The `get_extents()` method does this in three steps:
### Step 1: PTS to SPN
Binary search the resolved EP map for the in_time and out_time:
- **start_spn**: the SPN at or before in_time (seek backward to the nearest I-frame)
- **end_spn**: the SPN at or after out_time (include the full GOP)
### Step 2: SPN to Byte Offset
Each Source Packet is 192 bytes (188 bytes MPEG-TS payload + 4 bytes M2TS header):
```
byte_offset = spn * 192
```
### Step 3: Byte Offset to Sector
M2TS files are stored contiguously on disc. Sectors are 2048 bytes:
```
start_sector = start_byte / 2048
end_sector = (end_byte + 2047) / 2048
sector_count = end_sector - start_sector
```
The resulting `Extent` contains `start_lba` (relative to the M2TS file's starting sector on disc) and `sector_count`. The caller adds the file's absolute starting LBA from UDF to get disc-absolute sector numbers.
### Alignment Note
The 192-byte source packet size and 2048-byte sector size share no common factor beyond 1. A single sector contains roughly 10.67 source packets. The conversion rounds start down and end up to ensure complete coverage.
## Putting It Together
The full ripping pipeline chains three parsers:
1. **MPLS** provides clip IDs and in/out timestamps.
2. **CLPI** converts those timestamps to SPN ranges, then to sector extents.
3. **UDF** provides the file's starting LBA on disc for absolute sector addressing.
The `Disc::scan()` method in `src/disc.rs` orchestrates this: for each play item in each playlist, it loads the corresponding CLPI, calls `get_extents()` with the play item's in/out times, and collects the resulting sector ranges into the title's extent list.
## References
- BD-ROM Part 3, Section 5.5: Clip Information file format
- https://github.com/lw/BluRay/wiki/CLPI

135
docs/disc-to-rip.md
View file

@ -1,135 +0,0 @@
# Disc to Rip: End-to-End Flow
How libfreemkv goes from a disc in the drive to decrypted content ready for backup.
This is the starting point for understanding the library.
## The Pipeline
```
Insert disc
│
▼
1. Open drive (drive/mod.rs)
│ INQUIRY → identify drive
│ Match bundled profile → chipset, unlock parameters
│
▼
2. Init drive (drive/mod.rs → platform/mt1959)
│ Firmware upload (if needed, 10s recovery wait)
│ Unlock → vendor-specific command activates raw read mode
│ Speed calibration → probe_disc()
│
▼
3. AACS handshake (aacs/handshake.rs) — optional
│ Allocate AGID
│ Exchange certificates + nonces (ECDH)
│ Derive bus key
│ Read Volume ID + read_data_key
│ (fails gracefully if drive doesn't support AACS for this disc)
│
▼
4. Read UDF filesystem (udf.rs)
│ Sector 256: AVDP → find Volume Descriptor Sequence
│ VDS: Partition Descriptor (physical start) + Logical Volume (metadata start)
│ Metadata partition → File Set Descriptor → Root directory
│ Walk directory tree: BDMV/, AACS/, CERTIFICATE/
│ → docs/udf.md
│
▼
5. Read AACS files from disc (aacs/mod.rs)
│ AACS/Unit_Key_RO.inf → SHA1 = disc hash
│ AACS/Content000.cer → AACS version (1.0 or 2.0), bus encryption flag
│ MKB via SCSI → for key derivation fallback
│
▼
6. Resolve encryption keys (decrypt.rs → resolve_encryption)
│ BD AACS:
│ Path 1: disc hash → KEYDB.cfg → VUK (fast, 99% of discs)
│ Path 2: KEYDB media key + Volume ID → VUK
│ Path 3: MKB + processing keys → media key → VUK
│ Path 4: MKB + device keys → subset-difference tree → VUK
│ VUK → decrypt unit keys from Unit_Key_RO.inf
│ DVD CSS:
│ Table-driven cipher — no KEYDB needed
│ → docs/aacs.md
│
▼
7. Parse playlists (mpls.rs) — BD/UHD only
│ BDMV/PLAYLIST/*.mpls → titles with play items
│ Each play item: clip ID, in/out timestamps
│ STN table: video, audio, subtitle streams with codec + language
│ → docs/mpls.md
│
▼
8. Parse clip info (clpi.rs) — BD/UHD only
│ BDMV/CLIPINF/*.clpi → EP map (timestamp → sector mapping)
│ Coarse + fine entries → full PTS and SPN
│ SPN → byte offset → sector extents for reading
│ → docs/clpi.md
│
▼
9. Parse BD-J labels (labels/) — optional
│ BDMV/JAR/*.jar → Java class constant pool strings
│ 5 format parsers: Paramount, Criterion, Pixelogic, CTRM, Deluxe
│ Audio track labels: "English Descriptive Audio", "French 5.1", etc.
│
▼
10. Stream content (mux/disc.rs → DiscStream)
│ Read sectors → decrypt → TS demux → PES frames
│ Or: read sectors → decrypt → raw bytes (for ISO output)
│ Drive::read() is single-shot. DiscStream::fill_extents adapts the
│ batch size on failure (halve / probe-up). Bad-range retry is layer
│ 1 above this — Disc::patch re-runs against the mapfile.
│
▼
PES frames → output stream (MKV, M2TS, network, etc.)
```
## API Summary
```rust
// Open + init drive
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
drive.wait_ready()?;
drive.init()?;
drive.probe_disc()?;
// Scan disc (UDF + playlists + AACS — all automatic)
let disc = Disc::scan(&mut drive, &ScanOptions::default())?;
// Stream pipeline — PES frames from any source to any output.
// 0.18: input() returns Box<dyn FrameSource>, output() returns Box<dyn FrameSink>;
// direction is type-checked, so calling .write() on an input is a compile error.
let opts = InputOptions::default();
let mut input = libfreemkv::input("disc:///dev/sg4", &opts)?;
let title = input.info().clone();
let mut output = libfreemkv::output("mkv://Movie.mkv", &title)?;
while let Ok(Some(frame)) = input.read() {
output.write(&frame)?;
}
output.finish()?;
```
## Module Reference
| Module | Doc | Purpose |
|--------|-----|---------|
| drive/ | [drive-access.md](drive-access.md) | Open, identify, init, unlock, single-shot read |
| scsi/ | [drive-access.md](drive-access.md) | Platform SCSI transport (Linux, macOS, Windows) |
| udf.rs | [udf.md](udf.md) | UDF 2.50 filesystem |
| mpls.rs | [mpls.md](mpls.md) | MPLS playlists + STN streams |
| clpi.rs | [clpi.md](clpi.md) | CLPI clip info + EP map |
| ifo.rs | -- | DVD IFO parser |
| aacs/ | [aacs.md](aacs.md) | Key resolution + content decrypt + bus handshake |
| css/ | -- | DVD CSS cipher |
| decrypt.rs | -- | Unified decrypt dispatcher (AACS/CSS/None) |
| disc/ | [rip-recovery.md](rip-recovery.md) | Disc::scan + Disc::sweep + Disc::patch + mapfile |
| labels/ | -- | BD-J stream labels (5 format parsers) |
| mux/ | -- | Stream implementations (7 stream types) |
| pes.rs | -- | PES frame types + FrameSource / FrameSink traits |
| sector/ | -- | SectorSource / SectorSink + DecryptingSectorSource decorator |
| io/ | -- | Pipeline<I, R> + Sink trait + WritebackFile |
| halt.rs | -- | Halt cancellation token |
| keydb.rs | -- | KEYDB download, parse, save |
| error.rs | -- | Error codes (E1xxx-E8xxx) |
| event.rs | -- | Drive event system |

268
docs/drive-access.md
View file

@ -1,268 +0,0 @@
# Drive Access and Unlock
Technical reference for how libfreemkv opens, identifies, unlocks, and reads
optical drives.
---
## Drive
`Drive` is the primary API. It owns the SCSI transport, the matched
drive profile, and the chipset-specific platform driver.
### Opening a Drive
```rust
let mut drive = Drive::open(Path::new("/dev/sg4"))?;
```
`open()` performs: open device → send INQUIRY → match profile → instantiate
platform driver. The drive is ready for `wait_ready()` and `init()`.
### Drive Operations
| Method | Description |
|--------|-------------|
| `wait_ready()` | Wait for disc insertion (30s timeout, TUR polling) |
| `init()` | Firmware upload + unlock + speed calibration |
| `probe_disc()` | Probe disc surface for optimal speeds |
| `read(lba, count, buf, recovery)` | Read sectors. Single-shot — no inline retries or reset. |
| `reset()` | Eject-cycle escape hatch. Caller-invoked only; not on the read path. |
| `lock_tray()` | Prevent tray ejection during rip |
| `unlock_tray()` | Allow tray ejection (also runs on Drop) |
| `eject()` | Eject disc tray |
| `drive_status()` | Query physical state (disc present, tray open, etc.) |
| `has_profile()` | Whether a bundled profile matched |
| `close()` | Consume Drive, cleanup (also runs via Drop) |
### init() Sequence
`init()` orchestrates the full drive unlock:
1. Platform driver `run_init()` — sends vendor-specific SCSI commands
2. If firmware upload needed: upload, wait 10s for drive reset, retry
3. Speed calibration after unlock
4. Max 3 attempts before giving up
### read() — single-shot
`Drive::read(lba, count, buf, recovery)` is the single read method. It issues
exactly one READ(10) CDB and returns the result. The `recovery` parameter only
selects the per-CDB timeout:
| `recovery` | Timeout | Used by |
|------------|----------|------------------------------------------|
| `false` | 1.5 s | `Disc::sweep` fast skip-forward pass, `DiscStream::fill_extents` |
| `true` | 30 s | `Disc::patch` retry pass over the mapfile |
On any SCSI failure or timeout, `read` returns `Err(DiscRead)` immediately.
There are no inline retries, no SCSI reset, no Phase 1/2/3 escalation.
Recovery is layered above `Drive::read`:
- **Layer 1 — `Disc::patch`** loops over the ddrescue mapfile and re-issues
`read(.., recovery=true)` against each non-`+` range.
- **Layer 3 — `DiscStream::fill_extents`** halves the request size on
failure, retries at the same LBA, and probes back up on a clean-read
streak.
Inline recovery (5× gentle retry → close + reset + reopen → 5× more) was
removed in 0.13.6. See the stop-wedge postmortem (2026-04-25)
for rationale: the inline reset wedged drive firmware on the LG BU40N (Initio
USB-SATA bridge) without ever recovering a sector. See
[`rip-recovery.md`](rip-recovery.md) for the full three-layer model.
---
## SCSI Transport
### Trait
```rust
pub trait ScsiTransport: Send {
fn execute(
&mut self,
cdb: &[u8],
direction: DataDirection,
data: &mut [u8],
timeout_ms: u32,
) -> Result<ScsiResult>;
}
```
All drive communication goes through this trait. The library never opens file
descriptors or calls ioctls outside of a `ScsiTransport` implementation.
### Platform Backends
| Platform | Implementation | Device |
|----------|---------------|--------|
| Linux | `SgIoTransport` — async `write`/`poll`/`read` on `/dev/sg*` | `/dev/sg*` |
| macOS | `MacScsiTransport` — IOKit SCSITask | IOKit service |
The Linux backend uses the sg driver's asynchronous interface: `write()` submits
the command, `poll()` waits with an enforceable wall-clock timeout, `read()`
retrieves the result. If `poll()` times out, the fd is abandoned (closed in a
background thread) and a fresh fd opened — the kernel's USB error recovery
cannot block us. Opens with `O_RDWR | O_NONBLOCK`.
The macOS backend uses a C shim (`macos_shim.c`) for IOKit exclusive access.
The shim handles:
1. `shim_open_exclusive(bsd_name)` — unmounts the target device via `diskutil`,
then walks the IOKit registry to find the `IOBDServices` matching the
requested BSD name (IOBDServices → IOBDBlockStorageDriver → IOMedia → "BSD Name"),
then creates MMCDeviceInterface → SCSITaskDeviceInterface → ObtainExclusiveAccess.
2. `shim_list_drives()` — registry-based enumeration with zero SCSI, zero exclusive
access, zero unmounts. Reads IOBDServices "Device Characteristics" for
vendor/model/firmware and child IOMedia "BSD Name" for the device path.
3. `shim_execute()` / `shim_close()` — raw CDB dispatch and cleanup.
On non-zero SCSI status, the transport parses sense key from the sense buffer
and returns `Error::ScsiError`.
`SgIoTransport::reset` (Linux) does pure userspace state cleanup: an open +
close pair to make the kernel cancel any SG_IO commands queued against a
previous fd, a 2 s sleep to let the kernel finish that cancellation, then a
fresh fd to send ALLOW MEDIUM REMOVAL to clear any stale tray lock. It does
NOT issue `SG_SCSI_RESET` or escalate via STOP+START UNIT. Both were tried
in 0.13.0–0.13.5 against the LG BU40N (Initio USB-SATA bridge); both failed
to recover wedged drives and made the wedge worse. The macOS reset (which
had been a no-op) was removed entirely in 0.13.6, and the top-level
`scsi::reset()` / `reset_with_timeout()` / `reset_blocking()` wrappers were
removed at the same time (no callers).
### CDB Builders
The `scsi` module provides platform-agnostic CDB constructors:
| Function | CDB | Use |
|----------|-----|-----|
| `inquiry()` | INQUIRY (0x12) | Drive identification |
| `get_config_010c()` | GET CONFIGURATION (0x46) | Feature 010C firmware date |
| `build_read_buffer()` | READ BUFFER (0x3C) | All platform commands |
| `build_set_cd_speed()` | SET CD SPEED (0xBB) | Speed control |
| `build_read10_raw()` | READ(10) (0x28) with flag 0x08 | Raw sector reads |
---
## Drive Identification
`DriveId::from_drive()` sends two standard SCSI commands and extracts identity
fields:
| Field | Source | SCSI Reference |
|-------|--------|----------------|
| `vendor_id` | INQUIRY bytes [8:16] | SPC-4 section 6.4.2 |
| `product_id` | INQUIRY bytes [16:32] | SPC-4 section 6.4.2 |
| `product_revision` | INQUIRY bytes [32:36] | SPC-4 section 6.4.2 |
| `vendor_specific` | INQUIRY bytes [36:43] | SPC-4 section 6.4.2 |
| `firmware_date` | GET CONFIGURATION Feature 010C | MMC-6 section 5.3.10 |
The match key is `"VENDOR|PRODUCT|REVISION|VENDOR_SPECIFIC"`. Profile matching
tries all four fields first, then falls back to matching without the firmware
date for drives where Feature 010C is unavailable.
---
## Drive Profiles
Profiles are JSON objects compiled into the binary (`profiles.json`).
Each profile contains:
| Field | Purpose |
|-------|---------|
| `vendor_id`, `product_revision`, `vendor_specific`, `firmware_date` | Matching fields |
| `chipset` | `"mediatek"` or `"renesas"` |
| `unlock_mode`, `unlock_buf_id` | READ BUFFER CDB parameters |
| `signature` | Expected 4-byte response signature |
| `unlock_cdb` | Pre-built unlock CDB (hex-encoded) |
| `register_offsets` | Offsets for hardware register reads |
| `capabilities` | Feature flags: `bd_raw_read`, `dvd_all_regions`, etc. |
Loading:
```rust
// Bundled (compiled-in) -- no file I/O
let profiles = profile::load_bundled()?;
// External file
let profiles = profile::load_all(Path::new("/path/to/profiles.json"))?;
```
---
## Chipsets
### MediaTek MT1959
Covers all LG, ASUS, and HP optical drives. Two sub-variants share identical
logic with different SCSI parameters:
| Variant | READ BUFFER mode | Buffer ID |
|---------|------------------|-----------|
| MT1959-A | 0x01 | 0x44 |
| MT1959-B | 0x02 | 0x77 |
The Platform trait maps to command handlers:
| Handler | Function | Description |
|---------|----------|-------------|
| 0 | `unlock()` | Send READ BUFFER, verify signature + verification bytes |
| 1 | `read_config()` | Read 1888-byte configuration block + 4-byte status |
| 2-3 | `read_register()` | Read hardware registers at profile-specified offsets |
| 4 | `calibrate()` | Probe disc surface, build 64-entry speed table |
| 5 | `keepalive()` | Periodic session maintenance |
| 6 | `status()` | Query current mode and feature flags |
| 7 | `probe()` | Generic READ BUFFER with dynamic parameters |
| 8 | `read_sectors()` | Speed lookup + SET CD SPEED + READ(10) with flag 0x08 |
| 9 | `timing()` | Timing calibration |
### Renesas (Planned)
RS8xxx/RS9xxx chipsets used in Pioneer and some HL-DT-ST drives.
Currently returns `Error::UnsupportedDrive` when a Renesas profile is matched.
---
## Why Unlock Is Needed
Optical drive firmware restricts what applications can read from disc. Without
unlock:
- **READ(10) works for unencrypted filesystem data.** UDF structures, MPLS
playlists, and CLPI clip info are readable without unlock. Standard READ(10)
works on any drive.
- **READ(10) fails for encrypted content sectors.** The drive firmware returns
SCSI errors (sense key 0x05, illegal request) when an application attempts to
read sectors containing encrypted m2ts content without prior AACS
authentication via the bus key.
- **Raw mode bypasses firmware restrictions.** After unlock, the drive accepts
READ(10) with the raw read flag (CDB byte 1 = 0x08) for all sectors,
regardless of encryption status.
### AACS Before Unlock
AACS bus authentication uses standard MMC REPORT KEY / SEND KEY commands.
On some drives these must execute before unlock. The `Disc::scan()` handles
this internally — it manages the handshake/unlock ordering automatically.
---
## Speed Control
After `probe_disc()`, the platform driver maintains a speed lookup table
built by probing the disc surface. On each `read()` call, the driver:
1. Looks up the optimal speed for the target LBA.
2. Issues SET CD SPEED (0xBB) if the speed differs from current.
3. Performs the READ(10).
Available speeds:
| Format | Speeds |
|--------|--------|
| Blu-ray | 1x (4,500 KB/s) through 12x (54,000 KB/s) |
| DVD | 1x (1,385 KB/s) through 16x (22,160 KB/s) |
| Max | 0xFFFF (drive decides) |

217
docs/mpls.md
View file

@ -1,217 +0,0 @@
# MPLS Playlist Format
## What is MPLS?
MPLS (Movie PlayList) files define playback titles on a Blu-ray disc. Each `.mpls` file in `BDMV/PLAYLIST/` describes one title -- a sequence of clips with precise in/out timestamps and stream information. The main movie, bonus features, trailers, and menus each have their own MPLS file.
A disc may contain dozens of MPLS files. Most are short (menus, logos, transitions). The main movie is typically the longest playlist. libfreemkv filters out playlists shorter than 30 seconds.
## File Structure
MPLS files use big-endian byte order throughout. The file has three main sections:
```
Offset Size Field
------ ---- -----
0 4 Magic: "MPLS"
4 4 Version: "0200" (BD) or "0300" (UHD BD)
8 4 PlayList start offset (absolute from file start)
12 4 PlayListMark start offset
16 4 ExtensionData start offset
```
### PlayList Section
Located at the PlayList start offset. Contains all play items and sub-path entries:
```
Offset Size Field
------ ---- -----
0 4 PlayList length
4 2 Reserved
6 2 Number of PlayItems
8 2 Number of SubPaths
10 ... PlayItem entries (variable length)
```
### Play Items
Each play item references one clip and specifies what portion to play:
```
Offset Size Field
------ ---- -----
0 2 PlayItem length (bytes after this field)
2 5 Clip ID (ASCII, e.g. "00001")
7 4 Codec ID ("M2TS")
11 1 Connection condition (lower 4 bits)
12 1 Ref to STC_id
14 4 IN_time (45kHz PTS ticks)
18 4 OUT_time (45kHz PTS ticks)
22 8 UO_mask_table
30 1 Misc flags
31 1 still_mode
32 ... STN_table (first play item only is parsed)
```
**Connection condition** values:
- `1` = seamless connection (no gap between clips)
- `5`, `6` = non-seamless connection
**Timestamps** use 45kHz PTS (Presentation Time Stamp) ticks, the same timebase as MPEG transport streams. Duration of a play item = `OUT_time - IN_time`. To convert to seconds: divide by 45000.
A playlist's total duration is the sum of all play item durations.
### Clip ID Mapping
The Clip ID (e.g. "00001") maps to:
- `BDMV/STREAM/00001.m2ts` -- the transport stream
- `BDMV/CLIPINF/00001.clpi` -- the clip info (EP map, stream details)
## STN Table
The Stream Number Table describes all elementary streams available in the clip. It is parsed from the first play item (which defines the title's stream layout).
```
Offset Size Field
------ ---- -----
0 2 STN table length
2 2 Reserved
4 1 Number of primary video streams
5 1 Number of primary audio streams
6 1 Number of PG (subtitle) streams
7 1 Number of IG (interactive graphics) streams
8 ... Stream entries
```
### Stream Entries
Each stream entry has two parts: a stream reference and a stream attributes block.
**Stream reference:**
```
Offset Size Field
------ ---- -----
0 1 Entry length
1 1 Stream type (1=PlayItem, 2=SubPath, 3=InMux)
2 2 PID (MPEG-TS packet ID, big-endian)
```
**Stream attributes** (immediately follows the reference):
```
Offset Size Field
------ ---- -----
0 1 Attributes length
1 1 Coding type
2+ ... Type-specific fields
```
The layout of type-specific fields depends on the stream category:
**Video streams:**
| Offset | Size | Field |
|--------|------|-------|
| 1 | 1 | Coding type |
| 2 | 1 | Format (upper 4 bits) + frame rate (lower 4 bits) |
**Audio streams:**
| Offset | Size | Field |
|--------|------|-------|
| 1 | 1 | Coding type |
| 2 | 1 | Format (upper 4 bits) + sample rate (lower 4 bits) |
| 3 | 3 | Language code (ISO 639-2, e.g. "eng") |
**PG subtitle and IG streams:**
| Offset | Size | Field |
|--------|------|-------|
| 1 | 1 | Coding type |
| 2 | 3 | Language code |
## Coding Types
The coding type byte identifies the codec:
| Value | Codec | Category |
|-------|-------|----------|
| `0x02` | MPEG-2 | Video |
| `0x1B` | H.264 / AVC | Video |
| `0x24` | HEVC / H.265 | Video |
| `0xEA` | VC-1 | Video |
| `0x80` | LPCM | Audio |
| `0x81` | AC-3 (Dolby Digital) | Audio |
| `0x82` | DTS | Audio |
| `0x83` | TrueHD (Dolby TrueHD) | Audio |
| `0x84` | AC-3 Plus (E-AC-3) | Audio |
| `0x85` | DTS-HD HR | Audio |
| `0x86` | DTS-HD MA | Audio |
| `0xA1` | AC-3 Plus (secondary) | Audio |
| `0xA2` | DTS-HD HR (secondary) | Audio |
| `0x90` | PGS (Presentation Graphics) | Subtitle |
| `0x91` | PGS (Interactive Graphics) | Subtitle |
## Video Format Codes
| Value | Resolution |
|-------|-----------|
| 1 | 480i |
| 2 | 576i |
| 3 | 480p |
| 4 | 1080i |
| 5 | 720p |
| 6 | 1080p |
| 7 | 576p |
| 8 | 2160p |
## Video Frame Rate Codes
| Value | Frame Rate |
|-------|-----------|
| 1 | 23.976 fps |
| 2 | 24 fps |
| 3 | 25 fps |
| 4 | 29.97 fps |
| 6 | 50 fps |
| 7 | 59.94 fps |
## Audio Format Codes
| Value | Channels |
|-------|----------|
| 1 | Mono |
| 3 | Stereo |
| 6 | 5.1 surround |
| 12 | 7.1 surround |
## Audio Sample Rate Codes
| Value | Rate |
|-------|------|
| 1 | 48 kHz |
| 4 | 96 kHz |
| 5 | 192 kHz |
| 12 | 48/192 kHz (combo) |
| 14 | 48/96 kHz (combo) |
## How Playlists Map to Titles
libfreemkv's `Disc::scan()` reads every MPLS file from the disc and builds a `Title` for each:
1. Read all `.mpls` files from `BDMV/PLAYLIST/` via UDF.
2. Parse each with `mpls::parse()`.
3. Calculate duration by summing `(OUT_time - IN_time)` across all play items.
4. Discard playlists shorter than 30 seconds.
5. For each play item, load the corresponding CLPI file to get EP map data and compute sector extents (see [clpi.md](clpi.md)).
6. Extract stream info from the STN table of the first play item.
7. Sort titles by duration, longest first.
The resulting `Title` struct contains everything needed to rip: streams, duration, byte size, and the sector extents to read from disc.
## References
- BD-ROM Part 3, Section 5.3: PlayList file format
- https://github.com/lw/BluRay/wiki/MPLS

201
docs/rip-recovery.md
View file

@ -1,201 +0,0 @@
# Rip recovery — three-layer architecture
`libfreemkv` supports a multi-stage rip model for damaged or protection-bearing
discs: a fast forward sweep that tolerates read failures, in-loop request-size
adaptation that survives transient drive trouble without bailing, and targeted
retry passes against a persistent bad-range map. The stream pipeline
(`DiscStream` + `input`/`output`) operates against the resulting ISO image, so
the mux stage never touches the drive.
Recovery is layered cleanly. Each layer has one responsibility and does not
reach into the others.
| Layer | Where it lives | What it does |
|-------|---------------|--------------|
| 1 — Bad-range retry | `Disc::patch` (one pass over the mapfile per call) | Re-reads non-`+` ranges with the long timeout. Idempotent; caller invokes N times. |
| 2 — Single-shot primitive | `Drive::read` in `src/drive/mod.rs` | One CDB, one timeout, one result. No inline retries, no SCSI reset. |
| 3 — In-loop request adaptation | `DiscStream::fill_extents` adaptive batch sizer | Halves the batch on failure, retries at the same LBA, walks back up on a clean-read streak. |
The library exposes flat verbs; the caller drives the multipass loop. Autorip
runs `Disc::sweep` once, then loops `Disc::patch` until either the mapfile is
clean or the configured retry budget is exhausted, then hands the ISO off to
the mux pipeline. The `freemkv` CLI does the same shape with a
terminal-output progress sink. Layer 3 runs inside any consumer of
`DiscStream` (direct PES pipeline, ISO playback, etc.) without caller
involvement.
Three primitives compose the disc-side flow:
| Primitive | What it does |
|---------------------------|-----------------------------------------------------------------------|
| `Disc::sweep` | disc → ISO, one forward pass. Writes a sidecar `.mapfile`. Opt-in skip-on-error. |
| `Disc::patch` | Re-reads bad ranges from the drive. One pass per call; caller invokes N times. |
| `DiscStream` (ISO source) | Reads sectors from the ISO, feeds decrypt → demux → codec → mux. |
## Data model
### Mapfile
Format: [ddrescue](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)-compatible
plain text, greppable, tool-interoperable. Flushed to disk on every `record()`
so a crashed rip loses at most one block.
```
# Rescue Logfile. Created by libfreemkv v0.13.6
# Current pos / status / pass / pass_time
0x000000000 ? 1 0
# pos size status
0x000000000 0x12a35d000 +
0x12a35d000 0x000003000 -
0x12a360000 0x009c4a000 +
0x12d00a000 0x000064000 *
```
Status characters match ddrescue:
| Char | Meaning |
|------|----------------------------------------------------|
| `?` | Not yet attempted |
| `*` | Fast-pass failed; needs edge-trim |
| `/` | Trimmed; interior needs sector scrape |
| `-` | Unreadable this session |
| `+` | Finished (good) |
Position and size are hex byte offsets into the ISO.
### `SweepOptions` and `PatchOptions`
The library no longer dispatches between sweep and patch internally — the
caller picks the verb explicitly per pass. The two option structs are flat
and have no overlap:
```rust
SweepOptions {
decrypt: true,
resume: false,
batch_sectors: None,
skip_on_error: true, // damage-jump + zero-fill on read failure
progress: Some(&reporter),
halt: Some(flag),
}
PatchOptions {
decrypt: true,
block_sectors: None,
full_recovery: true,
reverse: true, // walk bad ranges high → low LBA
wedged_threshold: 50,
progress: Some(&reporter),
halt: Some(flag),
}
```
Caller-orchestrated dispatch (the policy `Disc::copy` used to embed):
- No mapfile → `sweep` (fresh Pass 1).
- Mapfile with `?` ranges → `sweep` with `resume: true`.
- Mapfile covers full disc, only `*` / `/` / `-` ranges → `patch`.
- Mapfile clean → done; no further pass needed.
Each consumer (autorip, `freemkv` CLI) implements the loop in roughly five
lines of `Mapfile::stats()` checks.
## Algorithm
### Pass 1 — fast sweep (`Disc::sweep`)
1. Read one ECC block (32 sectors for UHD, 16 for BD/DVD) at the current LBA.
2. On success: write data to ISO, mark `+`, advance.
3. On failure (with `multipass`): zero-fill, mark `*`, advance.
4. Track a sliding window of the last 16 ECC block results. When ≥12% are failures
→ **damage-jump**: skip ahead by `256×batch×multiplier` sectors (8 MB base for
UHD). Double the multiplier on each jump (8→16→32→64 MB...). Zero-fill the gap as `*`.
5. On 16 consecutive good reads: reset jump multiplier to 1, restore max read speed.
6. Speed control: damage zone entry → minimum speed, exit → maximum speed.
7. Only transport failures (USB bridge crash) abort the pass.
Pass 1 completes when every byte has been visited (either `+` or `*`).
### Pass 2+ — patch (`Disc::patch`)
`Disc::patch` reads the mapfile and iterates every non-`+` range. Default: **reverse** mode
(walks ranges from highest LBA to lowest, within each range from end to start).
1. Issue a single-sector read with 60 s timeout (`recovery=true`). Drive firmware
does its own ECC recovery inside that window.
2. On success: write the good bytes into the ISO, mark `+`.
3. On failure with non-marginal SCSI sense: bail immediately (drive won't produce data).
4. On failure with marginal sense: mark `-`, continue.
5. Update the mapfile after every block — crash-safe resume.
6. Wedged-drive exit: 50 consecutive failures with zero recovery → bail this pass.
### In-stream — adaptive batch halving (`DiscStream::fill_extents`)
When a consumer reads a `DiscStream` directly (no ISO intermediate),
`fill_extents` runs an adaptive sizer in front of `Drive::read`:
1. Try the current preferred batch size (e.g. 32 sectors, one BD ECC block).
2. On failure: halve the batch and retry at the same LBA. Emit
`EventKind::BatchSizeChanged { reason: Shrunk }`.
3. On a clean-read streak: probe back up toward the preferred size. Emit
`EventKind::BatchSizeChanged { reason: Probed }`.
4. If a single-sector read fails: skip (zero-fill, emit
`EventKind::SectorSkipped`) when `skip_errors` is set, otherwise return
`Err(DiscRead)`.
This is layer 3. It exists so a transient single-sector glitch in a 32-sector
batch can be isolated and read individually without the caller needing to
implement retry logic.
## Design choices
**`Drive::read` is single-shot.** No inline retry phases, no SCSI reset,
no eject cycle. The `recovery` flag controls only the per-CDB timeout
(1.5 s vs. 30 s); on any failure it returns `Err(DiscRead)` immediately.
Inline recovery (5× gentle retry → close + SCSI reset + reopen → 5× more)
was removed in 0.13.6. See the stop-wedge postmortem (2026-04-25) for rationale:
the inline reset on the LG BU40N (Initio USB-SATA bridge)
wedged drive firmware below the bridge without ever recovering a sector,
and the gentle-retry phase produced long stretches of 0 KB/s with no
recoveries to show for it. Recovery responsibility is now layered: layer 1
handles ranges, layer 3 handles request size, neither touches the
wedge-prone reset path.
**No `MODE SELECT` to disable drive retries.** Research showed neither ddrescue
nor MakeMKV does this. Drive firmware has access to raw analog signal, laser
power control, and drive-specific ECC tuning that userspace can't replicate —
disabling it throws away recovery headroom on marginal sectors. We fail fast
via short SG_IO timeouts in pass 1 and let the firmware work the long timeout
in pass 2 / patch.
**No SCSI reset from any retry path.** `SgIoTransport::reset` (Linux) is
trimmed to a kernel SG_IO state flush plus ALLOW MEDIUM REMOVAL — the
`SG_SCSI_RESET` ioctl and STOP/START UNIT escalation were removed in 0.13.6.
The macOS reset (which had been a no-op) was removed entirely. The top-level
`scsi::reset()` / `reset_with_timeout()` / `reset_blocking()` wrappers were
also removed (no callers). The remaining `Drive::reset()` is only invoked
explicitly by callers that need an eject-cycle escape hatch — it is never
reached from a read path.
**ISO intermediate, even for single-pass.** Pass 1 always writes an ISO. The
mux stage reads the ISO via `IsoSectorReader`. For single-pass (no retries),
this adds ~2-3 min (local disk mux) but gains resumability across crashes,
re-muxability without re-ripping, and a persistent forensic artifact. Callers
who need pure speed can bypass and use `DiscStream::new(Box::new(drive), …)`
directly — the lib doesn't forbid it, and layer 3 (adaptive batch halving)
still applies there.
**Mapfile in ddrescue format.** Plain text so users can `less` it, `diff` it,
or feed it to ddrescue's own tooling. Crash-safe (flush-per-record). Entries
coalesce on adjacent same-status ranges so files stay small.
**Patches target `-`, `*`, `/`, and `?` alike.** The status state machine is
ddrescue's but `patch` collapses the distinction — it just tries every
non-finished range with the long timeout. Future work can specialize (trim vs.
scrape vs. retry with direction reversal) if there's measured benefit.
## References
- [ddrescue manual, Algorithm chapter](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html)
- [ddrescue optical media notes](https://www.electric-spoon.com/doc/gddrescue/html/Optical-media.html)
- Source: [`src/disc/mapfile.rs`](../src/disc/mapfile.rs), [`src/disc/sweep.rs`](../src/disc/sweep.rs) (`Disc::sweep`), [`src/disc/patch.rs`](../src/disc/patch.rs) (`Disc::patch`), [`src/drive/mod.rs`](../src/drive/mod.rs) (`Drive::read`), [`src/mux/disc.rs`](../src/mux/disc.rs) (`DiscStream::fill_extents`).

Some files were not shown because too many files have changed in this diff Show more