# realtime-calib > Real-time, open-source multi-camera calibration: recover camera intrinsics (focal length, distortion) and 6-DoF extrinsics for a rig of USB cameras, with live feedback. Runs headless in Docker, driven from any device, with Caliscope-compatible and engine-ready exports (Unity, Unreal, Blender, three.js, ROS). ## docs ### architecture - [Architecture overview](/docs/architecture/overview.md): How realtime-calib is built: a Python calibration service, a React web app, LiveKit WebRTC streaming and a Caddy reverse proxy, orchestrated with Docker Compose. ### faq Frequently asked questions about realtime-calib: headless operation, phones and tablets, GPU requirements, ChArUco boards, importing videos, camera sync, metric scale, engine exports and licensing. - [Frequently asked questions](/docs/faq.md): Frequently asked questions about realtime-calib: headless operation, phones and tablets, GPU requirements, ChArUco boards, importing videos, camera sync, metric scale, engine exports and licensing. ### getting-started - [Installation](/docs/getting-started/installation.md): Install and run realtime-calib on your own hardware with Docker Compose — works on headless Linux servers; the web app is served over your local network. - [Quickstart: your first calibration](/docs/getting-started/quickstart.md): Your first multi-camera calibration end to end: define a ChArUco board, set up USB cameras, calibrate intrinsics and extrinsics live, then review in 3D, check and export. ### guides - [Define a calibration board](/docs/guides/calibration-board.md): Configure the ChArUco board or single ArUco marker realtime-calib detects, download it to print, and enter its real measured size for an accurate metric scale. - [Configure cameras](/docs/guides/configure-cameras.md): Detect the USB cameras, set one shared capture configuration (resolution, frame rate, resize factor) and order the rig — the first camera anchors the extrinsics. - [Export](/docs/guides/export.md): Check a calibration before handing it over, then export it to the convention your target uses: Caliscope-native TOML, the aniposelib TOML for Pose2Sim and anipose, a versioned OpenCV JSON, or engine-ready JSON for Unity, Unreal, Blender, three.js and ROS. - [Extrinsic calibration](/docs/guides/extrinsic-calibration.md): Recover the 6-DoF position and orientation of every camera in one shared frame, refine with a rigidity-constrained bundle adjustment, review the solved rig in 3D and keep the room's frame across recalibrations. - [Intrinsic calibration](/docs/guides/intrinsic-calibration.md): Estimate each camera's focal length and distortion: capture a sweep with live coverage overlays, tune the replay, solve on diverse keyframes and read the projection uncertainty. - [Start or load a session](/docs/guides/start-or-load-session.md): Create a new calibration session, import pre-recorded videos (including Caliscope recordings), or resume an existing one — a session is a folder on the server holding its recordings, board config and results. ### intro Local, real-time, open-source multi-camera calibration: intrinsics and 6-DoF extrinsics for USB camera rigs, live feedback, Caliscope-compatible exports. - [What is realtime-calib?](/docs/intro.md): Local, real-time, open-source multi-camera calibration: intrinsics and 6-DoF extrinsics for USB camera rigs, live feedback, Caliscope-compatible exports. ### open-source - [License](/docs/open-source/license.md): realtime-calib is free software under AGPL-3.0 — what that means in practice, plus commercial licensing and custom development for proprietary products. ### realtime-calib-vs-caliscope An honest comparison for multi-camera calibration: Caliscope's record-then-calibrate desktop app vs realtime-calib's live, one-pass, headless web workflow — and when to use which. - [realtime-calib vs Caliscope](/docs/realtime-calib-vs-caliscope.md): An honest comparison for multi-camera calibration: Caliscope's record-then-calibrate desktop app vs realtime-calib's live, one-pass, headless web workflow — and when to use which. ### reference - [Calibration best practices](/docs/reference/calibration-best-practices.md): How to capture data that yields an accurate calibration: board choice, tilt and frame coverage, lighting, working distance, and how to read reprojection error. - [Camera calibration glossary](/docs/reference/glossary.md): Plain-language definitions of camera-calibration terms: intrinsics, extrinsics, ChArUco, reprojection error, bundle adjustment, triangulation and more. - [Calibration output files](/docs/reference/output-calibration-files.md): Field-by-field reference of exported calibration files: camera matrix, distortion, rotation, translation and reprojection error — Caliscope TOML, aniposelib TOML, OpenCV JSON and engine JSON. ### research - [Accuracy & benchmarks](/docs/research/benchmarks.md): How realtime-calib's accuracy is measured and published: reprojection error, target rigidity, held-out views and scale, with datasets and exact commits so every number can be reproduced. - [Bibliography & references](/docs/research/bibliography.md): The tools and literature behind realtime-calib: Caliscope, OpenCV, SciPy, LiveKit and the peer-reviewed references grounding the method. - [How to cite](/docs/research/citation.md): How to cite realtime-calib in academic work — CITATION.cff and a ready-to-use BibTeX entry. - [Methodology](/docs/research/methodology.md): What realtime-calib computes, grounded in sources: ChArUco intrinsics, diversity-based keyframe selection, timestamp synchronization, pairwise stereo poses, anchor chaining and a rigidity-constrained bundle adjustment. --- # Full Documentation Content # Architecture overview realtime-calib is a small set of services orchestrated with Docker Compose, built around one idea: the heavy lifting (capture, detection, calibration) happens on the **server the cameras are plugged into**, and the operator drives it from a **web app on any device** over the local network. ## How this project was built[​](#how-this-project-was-built "Direct link to How this project was built") I'm a **PhD in human-movement science and a computer-vision developer**: my work is building applied technology for **health and sport**. I'll delimit my expertise up front, the way a researcher scopes their field before presenting a result — because it matters here. I'm comfortable with engineering, product design and *applying* computer vision, but I do **not** have deep training in the projective geometry and epipolar mathematics that underpin camera calibration. So for the calibration theory I stand on **Caliscope** and **OpenCV**, and I used **Claude Code (Opus 4.8, then Opus 5.5)** to write the code and to explain the harder concepts as I went. I first used Caliscope in my own work. It calibrates well, but a few frictions kept getting in the way — recording every camera in OBS first, no headless path, and export conventions that didn't match my projects. As VR and robotics keep growing the need for multi-camera rigs, it seemed worth turning a friction-free version into something others could use too. My own contribution is therefore the **product and engineering shape**, not the calibration math: a **single-pass, multi-device** tool — a responsive web app that captures *and* calibrates in one flow, usable on **headless Linux** servers — and the **architecture and stack** (a dual-channel WebRTC React application over LiveKit). ## System topology[​](#system-topology "Direct link to System topology") | Service | Role | Stack | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- | | `calibration-service` | Capture, board detection, overlay burn-in, LiveKit publishing, calibration solves (intrinsic / extrinsic / bundle adjustment), HTTP API, session state | Python 3.14, FastAPI + asyncio, OpenCV, SciPy, LiveKit SDK | | `calibration-webapp` | Operator wizard + 3D review, served as static files by Caddy | React, TypeScript, Vite, Mantine, Redux Toolkit, R3F/drei | | `livekit-token-server` | Issues subscribe-only LiveKit JWTs to the web app | Python (Flask) | | `caddy` | Reverse proxy and static serving, plain HTTP — the **only host-exposed entry point** | Caddy v2 | | `livekit` | WebRTC SFU carrying the camera streams and the telemetry data channel | upstream `livekit/livekit-server` | It is a **single stack**: Caddy is the always-on entry point, in plain HTTP on the local network — tablet via `http://`, same-machine via `http://localhost`. No certificate is involved: the web app only receives video and data, which browsers allow without a secure context. Caddy routes `/api` to the calibration service, `/token` to the token server, `/livekit` to LiveKit signaling, and serves the web app for everything else. Only the WebRTC **media** flows outside Caddy, directly between browser and SFU. ## Two channels: commands vs. real time[​](#two-channels-commands-vs-real-time "Direct link to Two channels: commands vs. real time") The web app talks to the server over two complementary paths: * **HTTP (through `/api`)**: everything transactional: create, open or import sessions, configure cameras and boards, start and stop recordings, trigger computes, run the pre-export checks, export. The service owns the session state; the web app rehydrates from it. * **WebRTC (through LiveKit)**: everything continuous: one video track per camera (with detection overlays burned in server-side), plus a **data channel** pushing live telemetry: coverage, sharpness, co-visibility, and a snapshot of every camera's health (live, opening, in error, with the reason). The snapshot is resent on every tick rather than as one-off events, so a dropped packet or a tablet that joins late catches up within a second. ## Inside the calibration service[​](#inside-the-calibration-service "Direct link to Inside the calibration service") A single process: an asyncio event loop for the API and the orchestration, and threads for everything that blocks. Key properties: * **Recording stays native, the preview is light.** Recordings are written at the camera's native resolution, with each frame's timestamp. The live preview is downscaled (at most 960 px wide) for streaming, and live detection runs on that preview: it only drives the operator's feedback. The preview follows the camera's frame rate, or a reduced rate set in Settings. * **Solves are on-demand and replay-based.** A sweep is recorded first, then the compute re-detects the board on the native recording with the operator's Prepare settings (trim, stride, caps), in a worker thread so the live preview never freezes. * **Devices are never released mid-read.** Each open camera has its own thread, which serializes open, grab and release, so stopping a capture can never close a device while a frame is being read. A camera that fails to open or stops delivering frames is reported to the operator and reopened with a backoff. * **One long or mutating operation at a time.** Computes, Minimize, recording starts, session changes and camera or board changes take a single service-wide slot; a second request gets a "busy" answer instead of interleaving with the first. * **The session folder is the source of truth.** Recordings, board config and results all live there, written atomically (a crash never leaves a half-written file); the web app holds no durable state and rehydrates from the service on load. * **CPU-only.** No GPU required; the heavy cost is OpenCV detection, while the bundle adjustment itself takes under a second. --- # Frequently asked questions ## Does realtime-calib run headless, without a GUI?[​](#does-realtime-calib-run-headless-without-a-gui "Direct link to Does realtime-calib run headless, without a GUI?") Yes. The service runs in Docker on the machine the cameras are plugged into: **no desktop environment or GUI is needed on that host**. You drive the whole calibration from a web app on any device on the same local network. See [Installation](/docs/getting-started/installation.md). ## Can I run a calibration from a phone or a tablet?[​](#can-i-run-a-calibration-from-a-phone-or-a-tablet "Direct link to Can I run a calibration from a phone or a tablet?") Yes. The operator interface is a **responsive web app** served over your LAN in plain HTTP; desktop, tablet and phone all work, in landscape or portrait. The heavy computation stays on the server. ## Do I need a GPU for camera calibration?[​](#do-i-need-a-gpu-for-camera-calibration "Direct link to Do I need a GPU for camera calibration?") No. Everything (board detection, live overlays, intrinsic and extrinsic solves, bundle adjustment) runs on **CPU**. No cloud either: camera streams never leave your local network. See the [architecture overview](/docs/architecture/overview.md). ## Which calibration board should I use?[​](#which-calibration-board-should-i-use "Direct link to Which calibration board should I use?") It depends on the step. **For intrinsic calibration, use a ChArUco board** (the intrinsic solve requires one). Its ArUco markers make detection robust to occlusion and partial views, while its interpolated chessboard corners give subpixel accuracy: a good fit when the board fills a fair part of the frame, one camera at a time. **For extrinsic calibration in a large volume** (a room, a hall, a wide capture space), a single **ArUco marker** is often the better choice. Seen from across a large volume, the squares of a ChArUco board become too small in the image to detect reliably; the alternative, printing a very large ChArUco board, is bulky and expensive. A single ArUco marker stays detectable at a distance, so a **small, cheap board is enough** to link the cameras. Whichever you use, **measure it with calipers after printing** and enter the real size: that measurement sets the metric scale. See [Define a calibration board](/docs/guides/calibration-board.md) and the [best practices](/docs/reference/calibration-best-practices.md). ## Can I calibrate videos I have already recorded?[​](#can-i-calibrate-videos-i-have-already-recorded "Direct link to Can I calibrate videos I have already recorded?") Yes. **Load from files** on the Dashboard imports an archive (ZIP or tar) of pre-recorded videos, named `cam_` under `intrinsics/` and optionally `extrinsics/`, and runs the whole wizard on them with no camera attached. A Caliscope recording drops in as it is; videos without real timestamps are synchronized the way Caliscope aligns them. See [Start or load a session](/docs/guides/start-or-load-session.md#load-from-files). ## Can I export the calibration to Unity, Unreal, Blender, three.js, ROS or OpenCV?[​](#can-i-export-the-calibration-to-unity-unreal-blender-threejs-ros-or-opencv "Direct link to Can I export the calibration to Unity, Unreal, Blender, three.js, ROS or OpenCV?") Yes. Export writes **one file per selected target** with the axis remap and handedness already applied: three.js / OpenGL (Y-up, right-handed), Blender / ROS (Z-up, right-handed), Unity (Y-up, left-handed), Unreal (Z-up, left-handed), plus a versioned OpenCV JSON that OpenCV pipelines read without conversion, Caliscope's `camera_array.toml` and the aniposelib TOML used by Pose2Sim and anipose. See [Export](/docs/guides/export.md). ## Is realtime-calib compatible with Caliscope?[​](#is-realtime-calib-compatible-with-caliscope "Direct link to Is realtime-calib compatible with Caliscope?") Yes, both ways. `camera_array.toml` is written in the layout Caliscope v0.11.5 loads, translations in metres, with project-specific fields strictly additive; the aniposelib export covers Pose2Sim, anipose and older Caliscope projects. And Caliscope recordings can be imported and calibrated here. For a full comparison, see [realtime-calib vs Caliscope](/docs/realtime-calib-vs-caliscope.md). ## Do my cameras need hardware synchronization (genlock)?[​](#do-my-cameras-need-hardware-synchronization-genlock "Direct link to Do my cameras need hardware synchronization (genlock)?") No hardware sync is required. Each frame is stamped with the camera driver's own buffer timestamp, and frames are **grouped into synchronized instants** within just under one frame period. Groups where the target moved between the cameras' exposures are detected and set aside, and you can tighten the **maximum sync spread** (in milliseconds) when preparing the [extrinsic solve](/docs/guides/extrinsic-calibration.md). ## Why does the live preview look lower-quality than the resolution I selected?[​](#why-does-the-live-preview-look-lower-quality-than-the-resolution-i-selected "Direct link to Why does the live preview look lower-quality than the resolution I selected?") That is expected, and it does **not** affect your calibration. To keep the CPU encoder and the network light, the **live preview** streamed to your browser is downscaled to at most **960 px wide** (e.g. 960×540 from a 1080p camera), and the live detection overlay is computed on that preview. It is a display stream only. **Recording and the calibration itself run at the full resolution you selected**: every compute re-detects the board on the native recordings. You can confirm it by checking the resolution of the **recorded videos** in the session folder: they are written at the capture resolution, not at 960 px. See [Configure cameras](/docs/guides/configure-cameras.md) for how the calibration resolution is chosen. ## What operating system does the camera server need?[​](#what-operating-system-does-the-camera-server-need "Direct link to What operating system does the camera server need?") A **Linux host**: cameras are read via V4L2 and the stack runs with Docker Compose. The operator device only needs a modern browser, on any OS. See [Installation](/docs/getting-started/installation.md). ## Can I use RTSP or IP cameras, or only USB cameras?[​](#can-i-use-rtsp-or-ip-cameras-or-only-usb-cameras "Direct link to Can I use RTSP or IP cameras, or only USB cameras?") Right now, **only USB cameras are supported** for live capture: they are read via V4L2 on the Linux host. RTSP streams and IP cameras are **not supported yet**, though videos recorded by any camera can be [imported](/docs/guides/start-or-load-session.md#load-from-files). Live RTSP / IP-camera support is a natural extension. If you need it, **[open an issue or a feature request on GitHub](https://github.com/hans-brgs/realtime-calib/issues)**: demand is what drives the roadmap. ## How accurate is the calibration?[​](#how-accurate-is-the-calibration "Direct link to How accurate is the calibration?") realtime-calib follows the same calibration lineage as Caliscope and OpenCV: ChArUco intrinsics with the classic 5-coefficient distortion model, stereo-initialized extrinsics refined by a bundle adjustment that holds the target to its printed geometry ([methodology and sources](/docs/research/methodology.md)). On recorded hand-held sweeps, the per-camera reprojection error lands between 0.1 and 0.8 px at the output resolution, with the reconstructed target rigid to under 0.5 mm. Accuracy still depends mostly on **your capture** (board quality, tilt, frame coverage): see the [best practices](/docs/reference/calibration-best-practices.md). Each result comes with its own quality report, so you can judge it rather than trust it. Public [benchmarks](/docs/research/benchmarks.md) are being assembled. ## How do I know the metric scale is right?[​](#how-do-i-know-the-metric-scale-is-right "Direct link to How do I know the metric scale is right?") The scale comes from the size you measured on the printed target: a print 2 % off makes every distance 2 % off, and no internal check can see it, since the whole solve scales together. To check it, write a **site template** with a few camera-to-camera distances measured with a tape: the export then compares the calibration's scale against them and names a distance that disagrees. See [Export](/docs/guides/export.md#site-template). ## Is realtime-calib free?[​](#is-realtime-calib-free "Direct link to Is realtime-calib free?") Yes: free and open source under **AGPL-3.0**. If you need to embed it in a proprietary product or offer it as a closed service, a [commercial license and custom development](/docs/open-source/license.md#commercial-use) are available. --- # Installation realtime-calib runs entirely on your own hardware. The complete application is run with **Docker Compose**: that is the one supported way to bring up the full stack. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A **Linux host** with one or more **USB cameras** (read through V4L2). * **Docker** and **Docker Compose v2**. * A modern browser on the operator device (desktop, tablet or phone). ## One-time setup[​](#one-time-setup "Direct link to One-time setup") ``` # 1. Environment file cp .env.example .env # Edit .env: set HOST_IP to the machine's LAN IP (e.g. 192.168.1.42) and choose # dedicated LiveKit API keys. ``` No certificate is needed: the stack serves plain HTTP on your local network. Host IP The host IP is centralized in `HOST_IP` (in `.env`) and propagated to Caddy, LiveKit and the web app build. Do not hard-code IPs elsewhere. ## Launch[​](#launch "Direct link to Launch") The whole stack (calibration service, web app, LiveKit SFU, token server and the Caddy reverse proxy) is orchestrated by `docker-compose.yml` as a **single stack**. Caddy is the mandatory, always-on entry point. ``` docker compose up --build ``` Then open the web app: * **Tablet, phone or other LAN device**: `http://` * **Same machine**: `http://localhost` The web app only receives video and data from the server, which browsers allow over plain HTTP. One small exception: the **copy** button of the export preview needs a secure context, so it works at `http://localhost` but not from a tablet; download the files instead. ## Verifying the install[​](#verifying-the-install "Direct link to Verifying the install") ``` docker compose logs -f calibration-service ``` Open `http://` (or `http://localhost`) and you should land on the operator **Dashboard** ("Welcome to the calibration bench"). ## Where your data lives[​](#where-your-data-lives "Direct link to Where your data lives") Every calibration is a **session folder** under `./sessions/` on the host (mounted into the service). It holds the recordings, the board and camera configuration, the results and the exports. Back up or copy that folder to keep a calibration; nothing is stored in the browser. ## Networking notes[​](#networking-notes "Direct link to Networking notes") * **Caddy is the only host-exposed entry point**, serving plain HTTP on `80` (override with `CADDY_HTTP_PORT`). Everything else sits on an internal Docker bridge. * Internal services are not published on the host: `calibration-service` (`8000`) and `livekit-token-server` (`8080`) are reachable only through Caddy. * **LiveKit** runs on the bridge (not host networking). Its media ports are published to the host and advertised at `HOST_IP`: **UDP `50000-50010`** (WebRTC media) and **TCP `7881`** (ICE-TCP fallback). Signaling (`7880`) stays internal; Caddy proxies it as `ws`. * Traffic on your local network is not encrypted. The stack is meant for a trusted LAN, not for exposure to the internet. ## Upgrading from an HTTPS install[​](#upgrading-from-an-https-install "Direct link to Upgrading from an HTTPS install") Earlier versions served the web app over HTTPS with a locally generated certificate. To upgrade: * In `.env`, `CADDY_HTTPS_PORT` is no longer read. Set `CADDY_HTTP_PORT` only if port 80 is taken. * Pull the new version and rebuild the whole stack: `git pull`, then `docker compose up -d --build`. Every image changes, and the web app bakes its URLs in at build time. * Open `http://`. A browser that cached a redirect to `https` may need the address retyped with `http://`. Sessions recorded with an earlier version open as they are: their files are migrated when loaded. Solves computed by an earlier version are not comparable with new ones, because the calibration pipeline has changed since; recompute them to compare. Next Continue to the [Quickstart](/docs/getting-started/quickstart.md) to run your first calibration. --- # Quickstart: your first calibration This tutorial walks through a complete calibration end to end. It assumes the stack is [installed and running](/docs/getting-started/installation.md). Each step links to its detailed guide. ## First access from a tablet or phone[​](#first-access "Direct link to First access from a tablet or phone") Open `http://` in the device's browser (plain `http`, no certificate warning). Use the IP of the machine running the stack, the one set as `HOST_IP` in `.env`. ## 1. Start or load a session[​](#1-start-or-load-a-session "Direct link to 1. Start or load a session") Open the web app: you land on the **Dashboard** ("Welcome to the calibration bench"). Nothing in the wizard is reachable yet: **every step stays locked until a session exists.** Pick one of the two entry modes: * **New realtime calibration**: start the full wizard from scratch, with live capture. Each sweep is recorded to the session folder so it can be replayed and recomputed later. * **Load from files**: upload an archive of pre-recorded camera videos (for example a Caliscope recording) and run the same wizard on them, without any camera attached. Once a session is created or opened, the wizard rail unlocks and the wizard goes to the session's persisted step. Each screen explains itself behind the **ⓘ** button next to its title. On a wide screen, the settings panel on the right folds away from the button at the end of the title row, to give the camera view the whole width; on the capture steps, the action buttons then move under the view. → Details: [Start or load a session](/docs/guides/start-or-load-session.md) ## 2. Define a board: Target Config[​](#2-define-a-board-target-config "Direct link to 2. Define a board: Target Config") Go to **Target Config**. Set up the intrinsic board: a **ChArUco** board, which the intrinsic solve requires. The extrinsic step **inherits it by default**, or you can define a distinct extrinsic target, such as a single large **ArUco marker** for a big room. Download the PNG, print it, then **measure the printed square or marker with a caliper** and enter that size on the extrinsic board: it sets the metric scale of the whole rig. Save the intrinsic board first (the Extrinsic tab unlocks then), then the extrinsic one: **Camera Setup stays locked until both are saved.** → Details: [Define a calibration board](/docs/guides/calibration-board.md) ## 3. Configure your cameras: Camera Setup[​](#3-configure-your-cameras-camera-setup "Direct link to 3. Configure your cameras: Camera Setup") Go to **Camera Setup**. The service discovers the connected USB cameras and shows a live preview for each. Pick the shared resolution and frame rate, the output resize factor, and drag the cameras into order (the first one is the extrinsic anchor). **Apply** writes the configuration; **Continue** moves on. → Details: [Configure cameras](/docs/guides/configure-cameras.md) ## 4. Intrinsic calibration (camera by camera)[​](#4-intrinsic-calibration-camera-by-camera "Direct link to 4. Intrinsic calibration (camera by camera)") For each camera, start a capture and move the board through the whole field of view, tilted, while the live overlay shows the board and its coverage. Stop, then use **Prepare** to replay the recording, **trim** it, set the **sampling stride** and the **keyframe cap**, and compute. Read the results (reprojection error, coverage, projection uncertainty) before moving to the next camera. → Details: [Intrinsic calibration](/docs/guides/intrinsic-calibration.md) ## 5. Extrinsic calibration & 3D review[​](#5-extrinsic-calibration--3d-review "Direct link to 5. Extrinsic calibration & 3D review") Start a **synchronized sweep** and move the target through the shared volume so every camera pair sees it at the same instants; the co-visibility matrix fills as pairs accumulate joint views. Stop, then **Prepare** the compute (sampling stride, max groups, max sync spread) and solve. Read the per-camera error and the board rigidity, then **inspect the reconstructed rig in the 3D view**: frame the world on a board laid on the floor, snap-rotate the axes, optionally **Minimize**, or **align on a reference** calibration to keep the room's frame. → Details: [Extrinsic calibration](/docs/guides/extrinsic-calibration.md) ## 6. Check and export[​](#6-check-and-export "Direct link to 6. Check and export") The **Export** screen first shows the **pre-export checks**: worst camera error, epipolar consistency, target rigidity, world framing, cameras above the floor, and, with a site template, cabling, placement and scale. Then pick the length unit and the targets (Caliscope TOML, aniposelib TOML, OpenCV JSON, engine JSON for three.js, Blender / ROS, Unity, Unreal) and download. → Details: [Export](/docs/guides/export.md) You're done You now have a full intrinsic + extrinsic calibration, exported to the convention your target needs. See the [Calibration output files](/docs/reference/output-calibration-files.md) reference for the exact fields. --- # Define a calibration board **Target Config** is the first wizard step: configure the board(s) realtime-calib will detect, download them to print, then enter the measured size of the extrinsic target. Each board is saved with its own **Save** button, and Camera Setup unlocks once both are saved. ## Two roles: intrinsic and extrinsic[​](#two-roles-intrinsic-and-extrinsic "Direct link to Two roles: intrinsic and extrinsic") The screen has two tabs, **Intrinsic** and **Extrinsic**, one per role: * The **intrinsic board** must be a **ChArUco** board: the intrinsic solve needs its many identified chessboard corners. * The **extrinsic board** links the cameras into one world. By default it **inherits** the intrinsic board's geometry. Tick **"Use a different board for extrinsic"** to define a distinct target: either another ChArUco board or a **single ArUco marker**, which stays detectable from across a large room where the squares of a printable ChArUco board would be too small. When the extrinsic board inherits, it follows the intrinsic board: editing the intrinsic grid updates it, and keeps the size you measured. Save the intrinsic board first: the **Extrinsic** tab stays locked until it is saved. Above the **Save** button, a status line tells you where the current tab stands: saved, edited since the last save, or not saved yet. The next steps always use the saved boards, never unsaved edits, and **Save** is disabled while nothing has changed. ## Geometry (renders the printable PNG)[​](#geometry-renders-the-printable-png "Direct link to Geometry (renders the printable PNG)") * **Type**: ChArUco (a grid) or ArUco (a single marker). * **Dictionary**: an OpenCV predefined ArUco dictionary (default `DICT_4X4_100`). A 4×4 bit grid reads from farther away than 5×5 or 6×6; a smaller dictionary keeps more distance between valid patterns. * **Columns × rows** (ChArUco): the grid size, default 7 × 9. * **Marker ratio** (ChArUco): the ArUco marker inside each white square, as a fraction of the square (default 0.75). * **Marker ID** (ArUco): which marker of the dictionary to print. * **Inverted (ink saving)**: prints the board in negative (white on black). Inverted boards are detected like normal ones. * **Legacy layout (OpenCV < 4.6)** (ChArUco): OpenCV 4.6 changed the layout of boards with an even number of rows. Turn this on for a board printed by an older tool, which would otherwise detect no corner. A live preview updates as you edit, and you **Download the PNG** to print it. The preview and the download come from the same server-side render engine used for detection. For a single marker, **keep the white margin around it** when you trim the print: the corner refinement reads the marker's black border against that margin, and refuses views where the margin is too thin. ## Metric scale (measured after printing)[​](#metric-scale-measured-after-printing "Direct link to Metric scale (measured after printing)") The physical scale does **not** come from the render: it comes from **measuring your printed target**. After printing, measure a printed square (ChArUco) or the marker side (ArUco) with a caliper, and enter that value in millimetres on the **extrinsic** board. The intrinsic solve is scale-free and does not need it. That measurement is what puts the extrinsic translations into real-world units, and it carries straight through: a print 2 % off makes every distance the rig reports 2 % off. No internal quality check can see this, since the whole solve scales together; the tape-measured distances of a [site template](/docs/guides/export.md#site-template) can. ## Changing the target after a solve[​](#changing-the-target-after-a-solve "Direct link to Changing the target after a solve") A target change does not silently rescale or invalidate a finished calibration: * **A new measurement of the same target** keeps the extrinsic solve; the exported translations follow the new size. * **A new geometry** for the extrinsic target (type, dictionary, grid, ratio, marker ID, inverted, legacy layout) makes the solved array obsolete. The web app asks for confirmation first; confirming discards the solve but keeps the recorded sweep, so the Extrinsic step offers **"Recompute from the recorded sweep"**. → See also: [Calibration best practices](/docs/reference/calibration-best-practices.md): board type, dictionary and geometry. --- # Configure cameras Detect the USB cameras, set one shared capture configuration, and order them: the first camera is the extrinsic anchor. ## What happens here[​](#what-happens-here "Direct link to What happens here") The `calibration-service` enumerates the connected USB cameras (V4L2) and publishes a live preview of each over LiveKit. In the web app's **Camera Setup** view you can: * **Detect / re-detect** the connected cameras and confirm each is streaming. * Set the **capture configuration**: resolution, frame rate and resize factor. These are **shared by all cameras** and only offer the modes common to every detected camera. * **Order the cameras** by drag-and-drop. The camera at the top (**index 0**) is the **anchor** used to chain extrinsic poses. Cameras are identified by their stable USB path (`/dev/v4l/by-path/…`), so a camera keeps its identity as long as it stays on the same port. ## Apply, then continue[​](#apply-then-continue "Direct link to Apply, then continue") The camera order and the capture format sit in one card, because **Apply configuration** writes both at once. Your edits (parameters **and** order) form a draft that nothing reads until you click **Apply configuration**. Apply is enabled while the draft differs from the saved configuration; **Continue to Intrinsics** is enabled once there is nothing left to apply. This way you can check that a change took effect before moving on. Applying a changed configuration rebuilds the cameras, and a rebuild discards the calibrations already computed: their recordings were made with the previous configuration, and are keyed by camera index. The web app asks for confirmation first. ## Capture configuration[​](#capture-configuration "Direct link to Capture configuration") Because the rig is calibrated as one system, resolution and frame rate are picked **once and applied to every camera**: the selectors only offer modes supported by all detected cameras. The frame rate is **30 or 15 fps**, capped by each camera's native maximum for the chosen resolution. Lower rates spare the USB bus when many cameras share it. ## Resolution vs. resize factor[​](#resolution-vs-resize-factor "Direct link to Resolution vs. resize factor") Two independent controls set a pixel count, and they act at different places: * **Resolution** selects the camera's **native capture mode**: what is recorded and calibrated. A lower mode records and computes faster, but beware: **most USB cameras produce a lower resolution by cropping the sensor**, not by scaling it down, so dropping the resolution often **narrows the field of view**. You gain speed but lose coverage. * **Resize factor** *(s)* sets the **output resolution** of the exported calibration (1, 0.75, 0.5, ⅓ or 0.25) for downstream tools that work on images resized with `cv2.resize`. It changes neither what is recorded nor how it is calibrated: the **full field of view** is kept, and accuracy is unaffected. So the resize factor is the resolution your downstream pipeline works at; the only lever on the cost of capture and calibration is the native mode, with its field-of-view trade-off. Whichever you choose, the intrinsics are calibrated on the native frames, then reported at the output resolution `round(native × s)`. The stored **K** corresponds to that resolution, mapped the way `cv2.resize` maps pixel centres onto that declared size (`c' = s · (c + 0.5) − 0.5` on each axis, with the factor of the rounded size), so it projects exactly onto images resized to it. The distortion coefficients do not depend on the resolution. Every reported error, intrinsic and extrinsic, is in pixels at that output resolution too. ## Camera health[​](#camera-health "Direct link to Camera health") Each preview tile carries a status dot that follows the camera's real state: live, opening, or in error. * A camera that **cannot be opened**, or that **stops delivering frames** for about 3 seconds (unplugged, USB bandwidth exhausted), shows an overlay with the reason, and a banner lists the failing cameras. * The service retries on its own, with a growing delay (1, 2, 4, 8, 16, then every 30 seconds), and the camera comes back live without reloading the page once it streams again. * A camera that streams another resolution than the one configured is reported in error rather than taken live, since its recording would not match its calibration. ## Rig settings[​](#rig-settings "Direct link to Rig settings") The **Settings** window (from the rail) holds settings of the rig rather than of a session: * **Recording quality (JPEG)**: the quality of the recorded videos every compute re-detects from (85 to 100, default 95). * **Preview FPS**: the preview follows the camera's frame rate by default; a reduced preview rate spares the server's video encoder. Recording, detection and computes are never affected. * **Site template**: see [Export](/docs/guides/export.md#site-template). ## Imported sessions[​](#imported-sessions "Direct link to Imported sessions") In a session [loaded from files](/docs/guides/start-or-load-session.md#load-from-files), Camera Setup is read-only: it shows a frame from the middle of each camera's extrinsic sweep (the same moment on every camera, to check the order), or of its intrinsic sweep when there is no extrinsic one. The order comes from the file names. → Reference: [Calibration output files](/docs/reference/output-calibration-files.md) --- # Export Check the calibration, then export it in the convention your target needs. ## Pre-export checks[​](#pre-export-checks "Direct link to Pre-export checks") ![The Export screen: the camera\_array.toml preview on the left, the export targets and the checks before export on the right](/assets/images/export-checks-7fc1b572e0cde63cb929e750533b7c05.png) The Export screen shows a **Checks before export** panel under the targets, recomputed from the solve and also written as `checks.json` with every export. The checks are **read-only and never block** an export: they tell you what to look at before handing a calibration over. Each one reads ok, warn, fail, or unavailable (with its cause); tap its info icon for what it proves. | Check | What it looks at | Bands | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | Per-camera error | The worst camera's extrinsic error, output px | ok ≤ 0.6 · warn ≤ 1.2 px | | Epipolar consistency | Per camera pair, the median symmetric epipolar distance of the solve's observations, output px (needs 20 shared observations) | ok ≤ 0.5 · warn ≤ 1.0 px | | Target rigidity | The reconstructed target's deviation, as a share of its longest side | ok ≤ 0.25 % · warn ≤ 1 % | | World frame | The printed face's angle to the up axis: warns if the world is not framed on a target or is tilted, fails if upside down, notes an origin that drifted after a Minimize | | | Cameras above the floor | Cameras under the level framed target | fails if any | | Reference calibration | Whether the world still follows the [reference calibration](/docs/guides/extrinsic-calibration.md#keeping-the-rooms-frame-across-recalibrations) you aligned on, and if not, how far off it is | | These checks are **internal**: computed from the solve, they spot an inconsistency but cannot see an error the whole solve shares. The worst such error is **scale**: a mis-measured target rescales the whole rig uniformly, and every internal check still passes. A site template adds the external checks. ## Site template[​](#site-template "Direct link to Site template") A site template describes the room as you know it independently of the solve: which camera goes to which port, where each one sits, and a few distances you measured with a tape. Load it from the **Settings** window (a JSON file, stored next to the sessions and shared by all of them; load it again once corrected, or remove it). It enables four more checks, all unavailable without a template: | Check | Fails when | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cameras at their ports | A template camera is missing or at another port, i.e. a swapped cable or a changed order (an extra camera warns) | | Camera placement | A camera breaks its position, pitch or yaw bounds | | Scale (tape distances) | The scale implied by your tape distances is more than 3σ away from 1 (warns from 2σ); a tape distance that disagrees with the others is named. The scale is reported, never applied | | Export resolution | A camera does not export at the template's resolution | ``` { "name": "Lab A", "resolution": [960, 540], "cameras": [ { "device_path": "/dev/v4l/by-path/pci-0000:00:14.0-usb-0:1:1.0-video-index0", "port": 0, "position_m": { "y": [2.0, 2.25] }, "pitch_deg": [25, 36], "yaw_to_origin_deg_max": 12 } ], "distances_m": [ { "a": "/dev/v4l/by-path/…-usb-0:1:1.0-video-index0", "b": "/dev/v4l/by-path/…-usb-0:2:1.0-video-index0", "m": 3.42, "sigma_m": 0.01 } ] } ``` * Cameras are named by their `device_path` (shown in Camera Setup), each with the `port` (camera index) it is expected at. * Placement bounds are read in the exported world (Y up, metres): `position_m` gives a `[min, max]` range per axis (`y` is the height above the floor; an omitted axis is free), `pitch_deg` the angle below the horizontal, `yaw_to_origin_deg_max` how far the camera may look away from the origin. Leave a margin: a camera installed at 36° may solve at 36.6°. * `distances_m` join two cameras' optical centres. `sigma_m` (default 1 cm) should cover the uncertainty on where the optical centre sits, not only the tape's. Four to six distances make the scale check sensitive to a bias of about half a percent. * The template is validated strictly: a duplicate camera or port, a reversed range, an unknown field or a distance to an unknown camera is refused with the field named, so a typo cannot silently drop a bound. * `checks.json` records the template's name and a SHA-256 digest of it, so an export says which template it was checked against. ## Targets and units[​](#targets-and-units "Direct link to Targets and units") Export is **convention-first**: you pick the **target software** and the length **unit** (mm or m), and you get **one file per selected target**. The targets are independent and equal: * **Caliscope**: `camera_array.toml`, the layout Caliscope itself loads, one `[cameras.N]` table per camera. Its translations are always in **metres**, Caliscope's world unit. * **aniposelib**: `camera_array_aniposelib.toml`, the layout Pose2Sim and anipose read, one `[cam_N]` table per camera, in the selected unit. * **OpenCV**: `camera_array_opencv.json`, a versioned contract for OpenCV pipelines: world → camera `R` and `t`, in a Y-up right-handed world, always in **metres**, with the world's provenance stated. * **Engine JSON**: `camera_array_.json` for **three.js** (Y-up, right-handed), **Blender / ROS** (Z-up, right-handed), **Unity** (Y-up, left-handed) and **Unreal** (Z-up, left-handed), with the axis remap and handedness already applied. The unit selector applies to the aniposelib TOML and the engine JSON files. The export is downloaded as one archive holding only the files of the current selection, plus `checks.json`. Your choice of targets and unit is saved with the session. The preview's **copy** button needs a secure context: it works at `http://localhost`, not from a tablet on the LAN, where you download the files instead. → Reference: [Calibration output files](/docs/reference/output-calibration-files.md) --- # Extrinsic calibration Recover the 6-DoF pose of every camera in a single shared coordinate frame. ## The flow[​](#the-flow "Direct link to The flow") Like intrinsics, extrinsics are computed **on demand**, from a recording, in four phases: 1. **Capture**: click **Start synchronized sweep** and move the target through the shared volume so that **pairs of cameras** see it at the same instants. Every camera records, and the **co-visibility matrix** fills as each pair accumulates joint views; tap a cell to see which pair it stands for. Move steadily rather than fast: groups where the target moved between the cameras' exposures are set aside at compute time. 2. **Prepare**: replay the **synchronized frame groups** (every camera's frame of the same instant, side by side) and tune the compute: * **Sampling stride (1 group / N)**: how many candidate groups the detection walks through (default 2 for a single marker, 12 for a ChArUco board, whose detection costs more); * **Max groups (best kept)**: the sharpest groups kept for the solve, spread over the sweep (default 240 for a single marker, which only carries 4 corners per view, 80 for ChArUco); * **Max sync spread (ms)**: groups whose timestamps spread wider than this are discarded. 3. **Computing**: detection on the native recordings, pairwise poses, chaining from the anchor, then bundle adjustment. The dialog announces how long the detection phase will take. 4. **Result**: read the quality figures, then inspect the reconstructed rig in the **3D review**. If the solve is discarded later (a new target geometry, recomputed intrinsics), the recording is kept and this step offers **Recompute from the recorded sweep** next to Start synchronized sweep. Applying a new camera configuration, on the other hand, drops the sweep: its videos no longer match the cameras. ## Reading the result[​](#reading-the-result "Direct link to Reading the result") All errors are in pixels at the **output resolution** (native × resize factor), the same unit as the intrinsic errors. * **Reprojection error**: the RMS over every observation the solve used. * **Per-camera deviation**: what tells you to re-shoot one camera rather than redo the array. Up to 0.6 px reads green, up to 1.2 px amber: hand-held reference sweeps land between 0.1 and 0.8 px. Beyond that, look for a cause of that camera's own (focus, sync, intrinsics). The anchor camera is marked. * **Board rigidity**: how far the reconstructed target deviates from its printed geometry (RMS, mm). It does not depend on the reprojection error, so it catches a solve that lowered its residuals by bending the board. Up to 2 mm (the tolerance the solver is given) is nominal, up to 5 mm worth a second look. * **Groups / 3D points**: how much evidence the solve stood on. A low error over 3 groups is not the same result as over 40. * **Observations**: after a Minimize, how many observations were kept out of the total. A warning appears if the bundle adjustment stopped at its iteration ceiling instead of converging. ## The 3D review[​](#the-3d-review "Direct link to The 3D review") The view shows every camera as a frustum (its image plane tinted, a small triangle on its top edge), the target of the group selected with the slider, and its triangulated corners: * **Navigate**: drag to rotate freely (one finger on a tablet), scroll or pinch to zoom, right-drag or two fingers to pan. **Recenter** (top-right corner) returns to the default upright view. The view is framed once per solve: scrubbing the slider or folding a panel does not move it. * **World / Board** (top-right corner): draws either the world axes (X, Y, Z, over a floor grid once the world is framed on a target) or the axes of the target in the selected group (x, y, z). The world frame starts on the **anchor** camera (index 0). From the world-frame controls (top-left, foldable to a corner button) you can: * **Set frame on board**: pick a group where the target lies on the floor; the printed face of the target becomes the ground plane, and the origin goes to the marker's centre (single marker) or the board's first chessboard corner (ChArUco), so the cameras end up above the floor whatever the board type. The slider marks the framed group. * **Snap-rotate** the axes by ±90° about x, y or z. * **Minimize**: drop the worst 2.5 % of observations and re-fit (Caliscope's quality loop). The panel then shows the error before and after. Minimize keeps your framing; if it moves the framed target slightly off the origin, the pre-export checks report by how much. * **Align on reference**: see below. ## Keeping the room's frame across recalibrations[​](#keeping-the-rooms-frame-across-recalibrations "Direct link to Keeping the room's frame across recalibrations") Each calibration's world comes from the day's target, so a recalibration moves the origin, and anything placed in the old world (landmarks, a scene) no longer lines up. **Align on reference** re-aligns the solved rig on a previous calibration of the same room: 1. Load the previous calibration's `camera_array_opencv.json` (or the bare `{"cameras": [{"port", "R", "t"}]}` it grew from). 2. Read each mode's report: the rotation and shift it would apply, and each camera's residual, also when the fit is refused. * **Keep the floor** (default): a rotation about the vertical and a horizontal shift. The floor stays the one you framed; the tilt and height offset a full fit would add are reported, not applied. It needs a world framed on a level floor target. * **Rigid (6-DoF)**: a full rigid fit, with its angle uncertainty reported. It needs 3 cameras off one line. 3. Click **Align**. Nothing moves before that click. Cameras are matched by their USB path (by index only when one side has none, and then flagged). A fit is refused if it needs more than 20° of rotation, leaves more than 10 cm RMS of residual (another room, or a camera moved far), or implies a scale outside 0.8 to 1.25 (a units mismatch). The scale is reported, never applied. The reference stays with the session through recomputes, and the exported world records that it follows it. A camera moved by 20 to 30 cm is not reliably detected by the fit: after moving a camera, frame on the target again or use a fresh reference. ## Under the hood[​](#under-the-hood "Direct link to Under the hood") * Frames are timestamped from the camera driver's own buffer timestamps when they are plausible (otherwise the host clock) and grouped into synchronized instants, with a window just under one frame period, measured from each camera's mean recorded frame period. No hardware sync is needed. * Single-marker corners are refined from the marker's own black border, which removes the inward bias of OpenCV's contour refinement; ChArUco corners use the chessboard corners. * Pairwise relative poses via **`cv2.stereoCalibrate`** on each pair's shared views; a **co-visibility graph** links the cameras and poses are **chained from the anchor**. * A **bundle adjustment** (`scipy.optimize.least_squares`) jointly refines every non-anchor pose and the 3D points, a linear pass then a robust `soft_l1` pass, with **rigidity constraints** holding the reconstructed target to its printed geometry. The anchor stays fixed. → Explanation & sources: [Methodology](/docs/research/methodology.md) → See also: [Calibration best practices](/docs/reference/calibration-best-practices.md): shared views and multi-camera tips. --- # Intrinsic calibration Estimate each camera's focal length, principal point and distortion, one camera at a time, from the ChArUco board. ## The flow[​](#the-flow "Direct link to The flow") Intrinsics are computed **on demand**, from a recording, not in a continuous live loop. Each camera goes through four phases: 1. **Capture**: record a sweep while moving the board through the field of view. The live view shows the detected board, and the **live gauges** (marked *indicative*) show the board coverage so far (target ≥ 50 %) and whether the current frame is sharp enough. 2. **Prepare**: replay the recording and tune what the solver will use: **trim** the clip (start / end), set the **sampling stride** (1 frame every N, default 5) and the **keyframe cap** (default 50, from 6 to 100). 3. **Computing**: the service re-detects the board on the native recording, selects diverse **keyframes** and solves. 4. **Results**: inspect the estimated parameters and the quality figures below before moving to the next camera. ## Reading the results[​](#reading-the-results "Direct link to Reading the results") * **Reprojection error**: the RMS distance, in pixels at the output resolution, between the detected corners and where the solved model puts them. Useful, but a low value over poor coverage can still hide a bad calibration. * **Field-of-view coverage**: the share of the image covered by at least one keyframe's board. 70 % or more reads green; the outer ring of the image is hard to reach with a ChArUco board's interior corners, so 100 % is not expected. * **Board orientations**: how many of 8 tilt directions the keyframes span (4 or more reads green). * **Projection uncertainty (covered / never covered)**: how far the solved model could misplace a pixel's ray (1 σ, in output pixels), estimated from the solve itself. The first figure is where three or more keyframes covered the image, the second where none did, where the model extrapolates. The **Uncertainty** map shows it cell by cell: re-shoot any red zone you will rely on. Grey cells lie past the lens model's distortion fold, where no ray reaches the pixel, so no figure is given. * **Keyframes used** and their sharpness. ## Keyframe selection[​](#keyframe-selection "Direct link to Keyframe selection") The solve does not take every frame. After the structural gates (the board found, at least 6 corners, not a near-collinear sliver), the selection runs in two passes: 1. **Where**: farthest-point sampling over board **tilt** and **image position** places as many spots as the keyframe cap allows, spread over the poses you showed. 2. **Which**: every candidate frame goes to its nearest spot, and the **sharpest frame of each spot** is kept. Diversity decides how many keyframes and where; sharpness decides which frame to take at each spot. There is no absolute blur threshold, so a sweep in dim light still calibrates, and coverage is never traded for the crispest few frames (which cluster on the frontal, central hold). A low cap samples the spread of your sweep coarsely; the default of 50 is where identical cameras agree on their focal length. ## Under the hood[​](#under-the-hood "Direct link to Under the hood") realtime-calib follows Caliscope's intrinsic pipeline, adapted to current OpenCV: * ChArUco corners are detected with OpenCV's `CharucoDetector`, then refined with `cornerSubPix`. * The solve is **`cv2.calibrateCameraExtended`** with `CALIB_USE_INTRINSIC_GUESS` and no distortion-model flag: the classic **5-coefficient** model `[k1, k2, p1, p2, k3]`, with a free aspect ratio. * It starts from the same seed as Caliscope v0.11.5: focal length `max(width, height)`, principal point at the image centre. That seed does not depend on the order of the views, so the same keyframes always give the same result. OpenCV's own initialization only stands in if that solve fails. ## Recomputing after an extrinsic solve[​](#recomputing-after-an-extrinsic-solve "Direct link to Recomputing after an extrinsic solve") The extrinsic solve depends on every camera's intrinsics. Recomputing a camera's intrinsics once the array is solved therefore discards the extrinsic solve; the web app asks for confirmation first. The recorded sweep is kept, and the Extrinsic step then offers **"Recompute from the recorded sweep"**. → Explanation & sources: [Methodology](/docs/research/methodology.md) → See also: [Calibration best practices](/docs/reference/calibration-best-practices.md): how to capture a good intrinsic sweep. --- # Start or load a session Every calibration lives in a **session**: a folder on the server that holds its recordings, board config and results. The wizard stays **locked until a session exists**: open the web app, land on the **Dashboard** ("Welcome to the calibration bench"), and choose one of two entry modes. ## New realtime calibration[​](#new-realtime-calibration "Direct link to New realtime calibration") Start the full wizard from scratch, with live capture. You give the session a **folder name** (its id): the first character must be alphanumeric, then letters, digits, `.`, `_` or `-`, and it must be unique. The folder is created under the server's sessions directory (e.g. `sessions/mocap-2026-07-07`). The session opens at the first wizard step, **Target Config**. From there, everything you capture is recorded to that folder: each intrinsic capture and each extrinsic sweep is saved, so it can be replayed and recomputed later. ## Load from files[​](#load-from-files "Direct link to Load from files") Calibrate videos you already have, with no camera attached: **Load from files** uploads an archive (ZIP or tar, compressed or not) of pre-recorded videos, and the whole wizard then runs on the recordings. ``` my-session.zip ├── intrinsics/ required │ ├── cam_0.mp4 │ └── cam_1.mp4 └── extrinsics/ optional ├── cam_0.mp4 ├── cam_1.mp4 └── timestamps.csv optional ``` * Videos are named `cam_` (`.mp4`, `.mkv`, `.mov` or `.avi`; zero-padding is fine). The number is the camera index, `cam_0` is the anchor, and it must match across both folders. * One wrapper folder inside the archive is fine, and the singular folder names (`intrinsic/`, `extrinsic/`) of a Caliscope project are accepted too. * **Synchronization of the extrinsic videos.** A `timestamps.csv` in Caliscope's format (`cam_id,frame_time`) with real capture times is used for pairing. Without one, or with Caliscope's synthetic `inferred_timestamps.csv` (which is detected and ignored), the videos are aligned the way Caliscope aligns bare videos. * The videos are copied frame for frame into the session (remuxed, re-encoded only when OpenCV cannot read the original container), so frame *i* stays frame *i*, variable-frame-rate phone videos included. * In an imported session, Camera Setup is **read-only** and shows a frame from the middle of each recording instead of a live preview. Board definition, Prepare, compute, 3D review and export work as in a live session. * A failed import leaves no partial session behind, and the error names the file at fault. ## Resuming and switching sessions[​](#resuming-and-switching-sessions "Direct link to Resuming and switching sessions") The active session is held by the `calibration-service` and persisted on disk: the web app keeps no session state of its own, so you can close the browser, or switch from a laptop to a tablet, and pick up where you left off. Recent sessions are listed on the Dashboard; opening one switches the active session server-side and the wizard rail jumps straight to its persisted step. A **session checklist** lists what still needs attention, for example a session recorded with an earlier version that has no extrinsic board yet and asks you to revisit Target Config. Older session files are migrated transparently when they are opened. ## One operation at a time[​](#one-operation-at-a-time "Direct link to One operation at a time") The server runs **one long or mutating operation at a time**: a compute, a Minimize, a recording start, a session import, a camera or board change. If you trigger another one from a second device meanwhile, it is refused with a "busy" message rather than queued. While a sweep is recording, the camera configuration and the active session cannot change. --- # What is realtime-calib? **realtime-calib** is a **local, real-time, open-source** multi-camera calibration application. It recovers both the **intrinsics** (focal length, distortion) and the **extrinsics** (6-DoF position and orientation) of a rig of USB cameras, with live feedback, and exports calibration files ready for **Caliscope**, **OpenCV pipelines** and the major 3D engines. The whole calibration is driven from a web app, so you can run it from **any device on the same local network as the server the cameras are plugged into**: a laptop, a tablet or a phone. ## Why it exists[​](#why-it-exists "Direct link to Why it exists") realtime-calib grew out of using [Caliscope](https://github.com/mprib/caliscope) to calibrate multi-camera USB rigs for computer-vision projects. Caliscope does the calibration math well, but three recurring points of friction got in the way: * **Record-then-calibrate.** Each session meant pre-recording every camera in OBS through a dedicated sync plugin, which was unstable, crashed, and sometimes silently lost frame sync, forcing manual video re-editing before calibration could even start. * **No headless path.** Some clients ran on **headless Linux VMs**, so calibrating meant standing up an *extra* VM with a desktop, passing the cameras through, and driving a GUI, just to run a calibration. * **Export mismatch.** The output convention didn't always match the target project's coordinate system, so results needed hand-conversion. realtime-calib keeps what works about Caliscope (its calibration logic) and removes those three frictions. ## What makes it different[​](#what-makes-it-different "Direct link to What makes it different") * **One-pass calibration.** No separate video-recording step: capture, detection, quality feedback and computation happen live, in a single flow. Every sweep is still recorded to the session folder, so it can be replayed and recomputed later. * **Runs headless, driven from a browser.** The service runs in Docker on the machine the cameras are plugged into, with **no desktop or GUI required on that host**. The operator drives everything from a responsive web app served over the local network, on a **laptop, tablet or phone**. This fits headless servers, robotics rigs, motion-capture setups and production lines. * **Quality you can read.** Live coverage overlays while you capture; after the solve, the reprojection error per camera, the **rigidity of the reconstructed target** in millimetres, the intrinsic **projection uncertainty** across the image, and a set of **pre-export checks** (epipolar consistency, cameras above the floor, and, with a site template, cabling and scale against a tape measure). * **Multi-format export.** One calibration, exported to the convention your target actually uses: Caliscope-native TOML, the aniposelib TOML read by Pose2Sim and anipose, a versioned **OpenCV JSON**, or engine-ready JSON with the correct axis and handedness for **three.js / OpenGL** (Y-up, right-handed), **Blender / ROS** (Z-up, right-handed), **Unity** (Y-up, left-handed) and **Unreal** (Z-up, left-handed). The dangerous 3D math (axis remap, left-handed mirror) is done for you. * **Local & private, CPU-only.** Everything runs on your own hardware: no cloud, no GPU. Camera streams never leave the local network. * **Caliscope-compatible.** The `camera_array.toml` export is the layout Caliscope itself loads, and recordings made for Caliscope can be imported and calibrated here. ## The workflow[​](#the-workflow "Direct link to The workflow") An operator starts the stack (Docker), opens the web app, and either starts a **new realtime calibration** or **loads pre-recorded videos** from an archive. A wizard then walks through: 1. **Target Config**: define the board(s), ChArUco and/or a single ArUco marker. 2. **Camera Setup**: discover the USB cameras, set the capture mode and their order. 3. **Intrinsics**: calibrate each camera, one by one. 4. **Extrinsics**: solve the whole capture volume, with a **3D review** of the reconstructed rig as its final sub-step. 5. **Export**: read the pre-export checks, then download the files for your targets. Everything happens in real time: camera streams are published over **LiveKit**, and detection overlays are burned in server-side and streamed live. ## Who is this for?[​](#who-is-this-for "Direct link to Who is this for?") * **Operators & users** setting up a multi-camera rig for motion capture, volumetric capture, robotics or photogrammetry. * **Researchers** who need reproducible, inspectable calibration and a path to cite the method: see [Research](/docs/research/methodology.md). ## How it relates to Caliscope[​](#how-it-relates-to-caliscope "Direct link to How it relates to Caliscope") realtime-calib **reimplements** Caliscope's calibration logic (it is not a dependency) to make it real-time and single-pass, while staying **output-compatible**. Caliscope remains the conceptual reference for board definitions, the intrinsic solve, pairwise stereo initialization and bundle adjustment. For a side-by-side comparison (workflow, interface, exports, licensing), see [realtime-calib vs Caliscope](/docs/realtime-calib-vs-caliscope.md). Next step Head to [Installation](/docs/getting-started/installation.md), then run your [first calibration](/docs/getting-started/quickstart.md). --- # License realtime-calib is **free and open source software** licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0)**. ## What AGPL-3.0 means for you[​](#what-agpl-30-means-for-you "Direct link to What AGPL-3.0 means for you") * You can **use, study, modify and share** realtime-calib freely. * If you **distribute** it or **offer it over a network** (e.g. as a service), you must make your **complete corresponding source code** — including your modifications — available to users **under the AGPL-3.0**. The full license text is in the [`LICENSE`](https://github.com/hans-brgs/realtime-calib/blob/main/LICENSE) file at the root of the repository. ## Commercial use[​](#commercial-use "Direct link to Commercial use") The AGPL's network copyleft is not compatible with keeping your own code proprietary. If you want to: * embed realtime-calib in a **closed-source / proprietary** product, or * offer it as a **hosted service without** releasing your source, or * get realtime-calib **adapted to your needs** (custom development), a **commercial license** and **custom development** are available. This is a classic dual-licensing model (as used by MongoDB, Grafana, Nextcloud and others): the same code is AGPL for everyone, and a separate commercial license removes the copyleft obligations for paying users. 📧 **Get in touch:** Not legal advice This page summarizes intent. The AGPL-3.0 `LICENSE` file and any commercial agreement are the binding documents. --- # realtime-calib vs Caliscope [Caliscope](https://github.com/mprib/caliscope) (BSD-2-Clause) is the open-source multi-camera calibration tool this project owes the most to: **realtime-calib reimplements Caliscope's calibration logic** (the same board definitions, the same OpenCV solve lineage, the same bundle-adjustment shape) and stays **output-compatible** with it. If you are looking for a *Caliscope alternative*, this page lays out honestly how the two differ and when each one is the better fit. Statements about Caliscope refer to its v0.11.5 release. ## TL;DR[​](#tldr "Direct link to TL;DR") * Pick **Caliscope** if you want a **desktop application** that also carries a basic **3D reconstruction pipeline** (ONNX pose estimation such as RTMPose, SLEAP or DeepLabCut, and 3D trajectories as CSV/TRC), and calibrating from **pre-recorded, synchronized videos** fits your workflow. * Pick **realtime-calib** if you want to **calibrate live, in one pass**, from a browser on **any device**, against cameras plugged into a possibly **headless Linux server**, and you need exports in **engine conventions** (Unity, Unreal, Blender, three.js, ROS) or a ready-to-load **OpenCV** file on top of Caliscope's own formats. ## Side by side[​](#side-by-side "Direct link to Side by side") | | Caliscope | realtime-calib | | -------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Workflow | **Record first, then calibrate** from synchronized footage | **One pass, live**: capture, detection, quality feedback and compute in a single flow; pre-recorded videos can also be imported | | Interface | Desktop GUI (Python application) | **Web app** on the local network: desktop, tablet or phone | | Headless camera host | Not a supported path | **Yes**: Docker on a headless Linux server, no desktop needed | | Camera host OS | Wherever the Python desktop app runs | **Linux** (cameras read via V4L2), via Docker Compose | | Live feedback | Offline review of recordings | Detection overlays and coverage gauges, streamed live | | Beyond calibration | Basic reconstruction pipeline: ONNX pose estimation, 3D trajectories (CSV / TRC) | **Calibration only**, by design | | Exports | Its `camera_array.toml` + an aniposelib export (anipose, Pose2Sim) | The **same two files** + a versioned OpenCV JSON + engine JSON for three.js, Blender / ROS, Unity, Unreal | | Calibration math | ChArUco intrinsics, pairwise stereo extrinsics, bundle adjustment with target rigidity constraints | **Same lineage, reimplemented** for a real-time, single-pass flow | | GPU | Calibration is CPU-based | Not required: CPU-only | | License | BSD-2-Clause | AGPL-3.0, with a [commercial option](/docs/open-source/license.md#commercial-use) | ## When Caliscope is the better fit[​](#when-caliscope-is-the-better-fit "Direct link to When Caliscope is the better fit") * You want calibration **and** a first 3D reconstruction (landmark triangulation from pose-estimation models) in one desktop tool. * Your cameras are attached to a **desktop machine you sit at**, and a record-then-process workflow suits you. * You prefer a permissive **BSD-2-Clause** license. ## When realtime-calib is the better fit[​](#when-realtime-calib-is-the-better-fit "Direct link to When realtime-calib is the better fit") * Your cameras are plugged into a **headless server** (robotics rig, motion-capture room, production line) and you want to drive the calibration from a **phone, tablet or laptop** on the same network. * You want **live feedback while you capture** (coverage overlays and quality gauges) instead of discovering problems after recording, and a quality report built for a hand-over: per-camera error, target rigidity in millimetres, an intrinsic projection-uncertainty map and pre-export checks. * You recalibrate the same room regularly: a site template checks each new calibration's cabling, placement and scale, and a new calibration can be aligned on the previous one so the room's frame does not move. * Your calibration must land in a **game engine, a 3D tool or an OpenCV pipeline**: the exports carry the correct axis conventions and handedness for Unity, Unreal, Blender, three.js and ROS, so no hand-conversion of the 3D math. ## Compatibility between the two[​](#compatibility-between-the-two "Direct link to Compatibility between the two") * **Caliscope → realtime-calib.** A Caliscope recording (its intrinsic and extrinsic videos, with or without its timestamps file) can be [imported](/docs/guides/start-or-load-session.md#load-from-files) and calibrated here. Videos without real timestamps are synchronized exactly the way Caliscope aligns bare videos. * **realtime-calib → Caliscope.** `camera_array.toml` is written in the layout Caliscope v0.11.5 loads (`[cameras.N]` tables, translations in metres). Project-specific fields are strictly additive, so Caliscope ignores them. For an older Caliscope project (v0.5.4 and earlier), the aniposelib file exported in metres pastes into its `config.toml`. * Files exported by realtime-calib before this layout used top-level `[cam_N]` tables, which recent Caliscope versions read as an empty camera array: export them again. See the [calibration output files reference](/docs/reference/output-calibration-files.md) for every field. The calibration methodology itself (what is solved, with which OpenCV calls, and how the bundle adjustment is set up) is documented with its sources in [Methodology](/docs/research/methodology.md). --- # Calibration best practices How to capture data that yields an accurate calibration. ## How solid is this advice?[​](#how-solid-is-this-advice "Direct link to How solid is this advice?") I tried to find a **research article or meta-analysis dedicated to which capture and board parameters maximise camera-calibration accuracy** — one that would test them and publish validated ranges. I did not find a study specifically on that question. Some peer-reviewed work does give **partial direction**: Zhang (2000) shows that board **orientation** matters (near-frontal is degenerate, \~45° is best *in simulation*), and Muñoz-Salinas et al. (2018) show that **camera-to-marker distance / apparent size** matters (accuracy degrades and the pose becomes ambiguous as the marker shrinks in the image). But neither — nor any source I found — gives **ranges of values that maximise calibration**. That operational specificity — how many images, what board size, which dictionary, how much tilt — exists only on **practitioner and vendor sites (OpenCV, MATLAB, calib.io, OKLAB) that state the numbers without citing or justifying a primary source.** So read them as **empirical rules-of-thumb, not evidence-based constants**: recommendations that genuinely trace to primary literature are cited; the rest are marked with an asterisk (\*) — see the note at the end. ## At a glance[​](#at-a-glance "Direct link to At a glance") | Parameter | Typical value | Basis | | ----------------------- | -------------------------------------------------------------------------------- | ----------------------------------- | | Board type | **ChArUco** | OpenCV docs | | Grid (squares) | 5×7 … 8×11\* — wide-angle 9×7+\* | OKLAB\* | | Square size | 15–50 mm\* (larger = longer working distance) | OKLAB\* | | Marker/square ratio | 0.6–0.75\* (realtime-calib default 0.75) | OKLAB\* | | Dictionary | **smallest** that supplies the markers you need (e.g. `DICT_5X5_100`) | Garrido-Jurado 2014 | | Board fills | 30–70 % of the frame\* | OKLAB\* | | Min. marker/square size | ≥ 8–12 px\* | OKLAB\* | | Tilt | up to \~45–60°\* (≥ 2 non-parallel orientations) | Zhang 2000 · OKLAB\* | | Number of images | \~15–25\* (theoretical min 3) | OKLAB\*, MATLAB\* | | Lighting | 300–1000 lux\*, even, no glare | OKLAB\* | | Substrate | rigid & flat (aluminium / glass) | OKLAB\* | | Distortion model | classic 5-coefficient `[k1,k2,p1,p2,k3]` — rational 8 only for strong wide-angle | Zhang 2000 · Sun & Cooperstock 2006 | | Reprojection error | < 0.3 px good\* · 0.3–1.0 acceptable\* · > 1.0 investigate\* | OKLAB\* | ## Choosing a board[​](#choosing-a-board "Direct link to Choosing a board") * **Use a ChArUco board for calibration.** OpenCV explicitly recommends it: the ArUco markers identify each corner (no rotation ambiguity, tolerant of occlusion and partial views), while the interpolated **chessboard corners** give subpixel accuracy. \[OpenCV ChArUco docs; Garrido-Jurado et al. 2014; Romero-Ramirez et al. 2018] * **Pick the smallest ArUco dictionary that supplies the markers you need.** A smaller dictionary allows a larger minimum inter-marker (Hamming) distance and stronger error correction — `floor((d−1)/2)` correctable bits for minimum distance *d* — which lowers false detections. \[Garrido-Jurado et al. 2014] * **Match geometry to the job.\*** Bigger squares for longer working distances; more squares (denser grid) for wide-angle/fisheye lenses that need edge data; keep the marker at \~60–75 % of the square. realtime-calib defaults to a **7 × 9** grid at **0.75** ratio, with the `DICT_4X4_100` dictionary. \[OKLAB\*] * **Print sharp, mount flat and rigid, then measure the real scale.** Warped or taped-on paper introduces systematic error — use a rigid, flat substrate. Print at true size and **verify the square edge with calipers**; that measurement, not the nominal size, sets the metric scale. \[OKLAB\*] ## Capture strategy (this matters most)[​](#capture-strategy-this-matters-most "Direct link to Capture strategy (this matters most)") * **Show the board at several orientations.** Planar calibration needs **≥ 2 non-parallel orientations**, and **≥ 3 views** for a unique solution of all five intrinsics (two views only work if skew is fixed to zero). \[Zhang 2000] * **Use enough, varied views — about 15–25\*.** Accuracy improves with more views, the biggest gain from **2 → 3**, then diminishing returns. What matters is *diversity*, not count: 20 varied views beat 50 similar ones. \[Zhang 2000 for the 2→3 gain; OKLAB\*, MATLAB\*] With a continuous sweep, realtime-calib keeps up to 50 keyframes by default (spread over tilt and image position): on identical cameras, 25 still left the focal length dependent on which frames were picked. * **Tilt the board — roughly 45°, up to \~60°.** Near-frontal boards (\~5°) are a **degenerate configuration**. \[Zhang 2000; OKLAB\*] * **Cover the whole frame, corners and edges included.** Distortion is strongest at the periphery, so across your views the board must reach every corner. \[OKLAB\*, MATLAB\*] * **Fix the camera and the settings.** Mount it rigidly; use **manual, fixed exposure and focus** (auto settings drift between shots); light evenly and avoid glare. \[OKLAB\*] * **Avoid motion blur.** Blurred corners lose subpixel accuracy — keep the board (or camera) still at each capture. \[OKLAB\*] * **(Advanced) Guided / next-best-pose capture** — choosing each pose to minimise the parameter-covariance trace reaches higher accuracy with fewer images than random capture. \[Tan et al. 2025; Peng & Sturm 2019; Rojtberg & Kuijper 2018] About the Zhang numbers The **≥ 3 views**, the **2 → 3 gain**, the **\~45° optimum** and the **\~5° degenerate** threshold come from Zhang's 2000 experiment — a **synthetic Monte-Carlo simulation** (3 images, Gaussian corner noise σ = 0.5 px) on a **plain checkerboard, not a ChArUco board** — which explicitly does **not** model the foreshortening that degrades real corner detection at large tilt. Treat them as well-founded directions, not exact ChArUco constants. ## Working distance & apparent size[​](#working-distance--apparent-size "Direct link to Working distance & apparent size") Keep the board **large in the image** — working distance is relative to board size. As a marker's *apparent* size (pixels) shrinks, corner error grows and a **planar-pose ambiguity** appears: four coplanar points admit two poses related by a reflection about the camera's line of sight — worst for small or distant planes, planes far relative to the focal length, and wide-angle lenses at close range. Solvers return both candidates with their reprojection errors, but when the two are close the choice is unsafe. A **ChArUco board's many corners, solved jointly, mitigate this**; markers should stay above **\~8–12 px\***, and repeatability drops with distance (most in depth). \[Muñoz-Salinas et al. 2018; Collins & Bartoli 2014 (IPPE); Aliani et al. 2026; OKLAB\*] ## Distortion model[​](#distortion-model "Direct link to Distortion model") realtime-calib estimates the **classic 5-coefficient Brown–Conrady model** (`k1, k2, p1, p2, k3`) — what OpenCV computes when `calibrateCamera` is called without any model flag, and exactly what Caliscope does. Unusually for this page, this choice *is* well grounded in primary literature: * **Radial distortion dominates — and its first term dominates the rest.** Zhang calibrates with `k1, k2` only, noting that "any more elaborated modeling not only would not help … but also would cause numerical instability" — a conclusion he takes from Tsai (who, for industrial lenses, recommends a *single* radial term) and Wei & Ma. \[Zhang 2000; Tsai 1987; Wei & Ma 1994] * **Higher-order terms are a risk, not free accuracy.** Empirically, the r⁶ term (`k3`) does not improve accuracy and can corrupt the lower-order estimates when views are few or noisy; the radial coefficients are highly correlated, and superfluous parameters weaken the solution (over-parameterisation). Unconstrained polynomial fits can even turn non-monotonic — an outcome OpenCV's own documentation calls "a calibration failure". \[Sun & Cooperstock 2006; Remondino & Fraser 2006; Heller et al. 2014; OpenCV calib3d docs] * **Tangential terms (`p1, p2`) are cheap insurance.** Decentering distortion runs an order of magnitude below radial, but modelling it measurably helps — most on wide-angle lenses — and "increases the likelihood of accurate calibration" for a camera whose distortion is unknown. \[Weng et al. 1992; Sun & Cooperstock 2006; Remondino & Fraser 2006] * **The rational 8-coefficient model (`CALIB_RATIONAL_MODEL`) targets strong wide-angle distortion.** Rational/division models were introduced for extreme (wide-angle, catadioptric) optics; at low distortion the model families perform about the same, so the extra denominator terms buy nothing for a typical webcam — while OpenCV bug reports document rational solves whose denominator misbehaves inside the image. \[Claus & Fitzgibbon 2005; Ricolfe-Viala & Sánchez-Salmerón 2010] We found **no peer-reviewed head-to-head** of OpenCV's 5- vs 8-coefficient variants specifically — but our own data agrees with the model-family literature: on a real 4-camera dataset, refitting the same captures with the rational model left the reprojection RMSE unchanged (0.30–0.32 px) while producing wildly inconsistent coefficients across identical cameras (`k1` from −0.33 to −4.0); the classic 5 recovers a consistent `k1 ≈ −0.375` everywhere. One counterpoint if you need the last drop of accuracy: *all* low-order parametric models are approximations, and generic per-pixel camera models measurably beat them on depth and pose bias \[Schöps et al. 2020] — a different tool class, outside realtime-calib's scope. ## Evaluating results[​](#evaluating-results "Direct link to Evaluating results") * **Read the reprojection error, but don't game it.** As a rough guide, **< 0.3 px is good, 0.3–1.0 px acceptable, > 1.0 px worth investigating\*** (flatness, motion blur, bad corners). But a *lower* per-view error from a Zhang-style decoupled fit is **misleading** — each board pose gets its own free extrinsics, paid for with extrinsic-parameter uncertainty. Coverage and a shared-parameter solution beat a small RMSE. \[OKLAB\* for the thresholds; Petković et al. 2024 for the pitfall] * **Validate on something known.** Undistort a test image — straight lines should be straight — and measure a known dimension against ground truth. \[OKLAB\*] ## Multi-camera extrinsics[​](#multi-camera-extrinsics "Direct link to Multi-camera extrinsics") * **Finish with a global bundle adjustment, one pose fixed as the anchor.** Extrinsic parameters number `6(N+K−1)` for *N* cameras and *K* board positions once one frame is fixed to remove gauge freedom; the final step jointly minimises reprojection error across all cameras. \[Petković et al. 2024] * **Give each camera pair enough shared views**, and let ChArUco's unique corner IDs keep every camera referencing the same points. \[Heng et al. 2013; OpenCV ChArUco docs] * **For a large volume, a single ArUco marker can link the cameras** where a printable ChArUco board's squares would be too small to detect from across the room. Print it large, keep its white margin, mount it flat, and measure its side with calipers: that measurement sets the scale of the whole rig. * **Move the target steadily, not fast.** Without hardware sync, the cameras do not expose at the same instant; a target moving fast is seen at different places by the cameras of one group. realtime-calib sets such groups aside, but a slow sweep keeps more of them. * **Check the scale against something you measured.** Every quality figure computed from the solve itself passes with a mis-measured target, because the whole rig scales together. A few camera-to-camera distances measured with a tape, in a site template, are the external check. * **Read the target rigidity next to the reprojection error.** A solve can lower its reprojection error by bending the reconstructed target; the rigidity in millimetres, measured against the printed geometry, cannot be gamed that way. realtime-calib implements this pipeline — see [Methodology](/docs/research/methodology.md). ## Capture checklist[​](#capture-checklist "Direct link to Capture checklist") * ☐ Board rigid and flat; **real square size measured with calipers**. * ☐ Camera mounted rigidly; **manual fixed exposure and focus**. * ☐ Even lighting (\~300–1000 lux\*), no glare or reflections. * ☐ Board fills **30–70 %** of the frame\*; markers ≥ **\~8–12 px\***. * ☐ **\~15–25 varied views\***: some filling the frame, varied depths, strong tilts, all four corners covered, a few partial views. * ☐ No motion blur — hold still at each capture. * ☐ Reprojection error sub-pixel; investigate if **> 1 px\***. * ☐ Multi-camera: enough shared views per pair, a steady sweep; read the per-camera error and the target rigidity, not only the overall RMSE. * ☐ Scale checked against at least one tape-measured distance. ## Sources[​](#sources "Direct link to Sources") **Primary literature** * Zhang, Z. (2000). *A Flexible New Technique for Camera Calibration.* IEEE TPAMI 22(11) — [full text](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr98-71.pdf). * Brown, D.C. (1971). *Close-Range Camera Calibration.* Photogrammetric Engineering 37(8):855–866 — origin of the Brown–Conrady distortion model (no open copy found). * Tsai, R.Y. (1987). *A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses.* IEEE J. Robotics and Automation 3(4):323–344 — doi:[10.1109/JRA.1987.1087109](https://doi.org/10.1109/JRA.1987.1087109). * Weng, J., Cohen, P., Herniou, M. (1992). *Camera Calibration with Distortion Models and Accuracy Evaluation.* IEEE TPAMI 14(10):965–980 — doi:[10.1109/34.159901](https://doi.org/10.1109/34.159901). * Wei, G.-Q., Ma, S.D. (1994). *Implicit and explicit camera calibration: Theory and experiments.* IEEE TPAMI 16(5):469–480. * Claus, D., Fitzgibbon, A.W. (2005). *A Rational Function Lens Distortion Model for General Cameras.* CVPR 2005 — [full text](https://www.robots.ox.ac.uk/~vgg/publications/2005/Claus05a/claus05a.pdf). * Sun, W., Cooperstock, J.R. (2006). *An empirical evaluation of factors influencing camera calibration accuracy using three publicly available techniques.* Machine Vision and Applications 17(1):51–67 — doi:[10.1007/s00138-006-0014-6](https://doi.org/10.1007/s00138-006-0014-6) · open WACV 2005 companion: [full text](https://srl.mcgill.ca/publications/papers/2005-WACV-Sun.pdf). * Remondino, F., Fraser, C. (2006). *Digital camera calibration methods: considerations and comparisons.* ISPRS Archives XXXVI-5:266–272 — [full text](https://www.isprs.org/proceedings/xxxvi/part5/paper/remo_616.pdf). * Ricolfe-Viala, C., Sánchez-Salmerón, A.-J. (2010). *Lens distortion models evaluation.* Applied Optics 49(30):5914–5928 — doi:[10.1364/AO.49.005914](https://doi.org/10.1364/AO.49.005914). * Heller, J., Henrion, D., Pajdla, T. (2014). *Stable radial distortion calibration by polynomial matrix inequalities programming.* ACCV 2014 — [arXiv:1409.5753](https://arxiv.org/abs/1409.5753). * Schöps, T., Larsson, V., Pollefeys, M., Sattler, T. (2020). *Why Having 10,000 Parameters in Your Camera Model Is Better Than Twelve.* CVPR 2020 — [open access](https://openaccess.thecvf.com/content_CVPR_2020/papers/Schops_Why_Having_10000_Parameters_in_Your_Camera_Model_Is_Better_CVPR_2020_paper.pdf). * Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marín-Jiménez, M.J. (2014). *Automatic generation and detection of highly reliable fiducial markers under occlusion.* Pattern Recognition 47(6):2280–2292 — doi:[10.1016/j.patcog.2014.01.005](https://doi.org/10.1016/j.patcog.2014.01.005). * Romero-Ramirez, F.J., Muñoz-Salinas, R., Medina-Carnicer, R. (2018). *Speeded up detection of squared fiducial markers.* Image and Vision Computing 76:38–47 — doi:[10.1016/j.imavis.2018.05.004](https://doi.org/10.1016/j.imavis.2018.05.004). * Muñoz-Salinas, R., Marín-Jiménez, M.J., Yeguas-Bolívar, E., Medina-Carnicer, R. (2018). *Mapping and localization from planar markers.* Pattern Recognition 73:158–171 — doi:[10.1016/j.patcog.2017.08.010](https://doi.org/10.1016/j.patcog.2017.08.010). * Collins, T., Bartoli, A. (2014). *Infinitesimal Plane-Based Pose Estimation (IPPE).* IJCV 109:252–286 — [project](https://github.com/tobycollins/IPPE). * Petković, T. et al. (2024). *Multi-camera/projector calibration analysis.* arXiv:2410.18511 — [link](https://arxiv.org/abs/2410.18511). * Heng, L., Li, B., Pollefeys, M. (2013). *CamOdoCal.* IEEE/RSJ IROS 2013 — [link](https://people.inf.ethz.ch/pomarc/pubs/HengIROS13.pdf). * Aliani, C., Lorenzetto Bologna, C., Francia, P., Bocchi, L. (2026). *Optimising Camera–ChArUco Geometry for Motion Compensation in Standing Equine CT.* Sensors 26(4):1310 — doi:[10.3390/s26041310](https://doi.org/10.3390/s26041310). * Tan et al. (2025). *Next-best-pose extrinsic calibration.* arXiv:2511.18317 — [link](https://arxiv.org/abs/2511.18317). **Practitioner / vendor references** (empirical, no primary sources cited — asterisked values) * OpenCV — [ChArUco calibration](https://docs.opencv.org/4.x/da/d13/tutorial_aruco_calibration.html) · [ChArUco detection](https://docs.opencv.org/4.x/df/d4a/tutorial_charuco_detection.html) · [calib3d module](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html) (distortion model, flags, monotonicity warning). * calib.io — [Calibration Best Practices](https://calib.io/blogs/knowledge-base/calibration-best-practices). * MATLAB — [Prepare camera and capture images](https://www.mathworks.com/help/vision/ug/prepare-camera-and-capture-images-for-camera-calibration.html). * OKLAB — [ChArUco Calibration Boards: Complete Guide](https://www.oklab.com/blog/charuco-calibration-boards-complete-guide-to-professional-camera-calibration). On asterisked (\*) values Asterisked values come from practitioner or vendor sites (OpenCV, MATLAB, calib.io, OKLAB) that state them **without citing a primary peer-reviewed source**. We could not trace them to primary literature, so treat them as **empirical rules-of-thumb**, not evidence-based constants. --- # Camera calibration glossary Short, plain-language definitions of the terms used across this documentation — each with a pointer to the page where it matters. ## 6-DoF pose[​](#6-dof-pose "Direct link to 6-DoF pose") The position (three translations) and orientation (three rotations) of a rigid body in space — six degrees of freedom in total. Extrinsic calibration recovers one 6-DoF pose per camera. *See: [Extrinsic calibration](/docs/guides/extrinsic-calibration.md).* ## Anchor camera[​](#anchor-camera "Direct link to Anchor camera") The camera whose pose is **fixed as the origin** of the shared coordinate frame. Fixing one camera removes the gauge freedom (the whole rig could otherwise translate/rotate freely without changing the reprojection error). In realtime-calib the anchor is the first camera in the Camera Setup order (index 0), and the world frame can later be rebased onto a board. *See: [Configure cameras](/docs/guides/configure-cameras.md).* ## ArUco marker & dictionary[​](#aruco-marker--dictionary "Direct link to ArUco marker & dictionary") An **ArUco marker** is a square fiducial with a binary pattern encoding an ID. A **dictionary** is the family of valid marker patterns (e.g. `DICT_5X5_100`); smaller dictionaries keep more distance between patterns, which lowers false detections. *See: [Define a calibration board](/docs/guides/calibration-board.md).* ## Bundle adjustment[​](#bundle-adjustment "Direct link to Bundle adjustment") The final joint optimization: a non-linear least-squares solve that refines **all camera poses and 3D points together** by minimizing the total reprojection error. realtime-calib runs it with `scipy.optimize.least_squares` (trf, sparse Jacobian), a linear pass then a robust `soft_l1` pass, keeping the anchor fixed and holding the reconstructed target to its printed geometry (see *Board rigidity*). *See: [Methodology](/docs/research/methodology.md).* ## Board rigidity[​](#board-rigidity "Direct link to Board rigidity") How far the reconstructed calibration target deviates from its printed geometry, as an RMS of corner-to-corner distances in millimetres. The bundle adjustment is constrained to keep the target rigid; the reported rigidity is an independent judge of the extrinsic solve, since a solve can lower its reprojection error by bending the board but not this number. *See: [Extrinsic calibration](/docs/guides/extrinsic-calibration.md).* ## Camera extrinsics[​](#camera-extrinsics "Direct link to Camera extrinsics") Where a camera **is**: the rotation and translation relating the camera to the world (or to another camera). Extrinsics change whenever the camera moves; they are solved per rig, not per camera. *See: [Extrinsic calibration](/docs/guides/extrinsic-calibration.md).* ## Camera intrinsics[​](#camera-intrinsics "Direct link to Camera intrinsics") How a camera **projects**: focal length and principal point (the camera matrix **K**), estimated together with lens distortion. Intrinsics belong to the camera + lens + resolution combination and are independent of where the camera stands. *See: [Intrinsic calibration](/docs/guides/intrinsic-calibration.md).* ## ChArUco board[​](#charuco-board "Direct link to ChArUco board") A hybrid calibration target: a chessboard whose white squares carry ArUco markers. The markers **identify** each corner (robust to occlusion and partial views, no rotation ambiguity) while the chessboard corners provide **subpixel accuracy**. The recommended board type in realtime-calib. *See: [Define a calibration board](/docs/guides/calibration-board.md).* ## Co-visibility graph[​](#co-visibility-graph "Direct link to Co-visibility graph") A graph whose nodes are cameras and whose edges connect pairs that observed the board **at the same instants** often enough. Extrinsic chaining walks this graph from the anchor along the lowest-accumulated-error path. *See: [Methodology](/docs/research/methodology.md).* ## Coordinate convention[​](#coordinate-convention "Direct link to Coordinate convention") The combination of **up axis** (Y or Z) and **handedness** (left or right) that defines a target's world frame — e.g. Unity is Y-up left-handed, Blender and ROS are Z-up right-handed. Exports remap axes per target so you don't do that 3D math by hand. *See: [Calibration output files](/docs/reference/output-calibration-files.md).* ## Keyframe[​](#keyframe "Direct link to Keyframe") One of the small, deliberately **diverse** subset of captured detections that the intrinsic solver actually uses. Selection combines a sampling stride, structural gates (corner count, spread), farthest-point sampling over board tilt and image position to decide *where* keyframes are wanted, and sharpness to pick the best frame at each spot. *See: [Intrinsic calibration](/docs/guides/intrinsic-calibration.md).* ## Lens distortion[​](#lens-distortion "Direct link to Lens distortion") The deviation of a real lens from the ideal pinhole model — straight lines bowing (radial distortion) or shifting (tangential). realtime-calib estimates the classic **5-coefficient model** (`k1, k2, p1, p2, k3`) used by Caliscope. *See: [Intrinsic calibration](/docs/guides/intrinsic-calibration.md).* ## Metric scale[​](#metric-scale "Direct link to Metric scale") What turns a relative geometry into real-world units. It comes from **measuring the printed extrinsic target** (a square edge or the marker side, with calipers), not from the nominal print size, since printers rescale. A wrong measurement rescales the whole rig uniformly; only an external reference, such as tape-measured distances in a site template, can reveal it. *See: [Define a calibration board](/docs/guides/calibration-board.md).* ## Output resolution[​](#output-resolution "Direct link to Output resolution") The image size the exported calibration refers to: the native capture resolution times the **resize factor** chosen in Camera Setup. Calibration always runs on native frames; the camera matrix is then mapped to the output resolution the way `cv2.resize` maps pixels, and every reported error is in pixels at that resolution. *See: [Configure cameras](/docs/guides/configure-cameras.md).* ## Projection uncertainty[​](#projection-uncertainty "Direct link to Projection uncertainty") How far a solved intrinsic model could misplace the ray of a pixel (1 σ, in pixels), computed from the solve's own covariance and mapped across the image. It is low where the board went and grows where it never did, where the model extrapolates. *See: [Intrinsic calibration](/docs/guides/intrinsic-calibration.md).* ## Reprojection error[​](#reprojection-error "Direct link to Reprojection error") The distance, in pixels, between a detected board corner and where the calibrated model **re-projects** it. Usually reported as an RMS across detections, at the output resolution in realtime-calib. Lower is better, but a low number with poor frame coverage can still hide a bad calibration. *See: [Calibration best practices](/docs/reference/calibration-best-practices.md).* ## Session[​](#session "Direct link to Session") The folder on the server that holds everything about one calibration run — recordings, board config, results. It is the source of truth: the web app holds no durable state and rehydrates from it. *See: [Start or load a session](/docs/guides/start-or-load-session.md).* ## Site template[​](#site-template "Direct link to Site template") A JSON description of a calibration room, written by hand: which camera (by USB path) is expected at which port, bounds on each camera's position and orientation, and camera-to-camera distances measured with a tape. It adds external checks before export, the only ones that can catch a wrong scale or a swapped cable. *See: [Export](/docs/guides/export.md#site-template).* ## Stereo calibration[​](#stereo-calibration "Direct link to Stereo calibration") Solving the **relative pose of a camera pair** from views of the board that both cameras share (`cv2.stereoCalibrate`). realtime-calib uses it to initialize every co-visible pair before chaining and bundle adjustment. *See: [Methodology](/docs/research/methodology.md).* ## Triangulation[​](#triangulation "Direct link to Triangulation") Recovering a **3D point** by intersecting the rays from several cameras that observed it. realtime-calib triangulates board corners (DLT over all observing rays) to build the 3D point cloud that the bundle adjustment refines. *See: [Methodology](/docs/research/methodology.md).* --- # Calibration output files When you export, realtime-calib writes **one file per selected target**. Every target carries the same calibration — the same intrinsics and 6-DoF poses — in a different shape and coordinate convention. Everything lands in the session folder, which is the source of truth for a run. ## Files written[​](#files-written "Direct link to Files written") | File | Format | For | | ------------------------------ | ----------------------- | ---------------------------------------------------------------------------------- | | `camera_array.toml` | Caliscope-native TOML | Caliscope (v0.7 and later) | | `camera_array_aniposelib.toml` | aniposelib TOML | Pose2Sim, anipose, and Caliscope ≤ 0.5.4 projects | | `camera_array_opencv.json` | Versioned JSON contract | OpenCV pipelines | | `camera_array_threejs.json` | Engine JSON | three.js / OpenGL | | `camera_array_blender.json` | Engine JSON | Blender / ROS | | `camera_array_unity.json` | Engine JSON | Unity | | `camera_array_unreal.json` | Engine JSON | Unreal | | `checks.json` | JSON | The [pre-export checks](/docs/guides/export.md#pre-export-checks), always included | You pick which targets to export (any subset) and the length unit (mm or m). The unit applies to the aniposelib TOML and the engine JSON files; `camera_array.toml` and `camera_array_opencv.json` are always in metres. The download is one archive, `calibration_export.zip`, holding only the current selection. The **session folder** also holds the recordings, board config and computed results. In every file, the intrinsics refer to the **output resolution** (native × resize factor), with **K** mapped at pixel centres the way `cv2.resize` maps them (see [Configure cameras](/docs/guides/configure-cameras.md#resolution-vs-resize-factor)), and every error is in pixels at that resolution. ## Coordinate conventions[​](#coordinate-conventions "Direct link to Coordinate conventions") Each engine JSON is written in its target's world convention — a combination of **up axis** (Y or Z) and **handedness** (left or right): ![Y-up vs Z-up crossed with left- vs right-handed axis triads](/assets/images/coordinate_conventions-03619376955419f367478266fcddf41b.png) **Coordinate conventions.** The four world conventions realtime-calib exports to — up axis (Y or Z) combined with handedness (left- or right-handed). Each convention maps to a target engine: ![Convention grid labelled with Unity, OpenGL, Unreal, Blender and ROS](/assets/images/coordinate_conventions_softwares-6fe97d69c57030d54bb2aca0adb19646.png) **Target software per convention.** Unity (Y-up, left-handed), three.js / OpenGL (Y-up, right-handed), Unreal (Z-up, left-handed), Blender / ROS (Z-up, right-handed). Both TOMLs keep the solver's native world: OpenCV axes (right-handed, Y-down, Z-forward). The OpenCV JSON uses the three.js world (Y-up, right-handed), so both of them place the cameras identically. ## Caliscope TOML (`camera_array.toml`)[​](#caliscope-toml-camera_arraytoml "Direct link to caliscope-toml-camera_arraytoml") The file Caliscope itself writes and loads: one `[cameras.N]` table per camera, keyed by the camera index. Native field semantics are preserved; project-specific fields are strictly additive, and Caliscope ignores them. | Field | Meaning | | ---------------- | ---------------------------------------------------------------------------- | | `cam_id` | Camera index (the table key) | | `size` | Image size `[width, height]` the intrinsics refer to (the export resolution) | | `rotation_count` | Sensor rotation in quarter turns — always `0` | | `error` | Intrinsic reprojection error (RMS, px) | | `matrix` | 3×3 intrinsic matrix | | `distortions` | Distortion coefficients — 5, OpenCV classic model `[k1, k2, p1, p2, k3]` | | `translation` | Extrinsic translation (world→camera), **always in metres** | | `rotation` | Extrinsic rotation, Rodrigues vector (world→camera) | | `grid_count` | Number of board views (keyframes) used for the intrinsic solve | | `fisheye` | Lens model flag — always `false` | Additive extensions: `name` (operator label) and `device_path` (stable V4L identifier). ``` [cameras.0] cam_id = 0 size = [ 1920, 1080 ] rotation_count = 0 error = 0.21 matrix = [ [ 1000.0, 0.0, 960.0 ], [ 0.0, 1000.0, 540.0 ], [ 0.0, 0.0, 1.0 ] ] distortions = [ 0.0, 0.0, 0.0, 0.0, 0.0 ] translation = [ 0.0, 0.0, 0.0 ] rotation = [ 0.0, 0.0, 0.0 ] grid_count = 24 fisheye = false name = "cam_0" device_path = "/dev/v4l/by-path/pci-0000:00:14.0-usb-0:1:1.0-video-index0" ``` Files exported by earlier realtime-calib versions used top-level `[cam_N]` tables instead, which recent Caliscope versions read as an empty camera array: export them again. ## aniposelib TOML (`camera_array_aniposelib.toml`)[​](#aniposelib-toml-camera_array_aniposelibtoml "Direct link to aniposelib-toml-camera_array_aniposelibtoml") The layout aniposelib reads (`CameraGroup.load`), used by Pose2Sim and anipose: one top-level `[cam_N]` table per camera, plus a `[metadata]` table. | Field | Meaning | | -------------------------------------------------------------- | ------------------------------------------------------------------ | | `name`, `size`, `matrix`, `distortions`, `rotation`, `fisheye` | As in `camera_array.toml` | | `translation` | Extrinsic translation (world→camera), in the selected export units | | `port`, `rotation_count`, `error`, `grid_count` | What Caliscope ≤ 0.5.4 reads from its `config.toml` | | `device_path` | Additive extension (stable V4L identifier) | `[metadata]` holds `adjusted = false`: the poses were not refined by anipose's own bundle adjustment. Exported in metres, the `[cam_N]` tables paste unchanged into the `config.toml` of a Caliscope ≤ 0.5.4 project. Exported in millimetres, that project would read a world 1000 times too large. ## OpenCV JSON (`camera_array_opencv.json`)[​](#opencv-json-camera_array_opencvjson "Direct link to opencv-json-camera_array_opencvjson") A versioned contract (`"format": "realtime-calib/opencv-cameras"`, `"version": 1`) for pipelines that project with OpenCV: the poses load as they are, with no axis conversion. Within a version, each field keeps one meaning: a reader ignores unknown fields, an addition keeps the version, a change of meaning bumps it. * **`convention`**: the pose formula `x_cam = R · x_world + t`, the camera axes (OpenCV: x right, y down, z forward), the world (right-handed, Y up, metres), and the basis from the solver's world. * **`world`**: where the world comes from. `frame` is `target` (framed on a board), `reference` (aligned on a previous calibration, with an `alignment` block), `anchor_camera` (the anchor still at the origin) or `unknown`; `origin` describes it in words. `up = "y"` is present only when verified: the framed target level within 1°, printed face up, every camera above it. `target_offset_m` says how far a Minimize moved the framed target off the origin. * **`source`**: the session, the export time and the service version. * **`cameras`**, sorted by `port`: | Field | Meaning | | -------------------- | ------------------------------------------------------------------------------------------- | | `port` | Camera index in the session (an order, not an identity) | | `name` | Operator label | | `device_path` | The camera's identity (stable V4L path); `null` for an imported session | | `resolution` | `[width, height]` the intrinsics refer to (the export resolution) | | `K` | 3×3 intrinsic matrix | | `distCoef` | `[k1, k2, p1, p2, k3]` | | `R` | 3×3 rotation, world → camera | | `t` | 3×1 translation column, world → camera, **always in metres** | | `intrinsic_error_px` | Intrinsic reprojection error (RMS, output px) | | `extrinsic_error_px` | Extrinsic reprojection error of this camera (output px), never mixed with the intrinsic one | This is also the file to keep as a **reference** for [aligning a later calibration](/docs/guides/extrinsic-calibration.md#keeping-the-rooms-frame-across-recalibrations) on the same room. ## Engine JSON (`camera_array_.json`)[​](#engine-json-camera_array_targetjson "Direct link to engine-json-camera_array_targetjson") A self-describing document: a top-level `convention` block (up axis, handedness, axis `mapping`, `camera_forward` / `camera_up`), the `world_units`, the `anchor` camera name, and a `cameras` array. Each camera carries a **scene form** — what a scene graph applies to place the camera object: | Field | Meaning | | --------------------- | -------------------------------------------------------------------------------------------------------- | | `position` | Camera position in world units (mm or m) | | `quaternion` | Camera orientation `[x, y, z, w]` (camera→world) | | `matrix` | 4×4 camera→world transform | | `intrinsics` | `{ resolution, matrix, distortions, fov_deg }` | | `error` | The camera's extrinsic reprojection error (output px), or its intrinsic one when the array is not solved | | `name`, `device_path` | As in the TOMLs | **Right-handed** targets (three.js, Blender) additionally carry a **view form** — the OpenCV-style extrinsic for projection (`x_cam = R · x_world + t`): | Field | Meaning | | -------- | ---------------------------------------------- | | `view.R` | 3×3 rotation, world→camera | | `view.t` | translation vector, world→camera (world units) | **Left-handed** targets (Unity, Unreal) omit the view form: their world basis includes a mirror (`det = −1`), which would make the view rotation improper (a reflection, not a rotation) — project through the engine's own camera API instead. --- # Accuracy & benchmarks Reproducible accuracy numbers for realtime-calib. Work in progress Benchmark results will be published here, including comparisons against Caliscope on shared datasets. ## What gets measured[​](#what-gets-measured "Direct link to What gets measured") A displayed RMSE alone is a poor judge: dropping the hardest observations lowers it, and a solve can lower it by bending the reconstructed target. Changes to the pipeline are therefore judged on recorded sessions against a judge chosen for the question: * **Reprojection error**, per camera and overall, in pixels at a stated resolution (comparisons across tools must use the same one). * **Target rigidity**: the reconstructed target's corner distances against the printed geometry, in millimetres. It depends on no solver parameter. * **Held-out views**: the error on views the solve did not use, to tell a better model from a better fit. * **Repeatability**: the spread of independent solves of the same rig. * **Scale**: camera-to-camera distances against tape measurements, and corner positions against a ChArUco ground truth. * **Runtime** of the live path (frames per second per camera, API latency) and of the computes. ## Reproducibility[​](#reproducibility "Direct link to Reproducibility") The repository ships `calibration-service/tools/eval_extrinsic_session.py`, which re-solves a recorded session folder with the production pipeline and reports the RMSE at native and output resolution, the target rigidity in millimetres and the distances between cameras. Each published number will link to the dataset, the exact commit or tag and the command used to produce it, so results can be independently reproduced. --- # Bibliography & references The tools and literature behind realtime-calib. ## Software & tools[​](#software--tools "Direct link to Software & tools") * **Caliscope** — the conceptual reference for the calibration logic (board definitions, ChArUco calibration, PnP/stereo extrinsics, bundle adjustment). BSD-2-Clause. [github.com/mprib/caliscope](https://github.com/mprib/caliscope) * **OpenCV** — camera calibration, ArUco/ChArUco (`CharucoDetector`, OpenCV ≥ 4.8). [docs.opencv.org](https://docs.opencv.org/) * **SciPy** — `scipy.optimize.least_squares` for bundle adjustment. [scipy.org](https://scipy.org/) * **LiveKit** — real-time transport for camera streams and overlays. [livekit.io](https://livekit.io/) For the peer-reviewed literature behind the method and the capture recommendations, see [Calibration best practices](/docs/reference/calibration-best-practices.md). --- # How to cite If realtime-calib is useful in your research, please cite it. A machine-readable [`CITATION.cff`](https://github.com/hans-brgs/realtime-calib/blob/main/CITATION.cff) lives at the repository root — GitHub shows a **"Cite this repository"** button that generates APA and BibTeX from it. DOI coming A DOI (via Zenodo, minted on the first tagged GitHub release) will be added here and to the `CITATION.cff` once available. ## BibTeX[​](#bibtex "Direct link to BibTeX") ``` @software{realtime_calib, title = {realtime-calib: real-time multi-camera calibration}, author = {Bourgeois, Hans}, year = {2026}, url = {https://github.com/hans-brgs/realtime-calib}, note = {AGPL-3.0-or-later} } ``` Please also cite [**Caliscope**](https://github.com/mprib/caliscope), whose calibration logic this project reimplements, and [**OpenCV**](https://opencv.org/). --- # Methodology This section describes **what** realtime-calib computes and points to the authoritative sources for the underlying theory, rather than re-deriving it. If you want the mathematics, follow the references: they are the ground truth. Where Caliscope is cited, the version is given, since its pipeline has changed across releases. ## Capture and replay[​](#capture-and-replay "Direct link to Capture and replay") Live detection only drives the operator's feedback. Every solve re-reads the **native recordings** of the session folder and re-detects the board on them, with the settings chosen in the Prepare step. Recordings are MJPG in Matroska, written at a configurable JPEG quality (default 95). ## Corner detection[​](#corner-detection "Direct link to Corner detection") * **ChArUco**: OpenCV's `CharucoDetector` (OpenCV ≥ 4.8), followed by `cornerSubPix` on the interpolated chessboard corners. That pass is load-bearing: on OpenCV 4.13 it removes a +0.48 px offset of `detectBoard`'s corners ([opencv#25539](https://github.com/opencv/opencv/issues/25539)); a view whose refinement fails is dropped rather than kept with raw corners. * **Single ArUco marker**: OpenCV's contour refinement places a marker's corners 1 to 2 px inward on real footage (blur and gamma push the apparent edge toward its dark side), which made every single-marker world 1.3 to 3 % too large. At compute time, the corners are instead refined from the marker's own **black border**: being exactly one cell wide, its inner edges move outward by the same amount its outer edges move inward, and the homography of the corrected quad predicts the true edges under any perspective. A view whose border cannot be read reliably is dropped, never kept with the biased corners, and counted per camera. The live overlay keeps the contour refinement. * **Inverted boards** are re-inverted before detection, as Caliscope does (v0.5.4 and v0.11.5). * The ArUco detector thresholds each image with two adaptive windows (3 and 23 px) instead of OpenCV's default three, as Caliscope v0.11.5 sets on its ChArUco tracker: about 35 to 44 % less detection time, with solves unchanged within their own scatter. ## Intrinsic calibration[​](#intrinsic-calibration "Direct link to Intrinsic calibration") Per-camera intrinsics and distortion are estimated from ChArUco detections using OpenCV: * `cv2.calibrateCameraExtended` (the ChArUco-specific `calibrateCameraCharucoExtended` was removed in OpenCV ≥ 4.7) with `CALIB_USE_INTRINSIC_GUESS` and no distortion-model flag: the classic **5-coefficient** distortion model `[k1, k2, p1, p2, k3]`, with a free aspect ratio. Caliscope v0.5.4 calls `cv2.calibrateCamera` without any flag; v0.11.5 adds the intrinsic guess. * The guess is Caliscope v0.11.5's seed: focal length `max(width, height)`, principal point at `((W − 1)/2, (H − 1)/2)`. Unlike OpenCV's own initialization, it does not depend on the order of the views: on the same 50 keyframes of a real camera, OpenCV's initialization landed anywhere between 1.4 and 6.5 px of RMS error depending on their order. OpenCV's initialization only stands in when the seeded solve fails. * The solve runs at native resolution. The camera matrix is then mapped to the output resolution at pixel centres, `c' = s · (c + 0.5) − 0.5`, which is how `cv2.resize` maps pixels; a plain `s · K` would leave the principal point `(1 − s)/2` px off. **Sources** * OpenCV camera calibration documentation: [`calibrateCamera`](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html) and the ArUco/ChArUco modules. * Caliscope's intrinsic pipeline: [Caliscope](https://github.com/mprib/caliscope) (BSD-2-Clause). * The choice of distortion model is discussed with its literature in [Calibration best practices](/docs/reference/calibration-best-practices.md#distortion-model). ### Keyframe selection[​](#keyframe-selection "Direct link to Keyframe selection") Before solving, realtime-calib picks a diverse, capped subset of the detections: 1. **Stride**: decode and detect one frame every N within the trimmed clip. 2. **Structural gates**: the board found, at least 6 corners, corners not near-collinear (the solver's own hard floors). 3. **Diversity anchors**: farthest-point sampling over a 3-D feature (board **tilt**, and the detection **centroid** as image fractions) places as many anchors as the cap (default 50). 4. **Sharpest per anchor**: every candidate is assigned to its nearest anchor, and the sharpest candidate of each cell is kept. Diversity decides how many keyframes and where; sharpness (the Laplacian variance over the board region, compared only within a cell) decides which frame represents each spot. There is no absolute blur gate, so a uniformly dim sweep still calibrates. The cap of 50 rests on a measurement: on four identical cameras, their focal estimates scattered by about 10 px at 25 keyframes and settled to 4 to 6 px at 50, where two different selectors also agreed within 0.4 %. Nine alternative selectors were evaluated on 12 real cameras against held-out views and against the downstream extrinsic solve; none beat this one consistently, Caliscope v0.11.5's included. **Sources**: farthest-point sampling is a standard diversity-sampling technique; the coverage / diversity goal follows Caliscope's approach to keyframe quality. ### Coverage and projection uncertainty[​](#coverage-and-projection-uncertainty "Direct link to Coverage and projection uncertainty") * **Coverage** is a quad accumulation map: each keyframe's board hull is rasterized on a grid, and a cell counts when its centre lies inside the hull, so the metric is not inflated by cells an edge merely touches. * **Projection uncertainty**: the intrinsic covariance, with the per-view poses marginalized out, is propagated to each cell of the image, with the rotation it implies projected out. Two covariances are computed (independent corner noise, which is OpenCV's own covariance, and a cluster-robust one by view) and the larger reading is kept. Cells past the fold of the radial distortion, where no ray reaches the pixel, are reported as outside the model. On real cameras, the figure matched the disagreement between solves of random halves of the data to within a factor of about 1.5. ## Extrinsic calibration[​](#extrinsic-calibration "Direct link to Extrinsic calibration") Every camera's 6-DoF pose is recovered in a **single shared coordinate frame** (the anchor's), from synchronized multi-camera detections of the target: 1. **Timestamps**: each frame carries the camera driver's buffer timestamp (V4L2, `CLOCK_MONOTONIC`), taken at grab. A base is chosen once per camera opening: the kernel stamps when they are plausible, the host clock otherwise. The kernel stamps lead the host's post-decode stamps by 20 to 50 ms, and by a varying amount. The base each camera used is recorded with the sweep. 2. **Grouping**: frames are grouped into synchronized instants with a window derived from the recorded frame intervals, just under one frame period, and a quorum of 2 cameras. The operator can tighten the maximum spread allowed within a group. 3. **Motion gate**: the cameras do not expose together, so a member captured a few milliseconds off its group's instant sees a moving target elsewhere. The compute tracks each member's corners into the neighbouring frames (Lucas–Kanade) for the target's image speed; a group whose worst member is more than 0.5 native px off (speed × time offset) is not used. A quarter of the group budget stays guaranteed, and the compute falls back to every group, with a warning, if the gated ones do not solve. 4. **Group selection**: the sharpest groups are kept up to the operator's cap, spread over the sweep in temporal bins. 5. **Pairwise poses**: for each co-visible camera pair, the relative transform is estimated with `cv2.stereoCalibrate` on their shared views (in normalized coordinates). 6. **Chaining from the anchor**: the pairwise estimates form a co-visibility graph; each camera's pose is chained from the **anchor** (camera index 0, fixed as identity) along the lowest-cumulative-error path (Dijkstra), with bridge-filling so indirect routes can beat noisy direct edges. 7. **Triangulation**: the target's corners are triangulated (DLT over all observing rays) into a 3D point cloud. 8. **Bundle adjustment**: `scipy.optimize.least_squares` (trf, sparse Jacobian) jointly refines every non-anchor pose and the 3D points, in two passes as Caliscope v0.11.5 does: a linear pass, then a robust `soft_l1` pass at a 1 px residual scale, converted with the array's median focal length so the outlier threshold does not drift with the lens. The anchor stays fixed, which removes the gauge freedom. **Imported videos** without real timestamps are synchronized the way Caliscope aligns bare videos (an inferred common-duration grid, then a greedy pass): on a real 4-camera Caliscope dataset, the slots match Caliscope's own sync map 396 out of 396. ### Target rigidity[​](#target-rigidity "Direct link to Target rigidity") Reprojection alone lets the solver bend the reconstructed target to absorb detection and sync noise, and leaves the world scale weakly constrained. The bundle adjustment therefore adds **rigidity constraints**, as Caliscope v0.11.5 does: within each group, the distance between two reconstructed corners must match the printed geometry, with a tolerance σ = 2 mm (so 2 mm of deformation costs about as much as 1 px of reprojection). * **Single marker**: its 4 sides and 2 diagonals. * **ChArUco**: the board's truss (neighbours and both diagonals of every cell), keeping every edge whose two corners the view triangulated, plus braces between the view's four extreme detected corners so that a partial view stays rigid across every fold line. The constraint rows go through the same robust pass. The reported reprojection error stays reprojection-only. **Board rigidity** (the RMS deviation of all within-group corner distances from the printed ones, in mm) is reported next to it as an independent judge: it measures the reconstruction against the physical target, which no solver parameter can game. ### Minimize[​](#minimize "Direct link to Minimize") Minimize is Caliscope's quality loop: the worst 2.5 % of observations (by reprojection error, over the whole array) are dropped, and the bundle adjustment is run again with the same rigidity constraints. ### Reported errors[​](#reported-errors "Direct link to Reported errors") All extrinsic errors are reported in pixels at the **output resolution**, like the intrinsic ones. The overall RMSE is re-aggregated from the per-camera terms weighted by their observation counts. When comparing with Caliscope, mind two differences: Caliscope v0.11.5 reports its RMSE after its own 2.5 % filter, and at the resolution of the videos it was given. **Sources** * OpenCV `stereoCalibrate`, triangulation and optical-flow documentation: [calib3d module](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html), [`calcOpticalFlowPyrLK`](https://docs.opencv.org/4.x/dc/d6b/group__video__track.html). * `scipy.optimize.least_squares`: [SciPy docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html). * Caliscope's extrinsic and bundle-adjustment implementation: [Caliscope](https://github.com/mprib/caliscope) (BSD-2-Clause), v0.11.5 for the two-pass robust solve and the rigidity constraints. ## Framing, alignment and checks[​](#framing-alignment-and-checks "Direct link to Framing, alignment and checks") * **Framing on a target**: the world is moved onto a target lying on the floor, its printed face up. The face is decided by the corner order (a ChArUco board's chessboard corners run the other way from a marker's), not by where the cameras are. * **Alignment on a reference** fits the solved rig onto a previous calibration of the same room, by matching camera identities: a yaw and a horizontal shift that keep the framed floor, or a full rigid (Kabsch) fit. * **Pre-export checks**: epipolar consistency per pair (the median symmetric epipolar distance of the solve's own observations), target rigidity, framing and cameras above the floor. With a site template, the implied scale from tape-measured camera distances is the weighted ratio `Σ(m·d/σ²) / Σ(d²/σ²)`, Caliscope v0.11.5's formula for its distance cues; realtime-calib only reports it, against bands at 2σ and 3σ of its own uncertainty. Why we cite instead of explain realtime-calib's contribution is making this pipeline **real-time, single-pass and inspectable**, not the calibration theory itself. We ground every claim on Caliscope and OpenCV rather than restating derivations, and every change to the pipeline is judged on recorded sessions against a judge chosen for the question (ground truth, held-out views, target rigidity), never on the displayed RMSE alone. ---