Skip to content

Multiview

Two or more cameras on one arena give 3D keypoints: rearing height, paws hidden from one view, and identities that hold when animals cross. This page covers the camera model, how views are fused, and how small camera movements are corrected between sessions.

← Overview · Backend · Results → Multiview

Camera model (cams.json)

A JSON list with one entry per camera, read the same way by Python and Rust:

[
  {"name": "Camera 1", "K": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]],
   "R": [[…], […], […]], "t": [tx, ty, tz], "dist": [k1, k2, p1, p2, k3], "size": [1920, 1080]}
]
Key Meaning
name the camera's name; videos are matched to cameras by it
K intrinsic matrix (focal lengths and principal point, in pixels)
R, t rotation and translation from world to camera: x_cam = R · X + t
dist lens distortion, OpenCV's Brown–Conrady model
size image width and height

World units are those of the calibration points (e.g. cm), with z up.

Calibrating

animalpose data calibrate rig_points.json --out cams.json

rig_points.json gives, per camera, the 3D positions of landmarks (points3d, e.g. arena corners measured in cm) and where they appear in that camera's image (points2d), optionally with known K and dist. Each camera is solved from at least 6 points (or by PnP when its intrinsics are known); --refine obs.npz bundle-adjusts the cameras against many observations.

A calibration holds while cameras stay put. Small knocks are corrected automatically (Camera drift); a camera moved somewhere else needs a new calibration.

Fusing views

animalpose predict --views top.mp4 side.mp4 --cameras cams.json --out session1

In the app, put a camera model in the profile and turn on Multiview (Desktop app).

  1. Matching videos to cameras. By file name (Camera 1.mp4 → "Camera 1", ignoring case and punctuation); failing that, by the number in the name (View-2, camera-02, cam1), lowest first, in the camera model's order. --views on the command line is matched by position.
  2. 2D tracking per view, exactly as for a single video.
  3. Identity across views. Tracks are paired by their epipolar distance (how well one view's keypoints lie on the lines the other view's keypoints project to), then clustered into animals with at most one track per view; pairs costing more than 15 px are not joined.
  4. Triangulation. Each keypoint is triangulated from every view that saw it with confidence ≥ 0.3, weighted by confidence. If a view reprojects more than 10 px off, every subset of views is tried and the largest consistent one kept. Points seen by fewer than 2 views stay NaN. The mean reprojection error is kept per point as a quality measure.
  5. Refinement and behavior in 3D (temporal3d, behavior3d), when the config names them.

Camera drift

Cameras get knocked, tripods sag and mounts creep. Before fusing, each camera is checked against how it looked when the camera model was first used, and its rotation is corrected (rust/src/multiview/drift.rs, src/animalpose/multiview/drift.py).

  1. Reference picture. The first session fused with a camera model saves each camera's median background (the arena without animals) as <model>.references/<camera>.png, beside the camera model (~/AnimalPose/cameras/ in the app).
  2. Registration. Each later session's median background is aligned to the reference, coarse to fine over an image pyramid, by solving for the camera's 3D rotation through its own lens model. Lighting differences are solved alongside (gain and offset), and robust weights let moved objects (a bottle, a cable) count less than walls and desk.
  3. Correction. The rotation is applied only when it makes the session agree better with the reference. Nothing caps how far a camera may turn.

Each camera's outcome is in camera_drift.json (corrected, unchanged, reference saved or skipped, with degrees turned and agreement before and after), and cameras.json holds the cameras as used. Settings are in the config's drift: block (Configuration).

To take a new reference (after cleaning up the arena, say), delete the .references folder: the next session saves new ones.

Tips

  • Name videos after their cameras, and keep one session's videos in one folder.
  • Views are paired by frame number (frame 100 of each video is one moment), so start the cameras together and keep them at the same frame rate. recording.json shows each clip's frame count and first and last frame time, so you can check the views line up.
  • Watch reproj_error in animals3d.npz: a few pixels is normal; tens of pixels mean a camera moved beyond what drift correction handles, or the calibration is off.