Skip to content

Backend

Pipeline

Frames

Detector

Pose

Tracker

Temporal refinement

Behavior

tracks · behavior · overlay

Assay
(experiment rules)

For each frame: the detector finds the animals, the pose network places keypoints in a crop around each one, and the tracker links them to the identities of earlier frames. When the video is done, each track is refined by the temporal network, classified frame by frame by the behavior network, and scored by the experiment's rules. A camera rig adds identity matching across views, drift correction and triangulation (Multiview).

The shared skeleton

All networks uses 25 keypoints, grouped into 5 regions:

Region Keypoints
head nose, left_eye, right_eye, left_ear, right_ear, head_center
trunk neck, spine_upper, spine_mid, spine_lower
tail tail_base, tail_mid, tail_end
forelimbs left/right_shoulder, left/right_front_knee, left/right_front_paw
hindlimbs left/right_hip, left/right_hind_knee, left/right_hind_paw

The skeleton has 25 edges and 8 left/right pairs. Each dataset is mapped onto the keypoints (DATASET_MAPPINGS: SuperAnimal top-view mouse and quadruped, AP-10K, CalMS21, Rat7M, Lightning Pose mirror mouse and CRIM13, DeepLabCut home cage, synthetic).

Detector

CenterNet-style and anchor-free:

  • Backbone: a small ResNet (the general detector) or ResNet-18/34/50, with a feature pyramid merged into one map at stride 4 (64 channels). Input is letterboxed to det_size (default 256 × 320).
  • Heads: an "animal" center heatmap, box size (log width and height) and a sub-cell offset; optionally a species head.
  • Decoding: peaks of the heatmap (3 × 3 max-pool), top max_det (20), boxes from offset and size. Boxes above det_thresh (0.3) start tracks; weaker ones down to tracker.low_thresh can continue them.
  • Fixed cameras can add silhouettes against the median background as extra candidates and drop detections that match the background, for small or blurry animals.

Pose

  • A square crop around each box (crop_padding 1.25 × the box), resized to crop_size (192).
  • The general pose network is a ResNet-18 with a feature pyramid, ImageNet-pretrained and fine-tuned on real recordings (CRIM13, Rat7M, mirror mouse) and synthetic data, including side views.
  • Each keypoint has a learned query vector; its heatmap is that vector against the shared features, so new keypoints can be added without changing the rest. A keypoint is the heatmap's peak plus a sub-pixel offset; its confidence is the peak's probability.
  • Left/right pairs (pair_decoding) are kept on different peaks, so a left and right paw never land on the same spot.
  • A 64-dimensional appearance embedding of each animal feeds re-identification in the tracker.

Tracking

A Kalman filter with Hungarian matching. Each track's box moves with constant velocity; noise scales with its size. Every frame matches in four stages:

  1. Confident detections (≥ high_thresh 0.5) by box overlap, keypoint similarity (OKS, motion-compensated) and appearance (weights 0.5 / 0.3 / 0.2).
  2. Weak detections (≥ low_thresh 0.2) by box overlap only, to keep tracks through occlusion (as in ByteTrack).
  3. Re-identification of lost tracks by appearance (a gallery of 30 embeddings per track), within an area that widens the longer the track has been lost.
  4. New tracks from confident detections left over (confirmed after 3 frames).

A track is dropped after max_lost (90) frames unseen. With max_animals set, no identities beyond that number are created, every reappearance is re-identified, and lost tracks never die: set it when you know how many animals there are. pair_identity then fixes left/right label swaps along each track.

Temporal refinement

A bidirectional LSTM per track with 2 layers of 256, over windows of 64 frames with 50 % overlap.

  • A prior first fills the track: a kernel average of nearby observations, and straight lines through gaps.
  • The LSTM predicts a correction to the prior and its own uncertainty; corrections that are too large fall back to the prior.
  • Output: smoothed keypoints, a confidence from the uncertainty, and an imputed flag where the point was not observed.

temporal3d is the same network for 3D tracks. An online version is a config switch.

Behavior

A temporal convolutional network per track:

  • Features: keypoints centered on the body, rotated to its axis and scaled by body length, with velocity, turning and keypoint speeds (and height in 3D).
  • Network: 5 dilated convolution blocks (dilations 1–16), one label per frame.
  • Classes of the general network: resting, sniffing, walking, turning, rearing, grooming, other.
  • Post-processing: a 5-frame moving median; the class balance adapts to the session (expectation–maximization, at most 4× a class's prior); frames below behavior_conf (0.6) are other.
  • Catalog: classes map into 133 named behaviors; rules add run (walking over 2 body lengths per second) and, between two animals, social behaviors (sniffing, following, chasing, …). See Behavior.

To build labels for a new behavior set, animalpose behavior cluster groups kinematics (k-means), name names clusters by published signatures, label writes session files and plot draws an ethogram and clip.

Assay

The experiment layer converts tracks into the measures of standard tests.

Experiment Modules
open field open_field, grooming, rearing
elevated plus maze plus_maze, grooming, rearing
three chamber contact, zones, grooming, rearing
grooming zones, grooming, rearing
  • Arena: found by a trained arena detector (median over 16 frames; flagged needs_review when unsure), by calibration, or by image analysis (the plus maze's arms); a hand-made layout always wins.
  • Units: distances are in body lengths or shares of the arena, never pixels.
  • Rules: each module is one rule set (zones and entries, wall distance, head dips, cup contact, grooming bouts and type, rearing), with thresholds in the config's assay: block.
  • Reproducible: each run records its code, config and model hashes and inputs; rerunning on unchanged inputs is skipped, and results are replaced atomically.

New experiments need no code: Adding an experiment setup.

Python and Rust

Python (src/animalpose/) Rust (rust/src/)
Role reference implementation; training, evaluation, export the desktop apps and the animalpose program
Runs on PyTorch: CUDA, Metal (MPS) or CPU its own tensor core on all CPU cores; Metal on Apple Silicon
Reads .pt .pt directly (no PyTorch) and .safetensors
Experiments (assay) — yes

The Rust tensor core (rust/src/tensor/) is pure Rust: convolutions as matrix products with matrixmultiply, spread over cores with rayon. With the mps feature (Apple Silicon) a network's forward pass is replayed as one Metal Performance Shaders graph, about 6× faster than the CPU; the LSTM stays on the CPU.

Parity. The Rust tests compare against reference outputs written by the Python package (animalpose fixtures, stored in tests/fixtures/):

Compared Tolerance
network outputs 2 × 10⁻⁴
boxes and scores 10⁻⁵
keypoints and refined tracks 10⁻³ px
confidences and probabilities 10⁻³
track identities, behavior labels, checkpoints exact

Intentional differences are marked // differs from Python: in the Rust source (for example, video frames are converted to color by FFmpeg rather than OpenCV, within a gray level or two).

Checkpoints. .pt files are read by a minimal pickle reader in Rust (rust/src/models/torch_pt.rs); .safetensors files carry the network's config and kind as metadata (animalpose export safetensors). A config's checksums: are verified before loading.

Video

Video goes through FFmpeg's command-line programs (rust/src/data/video.rs), so the Rust build needs no C toolchain: ffprobe reads each video's size, frame rate, frame count and every frame's real timestamp; ffmpeg decodes frames to BGR and encodes overlay.mp4 and recordings (H.264 with libx264 when available, else MPEG-4). Real timestamps matter because cameras drop frames: time_s in the results and the kinematics use them rather than an assumed frame rate. The Python package decodes with OpenCV and reads timestamps with ffprobe when it is installed.

Command line

Command What it does
predict analyze a video or frame folder (--video), a rig (--views + --cameras) or a synthetic demo (--synthetic)
assay rescore experiments on finished analyses, no networks (Rust)
train, finetune train a detector, pose, temporal or behavior network (Training)
label pseudo-label unlabeled recordings for self-training
behavior cluster, name, label, plot, sessions
data audit (camera profile), calibrate (camera model), convert (DeepLabCut → COCO), prepare, download
eval detection AP, keypoint PCK, tracking MOTA/IDF1, behavior F1, benchmarks
export safetensors, import-safetensors; TorchScript and ONNX (Python)
samples check the sample profiles bundled with the apps

Source map

Path Contents
src/animalpose/ Python: detector/, pose/, tracking/, temporal/, behavior/, multiview/, skeleton/, data/, train/, eval/, export/, cli/, api.py, pipeline.py
rust/src/tensor/ tensors, kernels, autodiff, the Metal graph
rust/src/models/ the four networks and checkpoint reading
rust/src/algos/ decoding, NMS, Kalman, Hungarian, tracker, background, kinematics, post-processing
rust/src/data/ configs, frames and video, synthetic data
rust/src/multiview/ cameras, fusion, triangulation, drift
rust/src/assay/ experiments: arena, modules, outputs
rust/src/cli/ the animalpose commands
rust/src/app/ the engine both desktop apps share (library, recording, processing)
rust/src/gui.rs, macos/ the egui window; the SwiftUI app over rust/src/ffi.rs