Backend
Pipeline
For each frame: the detector finds the animals, the pose network places keypoints in a crop around each one, and the tracker links them to the identities of earlier frames. When the video is done, each track is refined by the temporal network, classified frame by frame by the behavior network, and scored by the experiment's rules. A camera rig adds identity matching across views, drift correction and triangulation (Multiview).
The shared skeleton
All networks uses 25 keypoints, grouped into 5 regions:
| Region | Keypoints |
|---|---|
| head | nose, left_eye, right_eye, left_ear, right_ear, head_center |
| trunk | neck, spine_upper, spine_mid, spine_lower |
| tail | tail_base, tail_mid, tail_end |
| forelimbs | left/right_shoulder, left/right_front_knee, left/right_front_paw |
| hindlimbs | left/right_hip, left/right_hind_knee, left/right_hind_paw |
The skeleton has 25 edges and 8 left/right pairs. Each dataset is mapped onto the keypoints
(DATASET_MAPPINGS: SuperAnimal top-view mouse and quadruped, AP-10K, CalMS21, Rat7M,
Lightning Pose mirror mouse and CRIM13, DeepLabCut home cage, synthetic).
Detector
CenterNet-style and anchor-free:
- Backbone: a small ResNet (the general detector) or ResNet-18/34/50, with a feature pyramid
merged into one map at stride 4 (64 channels). Input is letterboxed to
det_size(default 256 × 320). - Heads: an "animal" center heatmap, box size (log width and height) and a sub-cell offset; optionally a species head.
- Decoding: peaks of the heatmap (3 × 3 max-pool), top
max_det(20), boxes from offset and size. Boxes abovedet_thresh(0.3) start tracks; weaker ones down totracker.low_threshcan continue them. - Fixed cameras can add silhouettes against the median background as extra candidates and drop detections that match the background, for small or blurry animals.
Pose
- A square crop around each box (
crop_padding1.25 × the box), resized tocrop_size(192). - The general pose network is a ResNet-18 with a feature pyramid, ImageNet-pretrained and fine-tuned on real recordings (CRIM13, Rat7M, mirror mouse) and synthetic data, including side views.
- Each keypoint has a learned query vector; its heatmap is that vector against the shared features, so new keypoints can be added without changing the rest. A keypoint is the heatmap's peak plus a sub-pixel offset; its confidence is the peak's probability.
- Left/right pairs (
pair_decoding) are kept on different peaks, so a left and right paw never land on the same spot. - A 64-dimensional appearance embedding of each animal feeds re-identification in the tracker.
Tracking
A Kalman filter with Hungarian matching. Each track's box moves with constant velocity; noise scales with its size. Every frame matches in four stages:
- Confident detections (≥
high_thresh0.5) by box overlap, keypoint similarity (OKS, motion-compensated) and appearance (weights 0.5 / 0.3 / 0.2). - Weak detections (≥
low_thresh0.2) by box overlap only, to keep tracks through occlusion (as in ByteTrack). - Re-identification of lost tracks by appearance (a gallery of 30 embeddings per track), within an area that widens the longer the track has been lost.
- New tracks from confident detections left over (confirmed after 3 frames).
A track is dropped after max_lost (90) frames unseen. With max_animals set, no identities
beyond that number are created, every reappearance is re-identified, and lost tracks never die:
set it when you know how many animals there are. pair_identity then fixes left/right label
swaps along each track.
Temporal refinement
A bidirectional LSTM per track with 2 layers of 256, over windows of 64 frames with 50 % overlap.
- A prior first fills the track: a kernel average of nearby observations, and straight lines through gaps.
- The LSTM predicts a correction to the prior and its own uncertainty; corrections that are too large fall back to the prior.
- Output: smoothed keypoints, a confidence from the uncertainty, and an
imputedflag where the point was not observed.
temporal3d is the same network for 3D tracks. An online version is a config switch.
Behavior
A temporal convolutional network per track:
- Features: keypoints centered on the body, rotated to its axis and scaled by body length, with velocity, turning and keypoint speeds (and height in 3D).
- Network: 5 dilated convolution blocks (dilations 1–16), one label per frame.
- Classes of the general network: resting, sniffing, walking, turning, rearing, grooming, other.
- Post-processing: a 5-frame moving median; the class balance adapts to the session
(expectation–maximization, at most 4× a class's prior); frames below
behavior_conf(0.6) areother. - Catalog: classes map into 133 named behaviors; rules add
run(walking over 2 body lengths per second) and, between two animals, social behaviors (sniffing, following, chasing, …). See Behavior.
To build labels for a new behavior set, animalpose behavior cluster groups kinematics (k-means),
name names clusters by published signatures, label writes session files and plot draws an
ethogram and clip.
Assay
The experiment layer converts tracks into the measures of standard tests.
| Experiment | Modules |
|---|---|
| open field | open_field, grooming, rearing |
| elevated plus maze | plus_maze, grooming, rearing |
| three chamber | contact, zones, grooming, rearing |
| grooming | zones, grooming, rearing |
- Arena: found by a trained arena detector (median over 16 frames; flagged
needs_reviewwhen unsure), by calibration, or by image analysis (the plus maze's arms); a hand-made layout always wins. - Units: distances are in body lengths or shares of the arena, never pixels.
- Rules: each module is one rule set (zones and entries, wall distance, head dips, cup
contact, grooming bouts and type, rearing), with thresholds in the config's
assay:block. - Reproducible: each run records its code, config and model hashes and inputs; rerunning on unchanged inputs is skipped, and results are replaced atomically.
New experiments need no code: Adding an experiment setup.
Python and Rust
Python (src/animalpose/) |
Rust (rust/src/) |
|
|---|---|---|
| Role | reference implementation; training, evaluation, export | the desktop apps and the animalpose program |
| Runs on | PyTorch: CUDA, Metal (MPS) or CPU | its own tensor core on all CPU cores; Metal on Apple Silicon |
| Reads | .pt |
.pt directly (no PyTorch) and .safetensors |
Experiments (assay) |
— | yes |
The Rust tensor core (rust/src/tensor/) is pure Rust: convolutions as matrix products with
matrixmultiply, spread over cores with rayon. With the mps feature (Apple Silicon) a
network's forward pass is replayed as one Metal Performance Shaders graph, about 6× faster than
the CPU; the LSTM stays on the CPU.
Parity. The Rust tests compare against reference outputs written by the Python package
(animalpose fixtures, stored in tests/fixtures/):
| Compared | Tolerance |
|---|---|
| network outputs | 2 × 10⁻⁴ |
| boxes and scores | 10⁻⁵ |
| keypoints and refined tracks | 10⁻³ px |
| confidences and probabilities | 10⁻³ |
| track identities, behavior labels, checkpoints | exact |
Intentional differences are marked // differs from Python: in the Rust source (for example,
video frames are converted to color by FFmpeg rather than OpenCV, within a gray level or two).
Checkpoints. .pt files are read by a minimal pickle reader in Rust
(rust/src/models/torch_pt.rs); .safetensors files carry the network's config and kind as
metadata (animalpose export safetensors). A config's checksums: are verified before loading.
Video
Video goes through FFmpeg's command-line programs (rust/src/data/video.rs), so the Rust build
needs no C toolchain: ffprobe reads each video's size, frame rate, frame count and every
frame's real timestamp; ffmpeg decodes frames to BGR and encodes overlay.mp4 and recordings
(H.264 with libx264 when available, else MPEG-4). Real timestamps matter because cameras drop
frames: time_s in the results and the kinematics use them rather than an assumed frame rate.
The Python package decodes with OpenCV and reads timestamps with ffprobe when it is installed.
Command line
| Command | What it does |
|---|---|
predict |
analyze a video or frame folder (--video), a rig (--views + --cameras) or a synthetic demo (--synthetic) |
assay |
rescore experiments on finished analyses, no networks (Rust) |
train, finetune |
train a detector, pose, temporal or behavior network (Training) |
label |
pseudo-label unlabeled recordings for self-training |
behavior |
cluster, name, label, plot, sessions |
data |
audit (camera profile), calibrate (camera model), convert (DeepLabCut → COCO), prepare, download |
eval |
detection AP, keypoint PCK, tracking MOTA/IDF1, behavior F1, benchmarks |
export |
safetensors, import-safetensors; TorchScript and ONNX (Python) |
samples |
check the sample profiles bundled with the apps |
Source map
| Path | Contents |
|---|---|
src/animalpose/ |
Python: detector/, pose/, tracking/, temporal/, behavior/, multiview/, skeleton/, data/, train/, eval/, export/, cli/, api.py, pipeline.py |
rust/src/tensor/ |
tensors, kernels, autodiff, the Metal graph |
rust/src/models/ |
the four networks and checkpoint reading |
rust/src/algos/ |
decoding, NMS, Kalman, Hungarian, tracker, background, kinematics, post-processing |
rust/src/data/ |
configs, frames and video, synthetic data |
rust/src/multiview/ |
cameras, fusion, triangulation, drift |
rust/src/assay/ |
experiments: arena, modules, outputs |
rust/src/cli/ |
the animalpose commands |
rust/src/app/ |
the engine both desktop apps share (library, recording, processing) |
rust/src/gui.rs, macos/ |
the egui window; the SwiftUI app over rust/src/ffi.rs |