Keypoint annotation, done properly.
Keypoint annotation marks semantic landmarks (joints, facial features, instrument tips) as single coordinates, and it powers pose estimation, gesture recognition, sports and clinical motion analysis, and robotics. It is also the annotation type where silent, systematic errors are easiest to make. This guide covers the workflow and the mistakes that matter.
Four steps, in order
01
Define the schema first
Fix the keypoint names, their order, and the skeleton pairs before anyone places a point. Every downstream consumer (training code, metrics, other annotators) depends on the order being identical across the whole dataset. Changing the schema mid-dataset means re-touching every finished image.
02
Place points, not regions
Each keypoint is a single (x, y) coordinate on a semantic landmark: a joint center, an eye, a tool tip. Aim for the anatomical point, not the visual center of a blob: a wrist is the joint, not the middle of the hand. Consistency across annotators matters more than sub-pixel precision.
03
Mark occlusion explicitly
An occluded landmark still has a location: a hip under a coat exists and is worth estimating. Formats carry a visibility flag per point (COCO: 0 not labeled, 1 labeled but hidden, 2 visible) so models can learn from occluded points without being penalized on them.
04
Review before export
Keypoint errors are systematic, not random: left/right flips, skipped occlusions, and order mistakes repeat across an annotator’s whole batch. A second-pass review of a sample per annotator catches the pattern early, far cheaper than discovering it in model metrics.
The four classic mistakes
Keypoint errors rarely look wrong on screen. They surface weeks later as models that confuse left and right, or metrics that plateau for no visible reason.
Left/right flips
“Left” means the subject’s left: the viewer’s right in a front-facing image, and the viewer’s left when the subject faces away. The single most common keypoint labeling error, and it silently poisons symmetry-aware models.
Occluded ≠ missing
Not-labeled (the point does not exist in the image or was skipped) and labeled-but-hidden are different signals. Collapsing them wastes the occlusion information most pose models can exploit.
Inconsistent order
The keypoint array is positional. If one annotation lists the points in a different order, every coordinate is attached to the wrong landmark, and nothing visually flags it until training.
Skeleton off-by-one
In COCO, the keypoints array is 0-based but the skeleton pairs are 1-based ids. Tools that mix these draw bones between the wrong joints.
The COCO keypoint format
The de-facto interchange format: 17 person keypoints, a 1-based skeleton, and per-point visibility flags. The full reference (order, skeleton pairs, JSON examples) lives on its own page.
COCO keypoint referenceKeypoint datasets to start from
Public pose and landmark datasets: the standard benchmarks plus downloadable keypoint datasets hosted on DataTorch.
Browse keypoint datasetsA keypoint tool that knows the skeleton.
DataTorch builds the schema into the label: guided point-by-point placement in the declared order, a live skeleton overlay, per-point visibility flags, and COCO import/export. The mistakes above become hard to make.
The annotation platform for specialized imagery: review, score, and share datasets your team works from.
© 2026 DataTorch. All rights reserved.