Keypoint formats compared: COCO 17, MPII 16, MediaPipe 33 and COCO-WholeBody 133

The four formats at a glance

Front view convention: in every format, "left" means the person's left, which appears on the viewer's right.

COCO: 17 keypoints

COCO stores each person as a flat list of 17 triplets, x, y, v, where v is 0 (not labeled), 1 (labeled but not visible) or 2 (labeled and visible). The order is fixed:

The full skeleton pairs, visibility rules and JSON examples are in the COCO keypoint format reference.

MPII: 16 keypoints

MPII Human Pose has around 25,000 images with over 40,000 annotated people. Its 16 joints start from the right ankle:

Some libraries reorder these joints when they merge MPII with other datasets, so check the source before you reuse indices.

MediaPipe Pose: 33 landmarks

MediaPipe's Pose Landmarker returns 33 landmarks per person, both in normalized image coordinates and in world coordinates:

COCO-WholeBody: 133 keypoints

COCO-WholeBody adds whole-body labels to COCO images: the 17 body keypoints, 6 foot keypoints, 68 face keypoints and 42 hand keypoints. The 6 foot points come in this order: left big toe, left small toe, left heel, right big toe, right small toe, right heel.

Converting between formats

The 12 body joints (shoulders, elbows, wrists, hips, knees, ankles) exist in all four formats, so most conversions are index lookups. What differs is the head, the torso center, and the hands and feet. This table maps every COCO keypoint to its counterpart in the other formats:

The code below works on one person at a time and represents every format the same way: a list of points, each (x, y, v) in pixels, where v follows COCO's rule (0 not labeled, 1 labeled but not visible, 2 visible). Converting to COCO's flat JSON list is then [value for point in points for value in point].

MISSING = (0.0, 0.0, 0)

# COCO index -> index in the other format (None = no counterpart)
COCO_TO_MPII      = [None, None, None, None, None, 13, 12, 14, 11, 15, 10, 3, 2, 4, 1, 5, 0]
COCO_TO_MEDIAPIPE = [0, 2, 5, 7, 8, 11, 12, 13, 14, 15, 16, 23, 24, 25, 26, 27, 28]

def midpoint(a, b):
    """Average of two points, labeled only if both are."""
    if a[2] == 0 or b[2] == 0:
        return MISSING
    return ((a[0] + b[0]) / 2, (a[1] + b[1]) / 2, min(a[2], b[2]))

def mediapipe_to_points(landmarks, width, height, visible=0.5):
    """MediaPipe landmarks (normalized) -> 33 (x, y, v) points in pixels.
    `visible` is your own threshold on the landmark's visibility score."""
    return [(lm.x * width, lm.y * height,
             2 if getattr(lm, "visibility", 1.0) >= visible else 1)
            for lm in landmarks]

To COCO

def mpii_to_coco(mpii):
    """16 MPII points -> 17 COCO points. Nose, eyes and ears are not in MPII."""
    return [mpii[j] if j is not None else MISSING for j in COCO_TO_MPII]

def mediapipe_to_coco(mp):
    """33 MediaPipe points -> 17 COCO points. Every COCO point has a match."""
    return [mp[j] for j in COCO_TO_MEDIAPIPE]

def wholebody_to_coco(wb):
    """133 COCO-WholeBody points -> 17 COCO points. The body part is identical."""
    return list(wb[:17])

To MPII

MPII needs four points the others don't have. Pelvis and thorax are commonly approximated as the midpoints of the hips and of the shoulders. Upper neck and head top cannot be derived reliably, so they are left unlabeled and have to be annotated if your model needs them.

def coco_to_mpii(c):
    """17 COCO points -> 16 MPII points."""
    return [
        c[16], c[14], c[12],          # 0 r ankle, 1 r knee, 2 r hip
        c[11], c[13], c[15],          # 3 l hip, 4 l knee, 5 l ankle
        midpoint(c[11], c[12]),       # 6 pelvis (approximation)
        midpoint(c[5], c[6]),         # 7 thorax (approximation)
        MISSING, MISSING,             # 8 upper neck, 9 head top: not derivable
        c[10], c[8], c[6],            # 10 r wrist, 11 r elbow, 12 r shoulder
        c[5], c[7], c[9],             # 13 l shoulder, 14 l elbow, 15 l wrist
    ]

def mediapipe_to_mpii(mp):
    return coco_to_mpii(mediapipe_to_coco(mp))

def wholebody_to_mpii(wb):
    return coco_to_mpii(wholebody_to_coco(wb))

To COCO-WholeBody

COCO-WholeBody starts with the COCO 17, so the body part copies over and the 116 foot, face and hand points stay unlabeled. From MediaPipe you can also fill four of the six foot points: heels map directly, and MediaPipe's "foot index" (the tip of the foot) is a rough stand-in for the big toe. There is no small toe, and MediaPipe's few face and hand points do not line up with WholeBody's 68 face and 21-per-hand layouts.

def coco_to_wholebody(c):
    """17 COCO points -> 133 COCO-WholeBody points, body only."""
    return list(c) + [MISSING] * 116

def mediapipe_to_wholebody(mp, toes_from_foot_index=True):
    wb = coco_to_wholebody(mediapipe_to_coco(mp))
    # WholeBody 17-22: l big toe, l small toe, l heel, r big toe, r small toe, r heel
    wb[19], wb[22] = mp[29], mp[30]           # heels
    if toes_from_foot_index:                  # approximation, see above
        wb[17], wb[20] = mp[31], mp[32]
    return wb

def mpii_to_wholebody(mpii):
    return coco_to_wholebody(mpii_to_coco(mpii))

To MediaPipe

MediaPipe output comes from its own model, so in practice you convert out of it, not into it. If you need MediaPipe's 33-point layout for labels, fill the 17 points COCO shares (the COCO_TO_MEDIAPIPE lookup in reverse) and leave the extra eye, mouth, hand and foot points unlabeled, or annotate them.

Which one to label in

• COCO 17 if you want to train common 2D pose models or start from public weights. Most tools read and write it.
• A superset of COCO if you need hands, feet or face. Keep the first 17 points in COCO order and add your extra points after them, so the body part stays compatible.
• Your own skeleton for animals, instruments or anything that is not a human body. The JSON structure stays the same, only the names and pairs change.

To label keypoints in any of these layouts in the browser, see how to annotate keypoints in DataTorch. For public datasets in each format, see keypoint and pose estimation datasets.

Sources

• COCO keypoints: the COCO dataset (cocodataset.org) and our COCO keypoint format reference.
• MPII Human Pose: Max Planck Institute for Informatics, https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/software-and-datasets/mpii-human-pose-dataset
• MediaPipe Pose Landmarker: https://developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker
• COCO-WholeBody: Jin et al., Whole-Body Human Pose Estimation in the Wild, ECCV 2020, https://github.com/jin-s13/COCO-WholeBody