The DSB2018 nuclei segmentation dataset, explained.
The 2018 Data Science Bowl dataset is the standard starting point for learning nuclei segmentation: 670 microscopy images with every nucleus individually outlined, spanning fluorescence and stained histology. What is in it, how the masks are stored, and which models are worth training on it, plus a hosted copy you can browse and download without touching Kaggle.
One dataset, deliberately messy
The images come from many labs, microscopes, stains, and magnifications on purpose: the competition’s question was whether one model can find nuclei across all of them. That variety is exactly what makes it a better teaching set than a single clean plate.
670
training images
stage1_train, each with its full set of expert masks
~29k
nucleus masks
one binary PNG per nucleus, no overlaps
5+
imaging conditions
fluorescence and brightfield histology, many magnifications and sizes
CC0
license
public domain: free for coursework, research, and commercial use
One binary PNG per nucleus
Each sample is a folder named by image id. The image lives in images/, and every individual nucleus gets its own binary mask in masks/. Masks never overlap, so the union is the semantic mask and numbering them gives you an instance label image:
stage1_train/
0a7d30b2.../ # one folder per image id
images/
0a7d30b2....png # the microscopy image
masks/
07a9bf1d....png # nucleus 1 (binary)
0e548d0a....png # nucleus 2 (binary)
... # one PNG per nucleusimport numpy as np, imageio.v3 as iio
from pathlib import Path
def instance_labels(sample_dir):
"""Merge per-nucleus PNGs into one
uint16 label image (0 = background,
1..N = nucleus id)."""
label = None
masks = sorted(Path(sample_dir, "masks").glob("*.png"))
for i, path in enumerate(masks, start=1):
m = iio.imread(path) > 0
if label is None:
label = np.zeros(m.shape, np.uint16)
label[m] = i
return labelThe Kaggle competition scored submissions in run-length encoding (RLE), one encoded row per nucleus. You only need RLE if you are replaying the leaderboard; for training, the PNG masks above are the ground truth.
Four models, four reasons
All four families below have public implementations that train on this dataset in an afternoon on a single GPU. Pick by what you want to learn, not by leaderboard score:
U-Net
The teaching baseline
The standard first model for a course or thesis. Its output is semantic (nucleus vs background), so touching nuclei merge into one blob: separating them needs a distance transform and watershed on top. Building that post-processing is half the lesson.
StarDist
Best default for nuclei
Predicts a star-convex polygon per nucleus, so instances come out separated by design. It was developed and benchmarked on this exact dataset, trains quickly on a free Colab GPU, and is pip-installable.
Cellpose
Strongest pretrained start
A generalist model with a pretrained nuclei mode that performs well before you train anything. The interesting exercise is fine-tuning it on your own images and measuring the gain.
Mask R-CNN
The competition winner
Most of the top 2018 leaderboard entries were Mask R-CNN variants. Heavier to train and tune than the others; reach for it when nuclei are irregular or you want detection boxes too.
The four mistakes everyone makes once
Instance, not semantic
The task (and the competition metric, mean AP over IoU 0.50 to 0.95) is counting and outlining individual nuclei. A binary foreground mask that welds touching nuclei together scores badly no matter how clean it looks.
The modality mix is the challenge
Fluorescence images are dark-field with bright nuclei; the histology tiles are stained purple on light backgrounds. Normalize per image and check your train/validation split covers both, or your model silently overfits one modality.
Known label noise
A handful of stage1 masks are imperfect (missed nuclei, rough edges). Community-corrected versions exist; for coursework the official masks are fine, but do not chase the last few AP points against noisy ground truth.
Image sizes vary widely
Sizes range from 256×256 up past 1024×1024. Random-crop to a fixed tile size for training rather than resizing whole images, so small nuclei keep their pixels.
Browse it before you download it.
DataTorch hosts a public copy of the full training set: all 670 images and their labels, no Kaggle account and no competition terms to accept. Browse the files in the browser, then download the dataset as a ZIP with a free account. When you outgrow the benchmark, the same tools let you annotate your own microscopy images and train on those.
The annotation platform for specialized imagery: review, score, and share datasets your team works from.
© 2026 DataTorch. All rights reserved.