Run a Hugging Face model on your images from the annotator

What you will build

A button in the annotator that runs a model from the Hugging Face Hub on the image you are looking at. Press it, and seconds later (minutes on the first run, see below) the model's boxes or polygons appear on the file, ready for you to keep or fix.

This takes about 15 minutes and needs:

• a DataTorch project with a few photos of everyday scenes, such as a street or a room (upload one under Storages with Upload files if the project has none),
• a Mac or Linux machine for the runner, with Python 3.10 or later, git, a few GB of free disk and access to GitHub and the Hugging Face Hub (no GPU needed),
• the id of a Hugging Face model. This page uses facebook/detr-resnet-50, an object detector.

DETR knows everyday objects such as person, car and bicycle, not cells, so try it on an everyday photo first. For microscopy or other images of your own, paste a Hub model trained on images like them. The action reads every image as 8-bit RGB, so for 16-bit or multi-page TIFFs, run your own model instead.

Step 1: connect a runner

The model runs on your own machine, not on ours. A runner is the small program that takes pipeline steps from your project and runs them there.

1. 1

   Install the package

   On the machine you want to use, run python3 -m pip install -U datatorch, then datatorch --version to check it is 0.8.0 or later. If pip installs an older version, that Python is older than 3.10 (the python3 that ships with macOS is 3.9).

2. 2

   Register the runner

   Open Runners in the project sidebar and press Install runner. You have installed the package already, so run the other three commands: datatorch login (on the page it opens, press New Key, then Create, and paste the key), datatorch runner create (give the runner a name), then datatorch runner start. Leave that terminal open: the runner works only while the command runs.

3. 3

   Add it to the project

   A new runner belongs to your account, not to the project yet. On the Runners page, press Manage runners, tick your runner and press Save. It now shows in the project's list, with a green dot while it is online.

Step 2: make the pipeline start from a button

1. 1

   Create a pipeline

   Open Pipelines in the project sidebar and press New pipeline. Give it a name, for example "Detect with DETR".

2. 2

   Choose the trigger

   On the same screen, under When should it run? choose From the annotator, then press Continue.

3. 3

   Customize the button

   The Button title starts as the pipeline's name. Pick an icon, leave Selection on The whole file (the Hugging Face action always reads the whole image) and press Continue.

Step 3: add the Hugging Face action

1. 1

   Pick the action

   Choose an action opens on its own, on the On my runner tab. Click DataTorch library, search for hugging and pick huggingface. The editor puts a Download file step in front of it that hands it the image, so leave that step in place.

2. 2

   Paste the model id

   Click the huggingface step on the canvas to open its settings. Under Inputs it gives the action, click set a value beside model, paste the model's id and press Enter. The id is the part of the model's address after huggingface.co/, for example facebook/detr-resnet-50.

Object detection is the default. For a segmentation or classification model, click 3 optional inputs not set, then set a value beside task and choose image-segmentation or image-classification. The threshold input is there too: it keeps only detections scored at or above it, 0.5 by default.

Step 4: check the labels

A COCO-trained detector such as DETR returns names like person, car or bicycle. When you pick the Hugging Face action, the editor sets the pipeline to create any of these labels your project does not have yet, so the first run adds them for you. To keep your label list as it is, open the step and set Labels the project does not have to Refuse the run and name them. A run that returns a label the project does not have then places nothing and names the missing labels.

Step 5: press the button

1. 1

   Save the pipeline

   Press Save at the top right of the editor (or Cmd+S on a Mac, Ctrl+S elsewhere). The button appears in the annotator only once the pipeline is saved, and leaving the editor before then discards it.

2. 2

   Open an image

   Click one of your photos under Storages to open it in the annotator (reload the page if it was already open). Your button is at the bottom of the left toolbar, below the drawing tools, with the icon you picked. Hover it to see its title. Start runs from here: Run in the pipeline editor starts a run with no file, which fails.

3. 3

   Press the button

   The run starts on your runner. The first run installs the action's packages (PyTorch and Transformers) and downloads the model, which can take several minutes while the annotator shows Running pipeline. Later runs reuse both, and a different model only downloads its own weights.

4. 4

   Review the result

   The model's shapes appear on the file. Select one and the Annotation Metadata tab shows the run that made it and the model's score. Keep the good ones, fix the close ones, delete the wrong ones and draw anything the model missed.

If nothing shows up

In the annotator, open the Pipelines tab beside Tool in the lower right panel (Window > Pipelines if your layout does not show it). It lists the latest runs of every pipeline in the project, newest first. Click your pipeline's latest run to see its steps and what it reported.

• A red Pipeline Error says no online runner is eligible. No runner this job may use is online. The tab's header counts the project's runners, and if you pinned the job under Runners, only the runner named there counts. If yours is missing from the project's Runners page, add it with Manage runners as in Step 1. If it is listed but offline, run datatorch runner start on that machine.
• The run finished but placed nothing. If the run names a reason in red, such as missing labels (only when the step refuses them), a signed-off dataset or a permission the person who pressed the button lacks, fix it and press the button again. If it names nothing, the model found nothing above its threshold (click the huggingface step in the run to see its log, which says Found 0 annotation(s)). Lower the threshold, or try a model trained on images like yours.
• The run failed (red). Click the huggingface step in the run to read its log. It says what went wrong, for example a model id the Hub does not have, a repository with several weights files, or the task to set on the step.
• Too many boxes, or too few. Each press adds a new set of shapes and never replaces the last one, so delete the last run's shapes first. Then open the pipeline, press Edit, click the huggingface step and change threshold (behind the optional inputs not set link until you set it). Higher gives fewer boxes, lower gives more. Press Save, then press the button in the annotator again.

Where the result goes

The shapes land on the file the button was pressed on, written as the person who pressed it. They land only if that person may add annotations (and create labels, when the run adds new ones). If not, the run places nothing and says why. A classification model places no shapes: its top label and score show on the run in the Pipelines tab.

Apart from creating labels, placing the shapes is all the pipeline does with the result. To send it somewhere else as well, such as a Slack message or an email, add ordinary steps to the pipeline.

Use your own model

The same button can run your own code instead. In Choose an action, open On my runner and pick one of your own actions. Each is a folder at actions/local/<name>/<tag> inside the runner's folder (~/.config/datatorch/runner on Linux, ~/Library/Application Support/DataTorch/runner on a Mac). The code stays in that folder. DataTorch keeps the action's yaml file so the editor can list it, plus the outputs and logs of each run. An action that ships a requirements.txt gets its own Python environment on the runner. The pipelines release notes show how to write a new action.