BIRL: Benchmark on Image Registration methods with Landmark validation

Build Status CircleCI Build status codecov Codacy Badge CodeFactor Documentation Status Gitter

This project/framework is the key component of Automatic Non-rigid Histological Image Registration (ANHIR) challenge hosted at ISBI 2019 conference. The related discussion is hosted on

The project contains a set of sample images with related landmark annotations and experimental evaluation of state-of-the-art image registration methods.

The initial dataset of stained histological tissues is composed by image pairs of related sections (mainly, consecutive cuts). Each image in the pair is coloured with a different stain. The registration of those images is a challenging task due to both artefacts and deformations acquired during sample preparation and appearance differences due to staining. For evaluation, we have manually placed landmarks in each image pair. There are at least 40 uniformly spread over the tissue. We do not put any landmarks in the background. For more information about annotation creation and landmarks handling, we refer to the special repository - Dataset: histology landmarks.


The dataset is defined by a CSV file containing paths to target and sensed image and their related landmarks (see ./data_images/pairs-imgs-lnds_mix.csv). With the change of the cover table, the benchmarks can be used for any other image dataset.



The project contains the following folders:

Before benchmarks (pre-processing)

In the data_images folder we provide some sample images with landmarks for registration. These sample registration pairs are saved in data_images/pairs-imgs-lnds_mix.csv. You can create your own costume cover table for a given dataset (folder with images and landmarks) by hand or use script bm_dataset/ assuming the same folder structure <dataset>/<image-set>/<scale>/<images-and-landmarks> as for the CIMA dataset.

Prepare synthetic data

There is a script to generate synthetic data. Just set an initial image and their corresponding landmarks. The script will generate a set of geometrically deformed images mimicking different stains and compute the new related landmarks.

python bm_dataset/ \
    -i ./data_images/images/Rat_Kidney_HE.jpg \
    -l ./data_images/landmarks/Rat_Kidney_HE.csv \
    -o ./output/synth_dataset \
    -nb 5 --nb_workers 3 --visual

Creating an image-pairs table

When the synthetic datasets have been created, the cover csv file which contains the registration pairs (Reference and Moving image (landmarks)) is generated. Two modes are created: “first2all” for registering the first image to all others and “each2all” for registering each image to all other. (note A-B is the same as B-A)

python bm_dataset/ \
    -i ./data_images/synth_dataset/*.jpg \
    -l ./data_images/synth_dataset/*.csv \
    -csv ./data_images/cover_synth-dataset.csv \
    --mode each2all

Customize the images and landmarks

We offer a script for scaling images in to particular scales for example

python bm_dataset/ \
    -i "./data_images/rat-kidney_/scale-5pc/*.jpg" \
    -scales 10 -ext .png --nb_workers 2

We introduce an option how to randomly take only a subset (use nb_selected) of annotated landmarks and also add some synthetic point (filling points up to nb_total) which are across set aligned using estimate affine transformation.

python bm_dataset/ \
    -a ./data_images -d ./output \
    --nb_selected 0.5 --nb_total 200

Moreover we developed two additional script for converting large images, handling multiple tissue samples in single image and crop to wide background.

Experiments with included methods

Even though this framework is completely customizable we include several image registration methods commonly used in medical imaging.


Install methods and run benchmarks

For each registration method, different experiments can be performed independently using different values of the parameters or image pairs sets.

Sample execution of the “empty” benchmark template:

mkdir results
python birl/ \
    -c ./data_images/pairs-imgs-lnds_mix.csv \
    -o ./results \
    -config sample_config.yaml \
    --preprocessing hist-matching gray \
    --unique --visual

or with relative paths:

mkdir results
python birl/ \
    -c ./data_images/pairs-imgs-lnds_histol.csv \
    -d ./data_images \
    -o ./results \
    -config sample_config.yaml \
    --preprocessing gray hist-matching

The general Image Registration benchmarks contain couple required and optional parameters which are shared among ‘all’ methods/benchmarks. The brief description is following…

Required parameters:

Optional parameters:


Measure your computer performance using average execution time on several simple image registrations. The registration consists of loading images, denoising, feature detection, transform estimation and image warping.

python bm_experiments/ -o ./results

This script generate simple report exported in JSON file on given output path.

Included registration methods

For each benchmark experiment, the explanation about how to install and use a particular registration method is given in the documentation. Brief text at the top of each file.

Some more image registration methods integrated in ImageJ are listed in Registration.

Add custom registration method

The only limitation of adding costume image registration methods that it has to be launched from python script or command line. The new registration benchmark should be inherited from ImRegBenchmark as for example BmTemplate.

The benchmark workflow is the following:

  1. self._prepare() prepare the experiment, e.g. create experiment folder, copy configurations, etc.
  2. self._load_data() the load required data - the experiment cover file
  3. self._run() perform the sequence of experiments (optionally in parallel) and save experimental results (registration outputs and partial statistic) to common table
  4. self._summarise() summarize the statistic over all experiments and run optional visualisation of performed experiments.

General methods that should be overwritten:

The new image registration methods should be added to bm_experiments folder.

Re-evaluate experiment

In case you need to re-compute evaluation or add visualisation to existing experiment you can use the following script. The script require complete experiment folder with standard registration-results.scv (similar to registration pairs extended by experiment results).

python bm_experiments/ \
    -e ./results/BmUnwarpJ \


The project is using the standard BSD license.


For complete references see bibtex.

  1. Borovec, J., Munoz-Barrutia, A., & Kybic, J. (2018). Benchmarking of image registration methods for differently stained histological slides. In IEEE International Conference on Image Processing (ICIP) (pp. 3368–3372), Athens. DOI: 10.1109/ICIP.2018.8451040

Appendix - Useful information

Configure local environment

Create your own local environment, for more information see the User Guide, and install dependencies requirements.txt contains a list of packages and can be installed as

@duda:~$ cd BIRL 
@duda:~/BIRL$ virtualenv env
@duda:~/BIRL$ source env/bin/activate  
(env)@duda:~/BIRL$ pip install -r requirements.txt  
(env)@duda:~/BIRL$ python ...

and in the end, terminating…

(env)@duda:~$ deactivate

Running docString tests - documentation and samples of doc string on pymotw and python/docs

Listing dataset in command line

find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/" >> dataset.txt