Registrator

Registrator in a nutshell.

  1. Plugin to align among each other the slices of a stack;

  2. This plugin is not multichannel;

  3. Python API reference: bmiptools.transformation.alignment.registrator.Registrator.

This plugin can be used to align among each other the slices of the input stack, in order to get a proper 3d reconstruction. The registration procedure of this plugin consist in at most two steps:

  1. rigid registration, where a global affine transformations is applied to each slice to match it geometrically with the next. The rigid registration can happen in two ways: either via ECC based matching algorithm [Evangelidis2008] (slower but it is more precise in principle) or via phase correlation based matching algorithm [Reddy1996] (faster but it may be less precise).

  2. non-rigid registration, which is used to eventually refine the result of the previous step, where a pixelwise transformation is applied to each pixel of a slice to match it with the corresponding pixel in the next slice (keeping into account the pixel surrounding) based on optical-flow-based matching [LeBesnerais2005]. This step is optional.

The Python API reference of the plugin is bmiptools.transformation.alignment.registrator.Registrator.

Transformation dictionary

The transformation dictionary for this plugin look like this.

{'load_existing_registration': False,
'loading_path': ' ',
'registration_algorithm':'ECC',
'padding_val': 0,
'destandardize': True,
'template_lh_boundary_factor': 1,
'template_rb_boundary_factor': 1,
'ECC_registration_setting': {'n_iterations': 5000,
                             'termination_eps': 1e-10,
                             'motion_model':'Translation',
                             'ecc_threshold': 0.7},
'phase_correlation_registration_setting': {'motion_model': 'Translation',
                                           'phase_corr_threshold': 0.4,
                                           },
'opt_bounding_box': {'use_bounding_box': True,
                     'y_limits_bbox': [-500,None],
                     'x_limits_bbox': [500,1500]},
'refine_with_optical_flow': False,
'OF_setting':{'optical_flow_attachment': 5,
              'save_mod_OF': False,
              'mod_OF_saving_path': ''},
'save_fitted_registration': False,
'saving_path': ' '
}

The plugin-specific parameters contained in this dictionary are:

  • load_existing_registration: if True an existing registration produced by this plugin is loaded and all the other fields below except loading_path are ignored.

  • loading_path: contains the path to the files containing the existing registration parameters.

  • registration_algorithm: field where one has to specify the algorithm used to perform the rigid registration. The possible options are:

    • 'ECC'

    • 'Phase_correlation'

  • padding_val: it is the value used for padding the images in order to reach a certain shape during the application of the registration.

  • destandardize: when True at the end of the registration ,the image is destandardized (image standardization take place before the optimization of the registration algorithm and is done according to the 0/1 mode of the Standardizer plugin).

  • ECC_registration_setting: contains the setting for the ECC registration algorithm. It is a dictionary and has to be specified as below:

    • n_iterations, is number of iterations used for the maximization of the ECC loss.

    • termination_eps, is epsilon value used to determine the convergence of the ECC maximization algorithm: if the difference between the ECC values after two iterations is less then this value, then the maximization stops.

    • motion_model, is the kind of motion model used for the estimation of the parameters used for the registration, and can be equal to

      • 'Translation';

      • 'Euclidean' ( i.e. rotation + translation);

      • 'Affine'.

    • template_lh_boundary_factor, it is the left/high boundary factor for the template window definition (recommended value: 1)

    • template_rb_boundary_factor, it is right/bottom boundary factor for the template window definition (recommended value: 1)

    • ecc_threshold, is the threshold on the ECC value at the end of the maximization procedure below which a two steps estimation procedure for the estimation of the transformation parameters is run.

  • phase_correlation_registration_setting: contains the setting for the registration algorithm based on the phase correlation techniques. It is a dictionary having the keys below:

    • motion_model, do not change. currently only 'Translation' is possible.

    • phase_corr_threshold, is threshold on the normalized correlation below which the two steps registration optimization is executed.

  • refine_with_optical_flow: if True a final refinement with optical flow registration is applied at after that the rigid registration has been applied on the stack.

  • OF_setting: is a dictionary containing the setting of the optical flow registration. This dictionary has to be specified as follow:

    • optical_flow_attachment, it is the attachment parameter of the optical flow registration algorithm.

    • save_mod_OF, if True the modulus of the optical flow field is saved.

    • mod_OF_saving_path, is the path where the modulus of the optical flow registration field is saved. If the above field is False this field is ignored.

  • save_fitted_registration: if True the parameters estimated for the rigid-registration are saved.

  • saving_path: is the path where the registration parameters are saved.

Further details useful the the usage of this plugin with the Python API can be found in the __init__ method of the class Registrator.

Use case

The typical use of this plugin are:

  1. Align the slices of a stack in order to get a proper 3d reconstruction.

Tip

The following things turn out to be useful, from a practical point of view.

  1. Using the bounding box the the time necessary for the estimation of the rigid registration can be reduced. By the way the time for the refined optical flow registration (if used to refine the result), does not change.

  2. The parameter describing the rigid registration can be saved and loaded in a later time, allowing the registration of a stack by using the parameter estimated using a different stack. This may be particularly useful in case one has to register two stack which are produced at the time from the same sample but with different imaging techniques.

  3. It can be reasonable to fit this plugin in the beginning of the image processing pipeline but apply it only at the end. This because this plugin change the dimension of the images, making the stack bigger. This means that more RAM memory is required and the overall computation is increased. Therefore, if there is no need to use the 3-dimensional information for the application of a plugin, is a good strategy. Note that practically all the image processing plugin in bmiptool act on the slices directly, and works without the need of information coming from (the local structures of) the other slices.

  4. The use of the 'Translation' motion model advised for the rigid-registration step.

  5. To achieve a non-rigid registration, the a rigid registration is always done before. This is done because the non-rigid registration is not good in estimating big translations, which can be vary easily estimated with rigid methods. Since the non-rigid method is particularly slow and is used to refine the registration, it is recommended to use Phase_correlation which is fast and enough to get a good result.

  6. In the ECC_registration_setting, if there are no particular reasons, template_lh_boundary_factor and template_rb_boundary_factor should not be changed.

Application example

As example consider the portion of a FIB-SEM stack of a biological sample, visualized as animated gif (saving mode available in the python API, see bmiptools.stack.Stack.save_as_gif()), in order to get the feeling of the 3-dimensional structure of the sample. Before any registration the stack look like below.

../_images/pre_registration.gif

A registration based on Phase_correlation only and using practically all the default parameters of the plugin (i.e. the ones in the empty_transformation_dictionary, but disabling the use of the bounding box only, due to the small dimension of the image) would give the result below.

../_images/post_registration.gif

When after the Phase_correlation step, also the refinement with optical flow is applied (i.e. setting refine_with_optical_flow = True in the transformation dictionary of the plugin) would give the result below.

../_images/post_registration_refined.gif

Note that after the optical flow refinement, the central structure in the top-right corner of the image changes more smoothly from one slice to the other. The other points of the slices are practically unchanged, showing the local nature of the refinement step of the registrator plugin.

Note

The script used to produce the images displayed can be found here. To reproduce the images showed above one may consult the examples/documentation_scritps folder, where is explained how to run the example scripts and where one can find all the necessary input data.

Implementation details

A registration algorithm in general takes as input two things: an input image and a reference image. The reference image is the reference on which the input image is aligned. The registration algorithms in this plugin consist essentially in two steps:

  1. In the first step a map between the pixel positions of the input image and the pixel position is established. This map can be the same for all pixels (in case of rigid registration) or be pixel dependent (in the non-rigid case). These maps are parametrized in a suitable manner, and the best parameters for this map are found by various optimization techniques.

  2. In the second step the input image is evaluated on the new pixel position obtained from the map derived in before. This evaluation is done by computing the pixel value in the new position via interpolation.

In what follows \(\phi[\alpha]\) will denote the registration procedure described by the two steps above, where \(\alpha\) are the registration parameters. For the rigid registration, implemented in this plugin it may take the following form

  • for translation, \(\phi[\alpha](I(x)) = I(x+t)\), where \(t\) is a suitable translation vector.

  • for a generic euclidean transformation, \(\phi[\alpha](I(x)) = I(Rx+t)\), where \(t\) and \(R\) is a suitable translation vector and rotation matrix respectively.

  • for a generic affine transformation, \(\phi[\alpha](I(x)) = I(Ax+t)\), where \(t\) and \(A\) is a suitable translation vector and (invertible) matrix respectively.

For non-rigid transformations one can formally write \(\phi[\alpha](I(x)) = I(f_\alpha(x))\), where \(f_\alpha\) is a suitable parameterizable function which may depends also on other image pixels and not only on \(x\).

Given a \(K \times J \times I\) stack \(S(k,j,i)\) for each slice \(S[k](j,i)\) the output of the registration algorithm is given as follow

\[S[k](j,i) \rightarrow S'_{output}[k](j',i') = \phi[\alpha](S[k](j,i))\]

where \(S'_{output}[k](j',i')\) is the \(k\)-th slice of the \(K \times J' \times I'\) output stack \(S'_{output}(k,j',i')\). The input and output stack have different size: indeed this plugin change the image dimension in order to accommodate the image “movements” due to the registration.

ECC

The ECC (Enhanced Correlation Coefficient) registration algorithm [Evangelidis2008] is a gradient-based registration technique. Given an input image \(I(x)\) and a reference image \(I_{ref}(x)\), the parameters of \(\phi[\alpha]\) are found by minimizing the following differentiable loss

\[\mathcal{L}_{ECC}(\alpha) = \sum_x\|\frac{I_{ref}(x)}{\|I_{ref}(x)\|}-\frac{\phi[\alpha](I(x))}{\|\phi[\alpha](I(x))\|}\|^2\]

where \(\|\cdot \|\) is the euclidean norm. The implementation used in this plugin is based on the one available in openCV.

Attention

If the optimization using this loss fails for a pair of slices in a stack (i.e. if the value of the loss function remains above a certain threshold \(ECC_{th}\)), a two step estimation is done: a first translation is estimated using the Phase correlation routine on a smaller part of the image, and later a refined estimation is done using again the ECC on the whole image. If the second estimation is still unsuccessful, only the result of the first step is used.

Phase correlation

The phase correlation registration algorithm [Reddy1996] simply compute the normalized cross-correlation between the input image and the reference one. Since the cross-correlation can be computed with a simple multiplication in Fourier space, by using Fourier shift theorem, if the input and reference image are linked by a rigid translation,the cross-correlation would be just a phase factor. When transformed back to the real space, the cross-correlation corresponds to a delta function centered in a certain point \(p\). The vector linking \(p\) with the center of the image correspond to the translation linking the two images.

In this plugin only translation vectors can be estimated with this method, despite in principle it is possible to estimate also rotation angles by means of a change of coordinates. The implementation of this algorithm used in this plugin is the one of openCV

Attention

If the optimization using this loss fails for a pair of slices in a stack (i.e. if the value of normalized cross-correlation remains above a certain threshold \(PC_{th}\)), a two step estimation is done: a first translation is estimated using the Phase correlation routine on a smaller part of the image, and later a refined estimation is done using again the Phase correlation routine on the whole image. If the second estimation is still unsuccessful, only the result of the first step is used.

Optical flow

The Lucas-Kanade optical flow registration algorithm [LeBesnerais2005] is another gradient based optimization techniques allowing for non-rigid registration. It is based on the minimization of the following loss function

\[\mathcal{L}_{OF}(\alpha) = \sum_{(i,j)} \sum_{(j',i') \in G(j,i)} \left[I_{ref}(j',i')-\tilde{I}(j'+f_j[\alpha](j',i'), i'+f_i[\alpha](j',i') )\right]^2\]

where \(\tilde{I}\) is the interpolated version of the input image \(I\), so that it can be evaluated on a generic point \((j+f_j[\alpha](j,i), i+f_i[\alpha](j,i) )\), which does not lie necessarily on the pixel grid. The non rigid character of this algorithm lies in the second sum, since \(G(j,i)\) is a patch centred in \((j,i)\). The goal of the optimization problem is to find the parameters \(\alpha\) defining the optical flow vector field, which tells how the position of the pixels in the input image and the one in the reference image are mapped. In particular, this quantity can be useful

\[\begin{split}\left( \begin{array}{c} j_{input}\\ i_{input}\\ \end{array} \right) = \left( \begin{array}{c} j_{ref}+f_\alpha(j_{ref},i_{ref})\\ i_{ref}+f_\alpha(j_{ref},i_{ref})\\ \end{array} \right).\end{split}\]

The two components of the optical flow field, \(f_\alpha(j_{ref},i_{ref})\) and \(f_\alpha(j_{ref},i_{ref})\), can be thought as two 2d images where the “movements” between the input and the reference image can be visualized. The magnitude of these “movements” can be summarized in the image containing the magnitude of the optical flow field.

This plugin uses the skimage implementation of this algorithm.

Further details

Websites:

Articles:

[Evangelidis2008] (1,2)

“Parametric Image Alignment using Enhanced Correlation Coefficient Maximization” - G.D. Evangelidis, E.Z. Psarakis - IEEE Trans. on PAMI, vol. 30, no. 10, 2008

[Reddy1996] (1,2)

“An FFT-based technique for translation, rotation, and scale-invariant image registration.” - Reddy B.S., and B. N. Chatterji. - IEEE transactions on image processing : a publication of the IEEE Signal Processing Society 5 8 (1996): 1266-71 .

[LeBesnerais2005] (1,2)

“Dense optical flow by iterative local window registration” - G. Le Besnerais and F. Champagnat, - IEEE International Conference on Image Processing 2005, 2005, pp. I-137, doi: 10.1109/ICIP.2005.1529706.