nicalab/reals

Robust and Efficient Image Alignment through Simultaneous Low Rank and Sparse Decomposition

Robust and Efficient Image Alignment through Simultaneous Low Rank and Sparse Decomposition

This repository is the official implementation of Robust and Efficient Image Alignment through Simultaneous Low Rank and Sparse Decomposition. This paper is accepted in WACV 2023.

Requirements

To install requirements:

conda create -n reals python=3.8.10
conda activate reals
pip install -r requirements.txt

Data for REALS is required for execution of the code.

• 2-D Zebrafish brain data: Static data with size [512(x), 512(y), 60(z)]. Additional perturbations or noise can be applied.
• 3-D Zebrafish brain data: Data with size [256(x), 512(y), 48(z), 60(t)]. Random translation and rotation are applied synthetically.
• Mouse brain data: Data with size [480(x), 752(y), 1000(t)]. Y.tif is the original video, and Y_DoG.tif is high pass filtered video.
• 2-D Zebrafish brain data with drift: Data with size [256(x), 512(y), 600(t)]. Original video has small drift.

To run run_reals.py, you have to generate data with perturbation from static 2-D Zebrafish brain data.

• generate_oscillation.py generates randomly perturbed data with corresponding transformation from static 2-D Zebrafish brain data.

If you want to add noise to above generated data,

• generate_gaussian_noise.py generates data with gaussian noise from perturbed data.
• generate_poisson_noise.py generates data with poisson noise from perturbed data.

Training

run_reals.py can reproduce most of the results in the paper. Run following command:

python -m scripts.run_reals --type normal

To perform REALS on 3-D Zebrafish brain data, run following command:

python -m scripts.run_reals_3d

We also provide the mini-batch and multi-resolution version of REALS in run_reals_minibatch.py, run_reals_multi.py respectively.

After running REALS on one sample data, there will be composite transformation tau, original images Y, aligned images L, low-rank component L_stat, and sparse component S in the resulting directory.

Evaluation

We provide 3 sample codes for evaluation. However, the codes can perfectly reproduce the results in original paper only when required results are fully available. So, it may require to modify some parts of the code to analyze the result with below codes.

• eval_heatmap.py shows the alignment inconsistency heatmap.
• eval_barplot.py shows the alignment inconsistency bar plot of REALS. It is to check performance with different batch size.
• eval_timeplot.py shows the (time, alignment inconsistency) plot. You need to save time and transformation at each iteration to plot the result.

Results

We will make it available after the review.

Contributing

The base code is from BEAR.

reals's Issues

pytorch version incompatible with 3090 GPU

REALS/requirements.txt

Line 69 in 3459515

torch==1.8.1

When running python -m scripts.run_reals --type normal, I encounter the following warning: The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

As well as:
RuntimeError: CUDA error: no kernal image is available for the execution on the device.

Adding the suggestion of copying the installation notes from SUPPORT:
Install Pytorch with the version compatible with your OS and platform from https://pytorch.org/get-started/locally/

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

Running python -m scripts.run_reals --type normal correctly targets dedicated GPU for processing after performing the suggestion.

Add optional selection for cuda device

REALS/scripts/run_reals.py

Line 19 in 3459515

parser.add_argument('--device', type=str, default='cuda:0')

If making into a GUI, might be useful to select a specific GPU if using a second GPU for processing.

setuptools==52.0.0.post20210125 does not exist in PyPI

REALS/requirements.txt

Line 62 in 3459515

setuptools==52.0.0.post20210125

setuptools==52.0.0.post20210125 cannot be completed as this version doesn't exist in the PyPI repository. Likely a carryover from a previously yanked version 51.0.0.post20210125 that was misnamed.

Changing to 52.0.0 solves the error and doesn't throw and error when calling python -m script.run_reals --type normal

numpy version dependency conflict in requirements.txt

REALS/requirements.txt

Line 36 in 3459515

numpy==1.20.2

This version of numpy throws the following dependency conflict:
ERROR: Cannot install -r requirements.txt (line 20), -r requirements.txt (line 22), -r requirements.txt (line 28), -r requirements.txt (line 29), -r requirements.txt (line 30), -r requirements.txt (line 31) and numpy==1.20.2 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested numpy==1.20.2
h5py 3.3.0 depends on numpy>=1.17.5; python_version == "3.8"
imageio 2.9.0 depends on numpy
mat73 0.50 depends on numpy
matplotlib 3.4.2 depends on numpy>=1.16
mkl-fft 1.3.0 depends on numpy>=1.16
mkl-random 1.2.2 depends on numpy<1.23.0 and >=1.22.3

Changing to numpy==1.22.3 solves the issue without throwing an error calling python -m scripts.run_reals --type normal

I haven't examined outputs to confirm this and the setuptools change yet.