Question about evaluation metric calculation

[iwuqing]

Question about evaluation metric calculation

[juhha1]

Thanks a lot for giving detailed answers and codes. It helped!

[iwuqing]

Thanks for your attention to our work!

These quantitative metrics we used are implemented based on some open-source Python libraries. The links are:

PSNR: https://scikit-image.org/docs/stable/api/skimage.metrics.html#skimage.metrics.peak_signal_noise_ratio
SSIM: https://scikit-image.org/docs/stable/api/skimage.metrics.html#skimage.metrics.structural_similarity
LPIPS：https://github.com/richzhang/PerceptualSimilarity
PSI：https://github.com/feichtenhofer/PSI
LPC-SI：https://ece.uwaterloo.ca/~z70wang/research/lpcsi/

Note that the first two (PSNR and SSIM) can be directly calculated for 3D volumes, while the last three (LPIPS, PSI, and LPC-SI) are designed for 2D slices. In our cases, we compute them by a slice-by-slice strategy that includes three steps:

1. Given a 3D volume, we extract its 2D MR slices from three orthogonal directions (axial, sagittal, and coronal directions).
2. We compute the scores for each 2D MR slice.
3. We average the scores of all the 2D MR slices to calculate the final scores.

The following code snippet is to compute LPIPS using the slice-by-slice strategy:

import SimpleITK as sitk
import numpy as np
import torchs
import lpips

def compute_lpips(gt_slice, recon_slice):
    lpips_alex = lpips.LPIPS(net = 'alex')
    h, w = gt_slice.shape
    gt_image = np.zeros((1, 3, h, w))
    recon_image = np.zeros((1, 3, h, w))
    for c in range(3):
        gt_image[:, c, :, :] = gt_slice
        recon_image[:, c, :, :] = recon_slice
    lpips_value = lpips_alex(torch.tensor(gt_image).to(torch.float32), torch.tensor(recon_image).to(torch.float32))
    return lpips_value

gt_volume_path = ''
recon_volume_path = ''

gt_volume = sitk.GetArrayFromImage(sitk.ReadImage(gt_volume_path))
recon_volume = sitk.GetArrayFromImage(sitk.ReadImage(recon_volume_path))

# In any volume, the center region often includes more image information than the margin region.
# Therefore, we extract 10 2D slices in the center from three directions, respectively.
# Here the volumes are 264*264*264 size
i_strart, i_end = 127, 137

lpips_value = 0.

for i in range(i_strart, i_end):
    # direction 1
    gt_slice = gt_volume[i, :, :]
    recon_slice = recon_volume[i, :, :]
    lpips_value += compute_lpips(gt_slice, recon_slice)
    # direction 2
    gt_slice = gt_volume[:, i, :]
    recon_slice = recon_volume[:, i, :]
    lpips_value += compute_lpips(gt_slice, recon_slice)
    # direction 3
    gt_slice = gt_volume[:, :, i]
    recon_slice = recon_volume[:, :, i]
    lpips_value += compute_lpips(gt_slice, recon_slice)

lpips_value = lpips_value / 30
print('LPIPS:', lpips_value)