brachiohyup/sipsa

CVPR2021, Pan-sharpening

SIPSA

SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery CVPR2021 Oral Accepted.

This is the official repository of "SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery", CVPR 2021 oral paper.

We provide the training code without the WorldView-3 dataset. If you find this repository useful, please consider citing our paper.

Reference

Jaehyup Lee, Soomin Seo, and Munchurl Kim, "SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery", CVPR 2021 oral paper.

Abstract: Pan-sharpening is a process of merging a highresolution (HR) panchromatic (PAN) image and its corresponding low-resolution (LR) multi-spectral (MS) image to create an HR-MS and pan-sharpened image. However, due to the different sensors’ locations, characteristics and acquisition time, PAN and MS image pairs often tend to have various amounts of misalignment. Conventional deeplearning-based methods that were trained with such misaligned PAN-MS image pairs suffer from diverse artifacts such as double-edge and blur artifacts in the resultant PANsharpened images. In this paper, we propose a novel framework called shift-invariant pan-sharpening with moving object alignment (SIPSA-Net) which is the first method to take into account such large misalignment of moving object regions for PAN sharpening. The SISPA-Net has a feature alignment module (FAM) that can adjust one feature to be aligned to another feature, even between the two different PAN and MS domains. For better alignment in pansharpened images, a shift-invariant spectral loss is newly designed, which ignores the inherent misalignment in the original MS input, thereby having the same effect as optimizing the spectral loss with a well-aligned MS image. Extensive experimental results show that our SIPSA-Net can generate pan-sharpened images with remarkable improvements in terms of visual quality and alignment, compared to the state-of-the-art methods.

Please check the attached network figure. I attched the wrong network figure on the arxiv and CVPR reposit by my mistake.

Requirements :

numpy, PIL, tensorflow 1.13, time, tifffile, datetime, socket

I have never run this codes with other version of tensorflow.

The codes are created by Jaehyup Lee, Jaeseok Choi, and Soomin Seo.

Contact

Please contact us via any of the following emails: [email protected] or leave a note in the issues tab.

sipsa's Issues

Can you share your rained model?

I would like to test you model on my WV3 data. Is it possible to share the model and a simple script how to run it on pairof WV3 MSI with PAN?
thanks

Some questions about the code.

What is the purpose of NIR and mask in the code? It seems that these two were not mentioned in your paper ?

Besides, what does this judgment do? Doesn't seem to change any behavior:

        if isTraining:
            ms = ms
            pan = pan
            ms2 = ms2
        else:
            ms = ms
            pan = pan
            ms2 = ms2

About the dataset

Thanks for your sharing and sorry to bother you: I am now studying on pan-sharpening, and I am interest in your work. However, I could not find the dataset that used in your experiments, could you please sent them to me? Thanks a lot! My email is: [email protected].