arplaboratory/satellite-thermal-geo-localization

[IROS 2023] Official repository for "Long-range UAV Thermal Geo-localization with Satellite Imagery"

satellite-thermal-geo-localization

This is the official repository for Long-range UAV Thermal Geo-localization with Satellite Imagery.

@INPROCEEDINGS{10342068,
  author={Xiao, Jiuhong and Tortei, Daniel and Roura, Eloy and Loianno, Giuseppe},
  booktitle={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={Long-Range UAV Thermal Geo-Localization with Satellite Imagery}, 
  year={2023},
  volume={},
  number={},
  pages={5820-5827},
  doi={10.1109/IROS55552.2023.10342068}}

Developer: Jiuhong Xiao
Affiliation: NYU ARPL
Maintainer: Jiuhong Xiao ([email protected])

Dataset

Dataset link: Download

The dataset folder should be created in the root folder with the following structure

satellite-thermal-geo-localization/datasets/satellite_0_thermalmapping_135/
├── test_database.h5
├── test_queries.h5
├── train_database.h5
├── train_queries.h5
├── extended_database.h5
├── extended_queries.h5 # Need to generate by yourself using trained TGM
├── val_database.h5
└── val_queries.h5

Check your md5sum after downloading the dataset:

3d2faea188995dd687c77178621b6fc7  extended_database.h5
7ef56acc4a746e7bafe1df91131d7737  test_database.h5
f891d9feca5f4252169a811e8c8e648d  test_queries.h5
f075a7a6db5d5d61be88d252f7d6e05b  train_database.h5
b44ee39173bf356b24690ed6933a6792  train_queries.h5
31923c28dd074ddaacf0c463681f7d2b  val_database.h5
fdcb2e12d9a29b8d20a4cbd88bfe430c  val_queries.h5

Conda Environment Setup

Our repository requires a conda environment. Relevant packages are listed in env.yml. Run the following command to setup the conda environment.

conda env create -f env.yml

Training

You can find the training scripts and evaluation scripts in scripts folder. The scripts is for slurm system to submit sbatch job. If you want to run bash command, change the suffix from sbatch to sh and run with bash.

TGM training

1. To train Thermal Generate Module (TGM), use one of the following scripts:

./scripts/train_bing_thermal_translation_100.sbatch # Best choice
./scripts/train_bing_thermal_translation_10.sbatch
./scripts/train_bing_thermal_translation_100_noncontrast.sbatch
./scripts/train_bing_thermal_translation_10_noncontrast.sbatch

After training TGM, find your model folder in ./logs/default/satellite_0_thermalmapping135-datetime
The satellite_0_thermalmapping135-datetime is your TGM_model_folder_name.

2. To generate thermal images from extended_database.h5, run one of the following scripts:

./eval_satellite_translation.sh   TGM_model_folder_name # Best choice
./eval_satellite_translation_nocontrast.sh   TGM_model_folder_name

After running this scripts, you can find a file train_queries.h5 in ./test/default/model_folder_name/satellite_0_thermalmapping135-datetime
Rename it to extended_queries.h5 and move it to ./datasets/satellite_0_thermalmapping135.

3. Rename the dataset folder to one of the following:

./datasets/satellite_0_thermalmapping_135/                  #./scripts/train_bing_thermal_translation_10.sbatch
./datasets/satellite_0_thermalmapping_135_100/              #./scripts/train_bing_thermal_translation_100.sbatch
./datasets/satellite_0_thermalmapping_135_nocontrast/       #./scripts/train_bing_thermal_translation_10_nocontrast.sbatch
./datasets/satellite_0_thermalmapping_135_100_nocontrast/    #./scripts/train_bing_thermal_translation_100_nocontrast.sbatch

The dataset name depends on the training script you use, which is listed on the right of each dataset name. This is to avoid confusion about which TGM setting is used to generate.

SGM training

To train Satellite-thermal Geo-localization Module (SGM), use one of the scripts in ./scripts per name, for example:

./scripts/train_bing_thermal_partial_conv4096_bn_lr_wd_DANN_after_ld_dp_contrast_extended_100.sbatch # Best choice

After training SGM, find your model folder in ./logs/default/satellite_0_thermalmapping135-datetime-uuid
The satellite_0_thermalmapping135-datetime-uuid is your SGM_model_folder_name.

Evaluation

To evaluate SGM, use one of the following scripts:

./scripts/eval.sbatch   SGM_model_folder_name   backbone_name
./scripts/eval_nocontrast.sbatch   SGM_model_folder_name   backbone_name

Note that running evaluation scripts requires two arguments: SGM_model_folder_name and backbone_name. The typical backbone we use is resnet18conv4.

Find the test results in in ./test/default/model_folder_name/satellite_0_thermalmapping135-datetime. There will be two folders. This first reports R@1 and R@5. The second reports R_512@1, R_512@5 and L^512_2.

Acknowledgement

Our implementation refers to the following repositories and appreciate their excellent work.

https://github.com/gmberton/deep-visual-geo-localization-benchmark
https://github.com/fungtion/DANN
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

satellite-thermal-geo-localization's Issues

Error downloading datasets

Hi! I would like to test and validate your work. However, I am experiencing some issue with the datasets. Here is the error I get when extracting the zip file:

I am working with Ubuntu 20.04.

Regarding package installations

I am unable to install packages through the env.yml file despite repeated attempts. Could you please share the requirement.txt file or specify the required packages separately?

Paper link

My reproduction results are low

Hi,
Thanks for your impressive work. When I tried to reproduce this work, it was difficult to achieve the results reported in the paper.
I experimented with the same configuration as the code. This is my training configuration:

2024-02-28 23:36:31   Arguments: Namespace(use_extended_data=True, G_test_norm='batch', G_tanh=False, GAN_epochs_decay=0, GAN_lr_policy='linear', GAN_resize=[512, 512], GAN_mode='lsgan', GAN_upsample='bilinear', GAN_save_freq=0, GAN_norm='batch', G_contrast=True, G_gray=False, G_loss_lambda=100.0, visual_all=False, DA_only_positive=True, D_net='none', G_net='none', lambda_DA=0.1, DA='DANN_after', add_bn=True, remove_relu=False, use_faiss_gpu=True, prior_location_threshold=-1, use_best_n=1, separate_branch=False, weight_decay=0.0, train_batch_size=12, infer_batch_size=48, criterion='triplet', margin=0.1, epochs_num=100, patience=100, lr=0.0001, lr_crn_layer=0.005, lr_crn_net=0.0005, optim='adam', cache_refresh_rate=1000, queries_per_epoch=5000, negs_num_per_query=10, neg_samples_num=1000, mining='partial', backbone='resnet18conv4', l2='before_pool', aggregation='netvlad', netvlad_clusters=64, pca_dim=None, num_non_local=1, non_local=False, channel_bottleneck=128, fc_output_dim=None, conv_output_dim=4096, unfreeze=False, pretrain='imagenet', off_the_shelf='imagenet', trunc_te=None, freeze_te=None, seed=0, resume=None, device='cuda', num_workers=8, resize=[512, 512], test_method='hard_resize', majority_weight=0.01, efficient_ram_testing=False, val_positive_dist_threshold=50, train_positives_dist_threshold=35, recall_values=[1, 5, 10, 20], brightness=None, contrast=None, saturation=None, hue=None, rand_perspective=None, horizontal_flip=False, random_resized_crop=None, random_rotation=None, datasets_folder='datasets', dataset_name='satellite_0_thermalmapping_135_100', pca_dataset_folder=None, save_dir='logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18')
2024-02-28 23:36:36   The outputs are being saved in logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18
2024-02-28 23:36:36   Using 2 GPUs and 96 CPUs
2024-02-28 23:36:37   Train query set: < TripletsDataset, satellite_0_thermalmapping_135_100 - #database: 10256; #queries: 10256 >
2024-02-28 23:36:46   Extended query set: < TripletsDataset, satellite_0_thermalmapping_135_100 - #database: 79950; #queries: 79950 >
2024-02-28 23:36:48   Val set: < BaseDataset, satellite_0_thermalmapping_135_100 - #database: 13011; #queries: 13011 >
2024-02-28 23:36:51   Test set: < BaseDataset, satellite_0_thermalmapping_135_100 - #database: 26568; #queries: 26568 >
2024-02-28 23:36:58   Domain adapataion: DANN_after
2024-02-28 23:36:58   Output dimension of the model is 4096

This is the GPUs I'm using:
2x3090 24G

This is the evalation resualts:

2024-02-29 18:13:03   Arguments: Namespace(use_extended_data=False, G_test_norm='batch', G_tanh=False, GAN_epochs_decay=0, GAN_lr_policy='linear', GAN_resize=[512, 512], GAN_mode='lsgan', GAN_upsample='bilinear', GAN_save_freq=0, GAN_norm='batch', G_contrast=True, G_gray=False, G_loss_lambda=100.0, visual_all=False, DA_only_positive=False, D_net='none', G_net='none', lambda_DA=1.0, DA='none', add_bn=True, remove_relu=False, use_faiss_gpu=True, prior_location_threshold=-1, use_best_n=1, separate_branch=False, weight_decay=0.0, train_batch_size=4, infer_batch_size=16, criterion='triplet', margin=0.1, epochs_num=1000, patience=3, lr=1e-05, lr_crn_layer=0.005, lr_crn_net=0.0005, optim='adam', cache_refresh_rate=1000, queries_per_epoch=5000, negs_num_per_query=10, neg_samples_num=1000, mining='partial', backbone='resnet18conv4', l2='before_pool', aggregation='netvlad', netvlad_clusters=64, pca_dim=None, num_non_local=1, non_local=False, channel_bottleneck=128, fc_output_dim=None, conv_output_dim=4096, unfreeze=False, pretrain='imagenet', off_the_shelf='imagenet', trunc_te=None, freeze_te=None, seed=0, resume='logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/best_model.pth', device='cuda', num_workers=8, resize=[512, 512], test_method='hard_resize', majority_weight=0.01, efficient_ram_testing=False, val_positive_dist_threshold=50, train_positives_dist_threshold=35, recall_values=[1, 5, 10, 20], brightness=None, contrast=None, saturation=None, hue=None, rand_perspective=None, horizontal_flip=False, random_resized_crop=None, random_rotation=None, datasets_folder='./datasets', dataset_name='satellite_0_thermalmapping_135', pca_dataset_folder=None, save_dir='test/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/satellite_0_thermalmapping_135-2024-02-29_18-13-03')
2024-02-29 18:13:03   The outputs are being saved in test/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/satellite_0_thermalmapping_135-2024-02-29_18-13-03
2024-02-29 18:13:05   Resuming model from logs/default/satellite_0_thermalmapping_135_100-2024-02-28_23-36-31-a12ee086-c2a8-4a19-8d6b-3feb96480a18/best_model.pth
2024-02-29 18:13:08   Test set: < BaseDataset, satellite_0_thermalmapping_135 - #database: 26568; #queries: 26568 >
2024-02-29 18:16:18   Final feature dim: 4096
2024-02-29 18:16:21   Recalls on < BaseDataset, satellite_0_thermalmapping_135 - #database: 26568; #queries: 26568 >: R@1: 72.5, R@5: 81.4, R@10: 84.6, R@20: 87.5
2024-02-29 18:16:21   Finished in 0:03:17

Can you help me find where the problem is, or can you provide the best model weights to test for possible problems

Inference on other datasets

Hi Jiuhong,

First, thank you for your excellent work on thermal geo-localization.

I am currently working on generating thermal images from another dataset's images using your model. However, I've encountered an issue with the output range of model.netG(). Specifically, the output range seems to be unstable, and I haven't found a reliable method to adjust it. In your test.py script, you use the line output_images = output * 0.5 + 0.5, but this approach doesn't always produce consistent results.

Could you suggest any solutions or best practices for stabilizing the output range of model.netG to generate a large number of images reliably?

Thank you for your help!

Besides, my thermal image generation inference codes are following.

image_path = ...
img = cv2.imread(image_path)
        
sp = img.shape
img = cv2.resize(img, (1024, 1024))
img_torch = torch.Tensor(img) / 255
img_permute = img_torch.permute(2, 0, 1).unsqueeze(0).cuda()
output = model.netG(img_permute)
output_images = output
output_images = torch.clamp(output_images, min=-1, max=1)

#####################################
output_images = output_images * .5 + .5
# output_images = (output_images - output_images.min()) / (output_images.max() - output_images.min())
# output_images = output_images + output_images.min()
#####################################

output_images = output_images.squeeze().cpu().detach().numpy()
output_images = cv2.resize(output_images, sp[:2][::-1])

How to find the corresponding cropped satellite patches?

Thanks for your impressive work. I have two questions about the construction of the dataset.

I am wondering how to make the cropped satellite map patch perfectly aligned with the given thermal image.
Could you share about your automatically aligning methods?

And it seems that the dataset does not contain any GPS tag. Would you mind to provide the raw GPS data? Or just offering the GPS information about the satellite map. In that case, we can use the satellite map from other sources to expand the dataset.

Ask for trained models

Hi Jiuhong,

I've been working on UAV geo-localization and find the approach fascinating and potentially very useful. Thank you for your great effort in thermal geo-localization. Since the training step costs a lot in time and resources, I wonder if would it be possible for you to share the pretrained model weights?

Thank you.

dataset not found

Hi,

First, I want to thanks for you sharing your experiements. I face a problem which is the satellite_0_thermalmapping_135_100 dataset not found while I follow your suggestion to run the scripts. Where can I download or find the dataset?

Thanks you.