Giter Site home page Giter Site logo

glue-factory's Introduction

Glue Factory

Glue Factory is CVG's library for training and evaluating deep neural network that extract and match local visual feature. It enables you to:

  • Reproduce the training of state-of-the-art models for point and line matching, like LightGlue and GlueStick (ICCV 2023)
  • Train these models on multiple datasets using your own local features or lines
  • Evaluate feature extractors or matchers on standard benchmarks like HPatches or MegaDepth-1500


Point and line matching with LightGlue and GlueStick.

Installation

Glue Factory runs with Python 3 and PyTorch. The following installs the library and its basic dependencies:

git clone https://github.com/cvg/glue-factory
cd glue-factory
python3 -m pip install -e .  # editable mode

Some advanced features might require installing the full set of dependencies:

python3 -m pip install -e .[extra]

All models and datasets in gluefactory have auto-downloaders, so you can get started right away!

License

The code and trained models in Glue Factory are released with an Apache-2.0 license. This includes LightGlue and an open version of SuperPoint. Third-party models that are not compatible with this license, such as SuperPoint (original) and SuperGlue, are provided in gluefactory_nonfree, where each model might follow its own, restrictive license.

Evaluation

HPatches

Running the evaluation commands automatically downloads the dataset, by default to the directory data/. You will need about 1.8 GB of free disk space.

[Evaluating LightGlue]

To evaluate the pre-trained SuperPoint+LightGlue model on HPatches, run:

python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite

You should expect the following results

{'H_error_dlt@1px': 0.3515,
 'H_error_dlt@3px': 0.6723,
 'H_error_dlt@5px': 0.7756,
 'H_error_ransac@1px': 0.3428,
 'H_error_ransac@3px': 0.5763,
 'H_error_ransac@5px': 0.6943,
 'mnum_keypoints': 1024.0,
 'mnum_matches': 560.756,
 'mprec@1px': 0.337,
 'mprec@3px': 0.89,
 'mransac_inl': 130.081,
 'mransac_inl%': 0.217,
 'ransac_mAA': 0.5378}

The default robust estimator is opencv, but we strongly recommend to use poselib instead:

python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite \
    eval.estimator=poselib eval.ransac_th=-1

Setting eval.ransac_th=-1 auto-tunes the RANSAC inlier threshold by running the evaluation with a range of thresholds and reports results for the optimal value. Here are the results as Area Under the Curve (AUC) of the homography error at 1/3/5 pixels:

Methods DLT OpenCV PoseLib
SuperPoint + SuperGlue 32.1 / 65.0 / 75.7 32.9 / 55.7 / 68.0 37.0 / 68.2 / 78.7
SuperPoint + LightGlue 35.1 / 67.2 / 77.6 34.2 / 57.9 / 69.9 37.1 / 67.4 / 77.8
[Evaluating GlueStick]

To evaluate GlueStick on HPatches, run:

python -m gluefactory.eval.hpatches --conf gluefactory/configs/superpoint+lsd+gluestick.yaml --overwrite

You should expect the following results

{"mprec@1px": 0.245,
 "mprec@3px": 0.838,
 "mnum_matches": 1290.5,
 "mnum_keypoints": 2287.5,
 "mH_error_dlt": null,
 "H_error_dlt@1px": 0.3355,
 "H_error_dlt@3px": 0.6637,
 "H_error_dlt@5px": 0.7713,
 "H_error_ransac@1px": 0.3915,
 "H_error_ransac@3px": 0.6972,
 "H_error_ransac@5px": 0.7955,
 "H_error_ransac_mAA": 0.62806,
 "mH_error_ransac": null}

Since we use points and lines to solve for the homography, we use a different robust estimator here: Hest. Here are the results as Area Under the Curve (AUC) of the homography error at 1/3/5 pixels:

Methods DLT Hest
SP + LSD + GlueStick 33.6 / 66.4 / 77.1 39.2 / 69.7 / 79.6

MegaDepth-1500

Running the evaluation commands automatically downloads the dataset, which takes about 1.5 GB of disk space.

[Evaluating LightGlue]

To evaluate the pre-trained SuperPoint+LightGlue model on MegaDepth-1500, run:

python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official
# or the adaptive variant
python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official \
    model.matcher.{depth_confidence=0.95,width_confidence=0.95}

The first command should print the following results

{'mepi_prec@1e-3': 0.795,
 'mepi_prec@1e-4': 0.15,
 'mepi_prec@5e-4': 0.567,
 'mnum_keypoints': 2048.0,
 'mnum_matches': 613.287,
 'mransac_inl': 280.518,
 'mransac_inl%': 0.442,
 'rel_pose_error@10°': 0.681,
 'rel_pose_error@20°': 0.8065,
 'rel_pose_error@5°': 0.5102,
 'ransac_mAA': 0.6659}

To use the PoseLib estimator:

python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official \
    eval.estimator=poselib eval.ransac_th=2.0
[Evaluating GlueStick]

To evaluate the pre-trained SuperPoint+GlueStick model on MegaDepth-1500, run:

python -m gluefactory.eval.megadepth1500 --conf gluefactory/configs/superpoint+lsd+gluestick.yaml

Here are the results as Area Under the Curve (AUC) of the pose error at 5/10/20 degrees:

Methods pycolmap OpenCV PoseLib
SuperPoint + SuperGlue 54.4 / 70.4 / 82.4 48.7 / 65.6 / 79.0 64.8 / 77.9 / 87.0
SuperPoint + LightGlue 56.7 / 72.4 / 83.7 51.0 / 68.1 / 80.7 66.8 / 79.3 / 87.9
SIFT (2K) + LightGlue ? / ? / ? 43.5 / 61.5 / 75.9 60.4 / 74.3 / 84.5
SIFT (4K) + LightGlue ? / ? / ? 49.9 / 67.3 / 80.3 65.9 / 78.6 / 87.4
ALIKED + LightGlue ? / ? / ? 51.5 / 68.1 / 80.4 66.3 / 78.7 / 87.5
SuperPoint + GlueStick 53.2 / 69.8 / 81.9 46.3 / 64.2 / 78.1 64.4 / 77.5 / 86.5

ETH3D

The dataset will be auto-downloaded if it is not found on disk, and will need about 6 GB of free disk space.

[Evaluating GlueStick]

To evaluate GlueStick on ETH3D, run:

python -m gluefactory.eval.eth3d --conf gluefactory/configs/superpoint+lsd+gluestick.yaml

You should expect the following results

AP: 77.92
AP_lines: 69.22

Image Matching Challenge 2021

Coming soon!

Image Matching Challenge 2023

Coming soon!

Visual inspection

To inspect the evaluation visually, you can run:
python -m gluefactory.eval.inspect hpatches superpoint+lightglue-official

Click on a point to visualize matches on this pair.

To compare multiple methods on a dataset:

python -m gluefactory.eval.inspect hpatches superpoint+lightglue-official superpoint+superglue-official

All current benchmarks are supported by the viewer.

Detailed evaluation instructions can be found here.

Training

We generally follow a two-stage training:

  1. Pre-train on a large dataset of synthetic homographies applied to internet images. We use the 1M-image distractor set of the Oxford-Paris retrieval dataset. It requires about 450 GB of disk space.
  2. Fine-tune on the MegaDepth dataset, which is based on PhotoTourism pictures of popular landmarks around the world. It exhibits more complex and realistic appearance and viewpoint changes. It requires about 420 GB of disk space.

All training commands automatically download the datasets.

[Training LightGlue]

We show how to train LightGlue with SuperPoint. We first pre-train LightGlue on the homography dataset:

python -m gluefactory.train sp+lg_homography \  # experiment name
    --conf gluefactory/configs/superpoint+lightglue_homography.yaml

Feel free to use any other experiment name. By default the checkpoints are written to outputs/training/. The default batch size of 128 corresponds to the results reported in the paper and requires 2x 3090 GPUs with 24GB of VRAM each as well as PyTorch >= 2.0 (FlashAttention). Configurations are managed by OmegaConf so any entry can be overridden from the command line. If you have PyTorch < 2.0 or weaker GPUs, you may thus need to reduce the batch size via:

python -m gluefactory.train sp+lg_homography \
    --conf gluefactory/configs/superpoint+lightglue_homography.yaml  \
    data.batch_size=32  # for 1x 1080 GPU

Be aware that this can impact the overall performance. You might need to adjust the learning rate accordingly.

We then fine-tune the model on the MegaDepth dataset:

python -m gluefactory.train sp+lg_megadepth \
    --conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
    train.load_experiment=sp+lg_homography

Here the default batch size is 32. To speed up training on MegaDepth, we suggest to cache the local features before training (requires around 150 GB of disk space):

# extract features
python -m gluefactory.scripts.export_megadepth --method sp --num_workers 8
# run training with cached features
python -m gluefactory.train sp+lg_megadepth \
    --conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
    train.load_experiment=sp+lg_homography \
    data.load_features.do=True

The model can then be evaluated using its experiment name:

python -m gluefactory.eval.megadepth1500 --checkpoint sp+lg_megadepth

You can also run all benchmarks after each training epoch with the option --run_benchmarks.

[Training GlueStick]

We first pre-train GlueStick on the homography dataset:

python -m gluefactory.train gluestick_H --conf gluefactory/configs/superpoint+lsd+gluestick-homography.yaml --distributed

Feel free to use any other experiment name. Configurations are managed by OmegaConf so any entry can be overridden from the command line.

We then fine-tune the model on the MegaDepth dataset:

python -m gluefactory.train gluestick_MD --conf gluefactory/configs/superpoint+lsd+gluestick-megadepth.yaml --distributed

Note that we used the training splits train_scenes.txt and valid_scenes.txt to train the original model, which contains some overlap with the IMC challenge. The new default splits are now train_scenes_clean.txt and valid_scenes_clean.txt, without this overlap.

Available models

Glue Factory supports training and evaluating the following deep matchers:

Model Training? Evaluation?
LightGlue
GlueStick
SuperGlue
LoFTR

Using the following local feature extractors:

Model LightGlue config
SuperPoint (open) superpoint-open+lightglue_{homography,megadepth}.yaml
SuperPoint (official) superpoint+lightglue_{homography,megadepth}.yaml
SIFT (via pycolmap) sift+lightglue_{homography,megadepth}.yaml
ALIKED aliked+lightglue_{homography,megadepth}.yaml
DISK disk+lightglue_{homography,megadepth}.yaml
Key.Net + HardNet ❌ TODO

Coming soon

  • More baselines (LoFTR, ASpanFormer, MatchFormer, SGMNet, DKM, RoMa)
  • Training deep detectors and descriptors like SuperPoint
  • IMC evaluations
  • Better documentation

BibTeX Citation

Please consider citing the following papers if you found this library useful:

@InProceedings{lindenberger_2023_lightglue,
  title     = {{LightGlue: Local Feature Matching at Light Speed}},
  author    = {Philipp Lindenberger and
               Paul-Edouard Sarlin and
               Marc Pollefeys},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year      = {2023}
}
@InProceedings{pautrat_suarez_2023_gluestick,
  title     = {{GlueStick: Robust Image Matching by Sticking Points and Lines Together}},
  author    = {R{\'e}mi Pautrat* and
               Iago Su{\'a}rez* and
               Yifan Yu and
               Marc Pollefeys and
               Viktor Larsson},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year      = {2023}
}

glue-factory's People

Contributors

aduverger avatar iago-suarez avatar phil26at avatar rpautrat avatar sarlinpe avatar veichta avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

glue-factory's Issues

Homography pretraining not converging

Hi! First of all, thank you for making the training and evalution code open-source. I am currently training LightGlue with the superpoint+lightglue_homography.yaml config using a batch size of 128 on a single Nvidia Tesla A100 GPU with 40 GB VRAM. I use python/3.8.5, cuda/11.8.0, cudnn/8.4.0.27, and torch/2.1.2.

However, I have observed that the total validation/training loss does not decrease below 0.8 during homography pretraining. This result is different compared to Figure 5 in the paper where the loss goes below 0.2. I am seeking insights or suggestions on why this discrepancy might be occurring. Are there any specific considerations or adjustments to the configuration that I might be overlooking?

Questions about training data

I have been looking forward to training code of LightGlue and SuperGlue for a long time. I can't wait to reproduce this work. However, the requirement of training data (~900G) is too large. So I want to ask:

Firstly, LightGlue claims there is no major difference between homography pretraining on Megadepth and Oxford-Paris 1M. How large is the different exactly? Can I just performance homography pretraining on Megadepth?

Secondly, is there a major difference between the linked preprocessed Megadepth and the one provided by previous works, e.g., DISK's? I have official raw data and preprocessed data provided by DISK of Megadepth. I don't want to another Megadepth.

stuck at the "Creating dataset ImagePairs" when evaluate lightglue on megadepth

Hi! I'm trying to evaluate lightglue on megadepth using given code and command:

python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official

but it is stucked at the out "gluefactory.datasets.base_dataset INFO] Creating dataset ImagePairs" for one hour. I guess it is not a normal behavior. By debuging, i find it is at the code hfile = h5py.File(str(output_file), "w") . Can you give me some advice? Thanks!
(I install the repo following the readme)
cc @sarlinpe @Phil26AT

DeDoDe

Would it be possible to train the lightglue matcher with the DeDoDe feature extractor + descriptor? not sure if this makes sense to try or not

Pre-training Error

When I was performing the Pre-train LightGlue, an error occurred. Is there any way to fix this error?
er

KeyError: 'loss/total'

[12/12/2023 10:27:28 gluefactory INFO] Starting epoch 0
/home/user0/.local/lib/python3.8/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
[12/12/2023 10:27:30 gluefactory INFO] [E 0 | it 0] loss {total 6.668E+00, last 5.244E+00, assignment_nll 5.244E+00, nll_pos 8.672E+00, nll_neg 1.815E+00, num_matchable 2.530E+02, num_unmatchable 2.575E+02, confidence 5.922E-01, row_norm 1.768E-01}
[12/12/2023 10:27:30 gluefactory INFO] [Validation] {}
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/user0/workspace/Training/glue-factory/gluefactory/train.py", line 691, in
main_worker(0, conf, output_dir, args)
File "/home/user0/workspace/Training/glue-factory/gluefactory/train.py", line 628, in main_worker
training(rank, conf, output_dir, args)
File "/home/user0/workspace/Training/glue-factory/gluefactory/train.py", line 547, in training
if results[conf.train.best_key] < best_eval:
KeyError: 'loss/total'

train.py, line 69
"best_key": "loss/total", # key to use to select the best checkpoint

I am running into KeyError: 'loss/total' when trying to run the command for training superpoint-open+lightglue.
Does anyone have a fix for this KeyError: 'loss/total' ?

Possibly faster training

This curves are with SuperPoint-Open
I have been experimenting with faster training and it seems that OneCycle policy with max_lr=3e-4 provides the same results (based on training and val metrics) than standard, but 2x faster.
Maybe that could help others, who would be training on other types of features.
image
image

    lr_schedule:
        max_lr: 3e-4
        epochs: 20
        steps_per_epoch: 782
        type: OneCycleLR

Struture of the MegaDepth dataset

According to the README document on GitHub, the second step to train LightGlue is to "fine-tune the model on the MegaDepth dataset". I am curious about the structure and content stored within the MegaDepth dataset being used. I hope to receive help from everyone. (I tried to download the dataset used to explore its structure and content, but due to memory limitations, I failed in the download process)

No module named 'resource'

when I run the command python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite eval.estimator=poselib eval.ransac_th=-1 after installing all dependencies, I got the error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "E:\glue-factory\gluefactory\eval\hpatches.py", line 6, in <module>
    import resource
ModuleNotFoundError: No module named 'resource'

A failed attempt to use official PTH as the initial training value

Hello, first of all, thank you for your open source training, which is very important for many of us developers.

At present, I want to use my personal small dataset for fine-tuning under the original official weights. My modification method is to change the reading of official weights around line 296 in the gluefactory/tarin.py file, but during training, it shows a very large loss.
if init_cp is not None:
local_path = '~/3DRE/sfm-learn/SFM_OWN/feature/LightGlue/wight/superpoint_lightglue_v0-1_arxiv.pth' # read offical pth
state_dict = torch.load(local_path, map_location='cpu')
model.load_state_dict(state_dict, strict=False)
# model.load_state_dict(init_cp["model"], strict=False)

After several epochs of training, the loss rapidly decreases, but this is followed by a rapid decrease in matching points. After debugging, it was found that the mscore has severely decreased, and only by adjusting the parameters filter_threshold to 0.001 or even lower can matching points be seen (resulting in a decrease in matching accuracy).

Besides, I also noticed that the names and quantities of keys for official PTH and TAR trained through official tutorials are different, and I am not sure if this is one of the reasons.

May I ask if the above issue is due to the official PTH not being able to directly read weights and train, or a change in the loss construction method, or some other reasons?

torch.multiprocessing.spawn.ProcessExitedException: process 3 terminated with signal SIGKILL

when I train lightGlue using
python -m gluefactory.train sp+lg_megadepth \ --conf gluefactory/configs/superpoint-open+lightglue_megadepth.yaml \ train.load_experiment=sp+lg_homography \ data.load_features.do=True --distributed

**process killed after **
[10/17/2023 04:26:12 gluefactory INFO] [E 4 | it 1000] loss {total 1.731E+00, last 7.856E-01, assignment_nll 7.856E-01, nll_pos 1.262E+00, nll_neg 3.087E-01, num_matchable 4.165E+02, num_unmatchable 7.160E+02, confidence 2.601E-01, row_norm 8.259E-01} .
Can you offer me some advice to solve this problem? Thanks ~

Mismatched model shape upon MegaDepth finetuning

I trained LightGlue on the homographies and then moved on to finetune against MegaDepth.
In the megadepth training script., just at the very start, when loading the model state, the training script throws this exception:

Exception has occurred: RuntimeError
Error(s) in loading state_dict for TwoViewPipeline:
	size mismatch for matcher.posenc.Wr.weight: copying a param with shape torch.Size([32, 2]) from checkpoint, the shape in current model is torch.Size([32, 4]).
  File "/mnt/nvme0n1/data/home/mmarder/glue-factory/gluefactory/models/base_model.py", line 133, in load_state_dict
    ret = super().load_state_dict(*args, **kwargs)

Any ideas what to check? What could have gone wrong? I don't think I changed anything substantially. I just played a bit with feature types and batch size.

Trained Weights

I do not have access to a large machine, If anyone has been able to train on the homography and megadepth dataset for superpoint-open can they share a link to the final output/weights for those of us who don't have machines capable of running the program?

Possible typo?

Hi,
First of all, thanks and congrats to the researchers and developers for this amazing set of contributions. They are really handy and will impact the community quite positively!!
I was looking through the code to have a better understanding and I hit a place that may have a typo.
Here both neg0 and neg1 are assigned to the weight matrix using the same dimension (m), which may hold true when the number of elements is the same in both images, but out of that case it may not work.
Shouldn't it be weights[:, -1, :n] = neg1 ?
Best,
Alberto

error when installing

ERROR: Could not find a version that satisfies the requirement lightglue (unavailable) (from gluefactory) (from versions: none)
ERROR: No matching distribution found for lightglue (unavailable)

Discrepancy in training config w.r.t. paper

Hi,

Just FYI, the configs are different compared to the paper:

Paper: lr 1e-5, by 0.95 in each epoch after 10 epochs.

Training details: Weights are initialized from the pre- trained model on homographies, Training starts with a learning rate of 1e-5 and we exponentially decay it by 0.95 in each epoch after 10 epochs, and stop training after 50 epochs (2 days on 2 RTX 3090). The top 2048 keypoints are extracted per image, and we use a batch size of 32. To speed-up train- ing, we cache detections and descriptors per image, requiring around 200 GB of disk space.

Config: lr 1e-4, start decay after 30th epoch

https://github.com/cvg/glue-factory/blob/main/gluefactory/configs/superpoint%2Blightglue_megadepth.yaml#L43C1-L53C23

train:
    seed: 0
    epochs: 50
    log_every_iter: 100
    eval_every_iter: 1000
    lr: 1e-4
    lr_schedule:
        start: 30
        type: exp
        on_epoch: true
        exp_div_10: 10

memory not release

Hi, when training the model with superpoint+lsd+gluestick-homography.yaml , memory usage is always increasing,and finally leading to the program to terminate. Anything solutions?

mixed precision training

Hello guys

I'm getting the following error when trying to set

mixed_precision=True
mp_dtype=torch.float16

it happens in the GNN module

    664     # Average the updates for each junction (requires torch > 1.12)
--> 665     update0 = update0.scatter_reduce_(
    666         dim=2,
    667         index=lines_junc_idx0[:, None].repeat(1, dim, 1),
    668         src=lupdate0,
    669         reduce="mean",
    670         include_self=False,
    671     )

RuntimeError: scatter(): Expected self.dtype to be equal to src.dtype

then I changed the calling to:

desc0, desc1 = self.gnn(desc0.half(), desc1.half(), line_enc0, line_enc1, lines_junc_idx0, lines_junc_idx1)
and it worked.

It seems like the output from the superpoint network comes in float32

However I think it's not the way to go...
Have anyone encountered this problem before?

my torch version is
'2.2.0+cu118'

I have a problem after training is complete

Hello, thank you very much for your open source code. I am currently encountering a problem. When I train image registration at a large angle, the configuration file is as follows:
data:
name: homographies
data_dir: revisitop1m
train_size: 150000
val_size: 2000
batch_size: 128
num_workers: 14
homography:
difficulty: 0.7
max_angle: 180
photometric:
name: lg
After I trained and fine-tuned, I loaded the generated trained model into lightglue. Even if I ran the code with the same image, the effect of the generated image was as

Uploading 2023-11-14 10-01-25屏幕截图.png…

About installation

I'm a novice who has only recently started learning about this and am currently learning LightGlue.
When I type
python3 -m pip install -e
it warn that -e option requires 1 argument
I don't know what to enter, can you tell me how to do?
Thank you very much!

How to train lightglue with my extractor?

Thanks for your impressive work!
I am trying to train lightglue with my key points extractor, but it seems that there are something with the extractor.
When I use my extractor without lightglue, it extract key points like the photo below(the blue points):
image

But when I trying to train lightglue, the key points looks like the photo below(the blue points):
image

Is there any docs about training lightglue with new extractor? Or could you please give some help?
Thanks!

download_revisitop1m(self) to download_revisitop1m(self, DATA_PATH)

Hi, when I tried to run the training command mentioned here, it returns the following error. It seems the method HomographyDataset.download_revisitop1m() should be revised as def download_revisitop1m(self, DATA_PATH)

sh-4.2$ python -m gluefactory.train sp+lg_homography --conf gluefactory/configs/superpoint-open+lightglue_homography.yaml
[10/05/2023 17:33:43 gluefactory INFO] Starting experiment sp+lg_homography
[10/05/2023 17:33:45 gluefactory INFO] Using device cuda
[10/05/2023 17:33:47 gluefactory.datasets.base_dataset INFO] Creating dataset HomographyDataset
/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/data/revisitop1m
[10/05/2023 17:33:47 gluefactory.datasets.homographies INFO] Downloading the revisitop1m dataset.
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/train.py", line 685, in
main_worker(0, conf, output_dir, args)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/train.py", line 622, in main_worker
training(rank, conf, output_dir, args)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/train.py", line 237, in training
dataset = get_dataset(data_conf.name)(data_conf)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/datasets/base_dataset.py", line 142, in init
self._init(self.conf)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/datasets/homographies.py", line 93, in _init
self.download_revisitop1m(data_dir)
TypeError: HomographyDataset.download_revisitop1m() takes 1 positional argument but 2 were given

Error:_pickle.UnpicklingError: invalid load key, '\x0a'.

Hi, thank you for such a great masterpiece.
I am a beginner and have only recently started learning about this area. Currently, I would like to learn LightGlue. But many problems have arisen that cannot be solved.

When I take this step: python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite
An error occurred ,as follows:

Traceback (most recent call last):
File "D:\DeepLearning\Anaconda3\envs\glue-factory-main-1\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\DeepLearning\Anaconda3\envs\glue-factory-main-1\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\eval\hpatches.py", line 194, in
s, f, r = pipeline.run(
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\eval\eval_pipeline.py", line 83, in run
pred_file = self.get_predictions(
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\eval\hpatches.py", line 85, in get_predictions
model = load_model(self.conf.model, self.conf.checkpoint)
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\eval\io.py", line 91, in load_model
model = get_model("two_view_pipeline")(model_conf).eval()
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\models\base_model.py", line 86, in init
self._init(conf)
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\models\two_view_pipeline.py", line 46, in _init
self.extractor = get_model(conf.extractor.name)(to_ctr(conf.extractor))
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\models\base_model.py", line 86, in init
self._init(conf)
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\models\lines\wireframe.py", line 156, in _init
self.point_extractor = get_model(self.conf.point_extractor.name)(
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory\models\base_model.py", line 86, in init
self._init(conf)
File "D:\DeepLearning\two\hwj_DeepLearning\LightGlue_LoFTR\LightGlue\glue-factory-main-1\gluefactory_nonfree\superpoint.py", line 199, in _init
torch.hub.load_state_dict_from_url(str(self.checkpoint_url)), strict=False
File "D:\DeepLearning\Anaconda3\envs\glue-factory-main-1\lib\site-packages\torch\hub.py", line 750, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "D:\DeepLearning\Anaconda3\envs\glue-factory-main-1\lib\site-packages\torch\serialization.py", line 815, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "D:\DeepLearning\Anaconda3\envs\glue-factory-main-1\lib\site-packages\torch\serialization.py", line 1033, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x0a'.

I did it on the Windows system
My environment is : python3.9+torch2.0.1+cuda11.7
Executed this step : python -m pip install -e . no error

I want to know why such an error occurred? Can you help me take a look?Thanks.

About training LightGlue in two stages

Hello, thanks for your work!
In the LightGlue paper, it is mentioned, 'we first train it to predict correspondences and only after train the confidence classifier. The latter thus does not impact the accuracy at the final layer or the convergence of the training.' However, in the actual code, it seems that they are trained together, and there is no distinction between the two stages.

Feature Extraction before LightGlue training on MegaDepth failed with "Missing key {'depth_keypoints', 'valid_depth_keypoints'}"

Hi, while I am going through the document and trying to pre-extract features on MegaDepth, I face the following error:

sh-4.2$ python -m gluefactory.scripts.export_megadepth --method sp_open --num_workers 8
0 / 193 0275
[10/10/2023 14:13:42 gluefactory.datasets.base_dataset INFO] Creating dataset MegaDepth
[10/10/2023 14:13:42 gluefactory.datasets.megadepth INFO] Sampling new train data with seed 0.
0%| | 0/314 [00:03<?, ?it/s]
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/scripts/export_megadepth.py", line 178, in
run_export(feature_file, scene, args)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/scripts/export_megadepth.py", line 145, in run_export
export_predictions(
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ec2-user/SageMaker/image_stitching/dev/glue-factory/gluefactory/utils/export_predictions.py", line 38, in export_predictions
raise ValueError(f"Missing key {set(keys) - set(pred.keys())}")
ValueError: Missing key {'depth_keypoints', 'valid_depth_keypoints'}

I checked that SuperPoint's prediction of MegaDepth data does not contain those keys, instead only having ["keypoints", "descriptors", "keypoint_scores"].

while Reproducing... can't reach the same performance

Hi! Thank you for your excellent work!
I've been trying to reproduce the results reported in the paper recently. Here's what I got:

  1. By using 2 4090s and following the official config, the results I got from pretraining on the homography dataset:
    09fa20b94f85008974ca40bed1057ac0

2.Finetune on Megadepth

Then I followed settings described in the paper : lr as 1e-5, decay by 0.8 after 10 epochs, and I got the checkpoint_best with loss:
Epoch 48 :New best val: loss/total=0.4931303240458171
the test results are as follows:
4f10be29187de7e8a9432651e6ba6b16

I also tried the settings in the official config: lr as 1e-4 , decay by 0.95 after 30 epochs, this is what i got:
Epoch 49: New best val: loss/total=0.3537058917681376
a007554757cc24d102906d06d8d86edb

And I've tried some other possible settings and still didn't reach the same results reported in the paper.
Could you plz give more details on how you finetune the model on the megadepth dataset? Or any other suggestions on improving the performance?

gluestick training time

I am trying to train gluestick with deep lsd config. I am using 2 A10 GPUs for training. The training has been going on for 5 days and the loss is at around 2.3 . In one of the repo issues, there was information about 2 days for the training period. Do you have any comments about it taking so long on my experiment?

Homography disk requirement

Hi,
In the README you mention 450Go of disk space required for the superpoint-open+lightglue_homography.yaml training config, but after trying to download everything, it failed after not having enough space on my 1To disk, which was empty when I started the script.
Is the number you provided wrong or am I missing something ?

About train

hi,
I met a mistake when I want to train aliked+lg,,when I ran the code:python -m gluefactory.train aliked+lg_homography \ # experiment name
the error shows that: omegaconf.errors.ConfigAttributeError: Missing key train
full_key: train
object_type=dict

by the way, I find that I only installed -e. ,not installed -e.[extra],
however,when I use the code: python3 -m pip install -e .[extra], I find a new error
which is : ERROR: Could not build wheels for homography-est, poselib, which is required to install pyproject.toml-based projects
Could you please tell me how can I deal with those problems?
thank you so much

Issues with lrscheduler

Hi. During training of the model this error started to appear after the latest update.

line 797, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'lr_scheduler' is not in struct
full_key: train.lr_scheduler
object_type=dict

I believe that configs files needs to be updated as well.

Anas.

Performance differences between 2 versions of SuperPoint with pretrained weights

Hi, thank you for your great work!

While I'm testing two-view pipeline with SuperPoint and LightGlue as the extractor and the matcher with the custom image data pair, I noticed there's a slight difference between the detected keypoints from two versions of SuperPoint weights loaded from gluefactory.models.extractors.superpoint_open and gluefactory_nonfree.superpoint (Open-sourced SuperPoint and Non-free SuperPoint). (Maybe I should raise this question in this repo

Output of SuperPoint (MIT Lincense)

Screenshot 2023-10-17 at 4 21 52 PM

Output of Non-Free SuperPoint

Screenshot 2023-10-17 at 4 21 59 PM

It seems Non-Free SuperPoint gives relatively reasonable keypoints. And this behavior persists for the image pairs generated by homographies dataset

Output of SuperPoint (MIT Lincense)

Screenshot 2023-10-18 at 10 26 22 AM

Output of Non-Free SuperPoint

Screenshot 2023-10-18 at 10 26 28 AM

Also noticed the scores of keypoints detected by open-sourced SuperPoint are extremely low so that most keypoints are likely excluded by thresholding

Screenshot 2023-10-18 at 10 30 07 AM
Screenshot 2023-10-18 at 10 30 45 AM

I am curious about whether this is an expected behavior of the two versions of SuperPoint weights or due to my misconfiguration.

question about the relative_pose_error

Hi, I'm I am dig in the problem of feature matching.

I conducted the following experiments on MegaDepth-1500 dataset: First I exact some keypoints in image A, and then I warpped them into the paired image B with the groundtruth T and depth map of A. Then I used OpenCV to solve for the pose and calculate the pose error using the relative_pose_error. Some thing strange occurred. For about 98% pairs the pose error is zero, but several cases have a R_err=180. I find that in the relative_pose_error func, the ambiguity of t_err is eliminated by t_err = torch.minimum(t_err, 180 - t_err), however, R_err did not undergo the same operation. Combining my experimental results, I wonder if this could be a bug? (Although I have looked through many related repositories, no one seems to be doing it this way.)

By the way, While testing lightglue, I found that the RANSAC threshold and different RANSAC methods have a significant impact on the results, and they cannot effectively reflect the quality of the match. From the matching results to the AUC, it involves essential matrix computation based on RANSAC, followed by pose recovery. There are too many uncontrollable factors in the process. Is there another indicator that can more directly reflect the quality of matching? Or to put it another way, why use such a indirect method to evaluate the effectiveness of feature matching methods?

Looking forward to your reply!
cc @sarlinpe @Phil26AT

Danger: Albumentations MotionBlur is shifting our images

I realized that for some images, Albumentations is damaging our GT homographies because when MotionBlur is applied in

The image is shifted without updating the GT homography. This is the default behaviour but can be easily fixed by allow_shifted=False which will create a motion blur kernel that does not move the image.

Could you please provide some checksum for the dataset?

Thanks for your impressive work.
I am trying to train lightglue, but it always fails when downloading data due to the network instability. As a result, I am not sure if it has been downloaded correctly.
Could you please provide a checksum, such as an md5, for the dataset?

Training on my own dataset

Hi,

Is there any way to import your own dataset to finetune the already pretrained superpoint+lightglue model?

I have looked through the code and from my understanding you only support finetuning on the megadepth dataset but not any custom created dataset for finetuning.

I have created my own dataset following the structure from megadepth1500. Do I need any other files? I Can see that you are importing the depthmap but is that necessary when using the camera translation, rotation and intrinsics? Also if I need other files than images and pairs_calibrated what files are needed in order to train using my own dataset (in the case that this is possible using glue-factory).

I wish to finetune on my own dataset for two reasons.

  1. I do not have enough space to download a 400gb dataset
  2. Finetuning for my special use case may result in a better feature matcher

train => disk +lg

Thanks for your impressive work!
I find something maybe wrong while I train mine custom datasets

in disk_kornia.py -> _forward -> for .... ->
image[: min(image.shape[0], i + chunk)] (here)

'''
for i in range(0, image.shape[0], chunk):
if self.conf.dense_outputs:
features, d_descriptors = self._get_dense_outputs(
image[: min(image.shape[0], i + chunk)]
)
dense_descriptors.append(d_descriptors)
else:
features = self.model(
image[: min(image.shape[0], i + chunk)],
n=self.conf.max_num_keypoints,
window_size=self.conf.nms_window_size,
score_threshold=self.conf.detection_threshold,
pad_if_not_divisible=self.conf.pad_if_not_divisible,
)
keypoints += [f.keypoints for f in features]
scores += [f.detection_scores for f in features]
descriptors += [f.descriptors for f in features]
del features
'''

I think it's range maybe wrong. I don't know if you agree? Thanks again

errors when evaling on datasets

when i run this command python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite
the following error appears


Running benchmark: hpatches
Experiment tag: superpoint+lightglue-official
Config:
{'checkpoint': None,
 'data': {'batch_size': 1,
          'name': 'hpatches',
          'num_workers': 16,
          'preprocessing': {'resize': 480, 'side': 'short'}},
 'eval': {'estimator': 'opencv', 'ransac_th': 0.5},
 'model': {'extractor': {'detection_threshold': 0.0,
                         'max_num_keypoints': 1024,
                         'name': 'gluefactory_nonfree.superpoint',
                         'nms_radius': 3},
           'ground_truth': {'name': None},
           'matcher': {'depth_confidence': -1,
                       'features': 'superpoint',
                       'filter_threshold': 0.1,
                       'name': 'matchers.lightglue_pretrained',
                       'width_confidence': -1},
           'name': 'two_view_pipeline'}}
Traceback (most recent call last):
  File "/media/sy/data2/anaconda3/envs/torch1.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/media/sy/data2/anaconda3/envs/torch1.13/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/sy/sy/glue-factory/gluefactory/eval/hpatches.py", line 190, in <module>
    s, f, r = pipeline.run(
  File "/home/sy/sy/glue-factory/gluefactory/eval/eval_pipeline.py", line 83, in run
    pred_file = self.get_predictions(
  File "/home/sy/sy/glue-factory/gluefactory/eval/hpatches.py", line 83, in get_predictions
    model = load_model(self.conf.model, self.conf.checkpoint)
  File "/home/sy/sy/glue-factory/gluefactory/eval/io.py", line 91, in load_model
    model = get_model("two_view_pipeline")(model_conf).eval()
  File "/home/sy/sy/glue-factory/gluefactory/models/base_model.py", line 84, in __init__
    self._init(conf)
  File "/home/sy/sy/glue-factory/gluefactory/models/two_view_pipeline.py", line 49, in _init
    self.matcher = get_model(conf.matcher.name)(to_ctr(conf.matcher))
  File "/home/sy/sy/glue-factory/gluefactory/models/base_model.py", line 84, in __init__
    self._init(conf)
  File "/home/sy/sy/glue-factory/gluefactory/models/matchers/lightglue_pretrained.py", line 20, in _init
    self.net = LightGlue_(dconf.pop("features"), **dconf).cuda()
TypeError: __init__() got an unexpected keyword argument 'name'


and when i run this command python -m gluefactory.eval.megadepth1500 --conf aliked+lightglue_megadepth
it gives the following result

AUC {0.5: [0.0, 0.0, 0.0]}
mAA {0.5: 0.0}
best threshold = 0.5
{'mepi_prec@1e-3': nan,
 'mepi_prec@1e-4': nan,
 'mepi_prec@5e-4': nan,
 'mnum_keypoints': 2048.0,
 'mnum_matches': 0.0,
 'mransac_inl': 0.0,
 'mransac_inl%': 0.0,
 'mrel_pose_error': nan,
 'rel_pose_error@10°': 0.0,
 'rel_pose_error@20°': 0.0,
 'rel_pose_error@5°': 0.0,
 'rel_pose_error_mAA': 0.0}

Questions about SIFT+LightGlue

Thank you for your great work!

I would like to confirm the following things about your SIFT+LightGlue.

  1. Which type of SIFT is used for training of the published SIFT+LightGlue weight, OpenCV or PyColmap?
  2. Is there a performance gap between OpenCV SIFT and PyColmap one for the published SIFT+Lightglue weight.
  3. In the glue-factory implementation, PyColmap SIFT scores are calculated as the combination of DoG (all values are 1?) scores and scales while OpenCV scores as just DoG scores. Which can we get better results by?

Why duplicating the log_assignment calculation?

First of all, appreciate for sharing the training code of the light glue.

At this moment, I'm replicating the first round of training on homograph dataset, here is the loss of the lightglue:

image
The loss_params calculates the log_assignment which looks like the same as in the output of lightglue, why do we need to do that again since there is already a log_assignment item in data?

KeyError: 'loss/total'

image
I have a small doubt regarding line 547 of the train.py file. According to the code, conf.train.best_key will be in numerical format, but why is it set as a string in the config section as best_key = 'loss/total'?

question about homography transformations

  1. I notice that in this repo the variances of the homography transformations are smaller than in the SuperPoint repo. Have you tried more variances of transformations, such as perspective transformation?

  2. If I have well aligned images taken by a fixed camera under different seasons and I want to train LG on my own dataset, is it correct that I need to write my own dataloader for training? Each pair of images is pixel aligned. Could you give more guidance on this? Thanks in advance.

Conversion of trained checkpoint to official lg checkpoint

Hello,
firstly i would express thanks for providing the training framework to the public! :)

Let me firstly explain what i did, and then what issue I am facing:

  • I downloaded the repo & setup the environment
  • copied the datasets/homgraphies.py to datasets/custom.py and modified it to load my custom data (checked that dataloading is cool)
  • composed a new training config, similar to what is expected in official lightglue repo
data:
    name: custom
    data_dir: ""
    metadata_dir: "" 
    train_size: null
    val_size: null
    batch_size: 128
    num_workers: 14
    homography:
        difficulty: 0.7
        max_angle: 359
    photometric:
        name: lg
model:
    name: two_view_pipeline
    extractor:
      name: gluefactory_nonfree.superpoint
      max_num_keypoints: 2048
      detection_threshold: 0.0
      nms_radius: 3
      trainable: False
    ground_truth:
        name: matchers.homography_matcher
        th_positive: 3
        th_negative: 3
    matcher:
      name: matchers.lightglue
      features: superpoint
      depth_confidence: -1
      width_confidence: -1
      filter_threshold: 0.1
      flash: true
train:
    seed: 0
    epochs: 5
    log_every_iter: 100
    eval_every_iter: 500
    lr: 1e-4
    lr_schedule:
        start: 20
        type: exp
        on_epoch: true
        exp_div_10: 10
benchmarks:
    hpatches:
      eval:
        estimator: opencv
        ransac_th: 0.5

And finetuned it for mere 5 epochs.

Then I wrote a custom python script that converts the best_checkpoint.tar to the same structure as official checkpoint (yes I went and manually inspected the layers)

# !/usr/bin/env python3
# -*- coding: utf-8 -*-

import torch

def extract_checkpoint(checkpoint_path, save_model_path):
    checkpoint = torch.load(checkpoint_path, map_location=torch.device('cpu'))
    print("Checkpoint Contents:")
    converted_checkpoint = {}
    for key, value in checkpoint.items():
        if key == "model":
            for k, v in value.items():
                if "matcher.transformers." in k:
                    # Extract the transformer layer number and the rest of the key
                    parts = k.split(".")
                    transformer_layer = parts[2]
                    remaining_key = ".".join(parts[3:])
                    
                    # Construct the new key name
                    new_key = f"self_attn.{transformer_layer}.{remaining_key}"
                    converted_checkpoint[new_key] = v
                elif k.startswith("matcher."):
                    # For other matcher parts, just remove 'matcher.' prefix
                    new_key = k.replace("matcher.", "")
                    converted_checkpoint[new_key] = v
                else:
                    print(k)

    torch.save(converted_checkpoint, save_model_path)

if __name__ == "__main__":
    checkpoint_path = "checkpoint_best.tar"
    save_model_path = "lg_finetuned_v4.pth"
    extract_checkpoint(checkpoint_path, save_model_path)

After I loaded the converted checkpoint to the official Lightglue repository, it did not produce any matches (tested on 1k images), keypoints were succesfully extracted though.

I am a bit worried that I took the wrong direction somewhere, and would really appreciate your guidance!

thanks:)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.