foundationvision/glee

[CVPR2024 Highlight]GLEE: General Object Foundation Model for Images and Videos at Scale

glee's Issues

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

GLEEmodel_swin load error, how to solve this?

Traceback (most recent call last):
File "test_demo.py", line 102, in
GLEEmodel_swin = GLEE_Model(cfg_swin, None, device, None, True).to(device)
File "/data/huangyuan/GLEE_ws/GLEE/projects/GLEE/glee/models/glee_model.py", line 59, in init
self.backbone = build_backbone(cfg)
File "/data/huangyuan/GLEE_ws/GLEE/detectron2/modeling/backbone/build.py", line 31, in build_backbone
backbone = BACKBONE_REGISTRY.get(backbone_name)(cfg, input_shape)
File "/data/huangyuan/GLEE_ws/GLEE/projects/GLEE/glee/backbone/swin.py", line 738, in init
self.init_weights(pretrained_weight)
File "/data/huangyuan/GLEE_ws/GLEE/projects/GLEE/glee/backbone/swin.py", line 652, in init_weights
checkpoint = torch.load(pretrained, map_location='cpu')
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 1004, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 456, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

No module named 'projects.GLEE'

你好：我在windows下运行Python app.py时，遇到GLEE\app.py", line 18, in
from projects.GLEE.glee.models.glee_model import GLEE_Model
ModuleNotFoundError: No module named 'projects.GLEE'
的错误，明明projects就在当前文件夹下，怎么就加载不了的呢？盼回复。
主要错误如下：Traceback (most recent call last):
File "E:\tool\GLEE\app.py", line 18, in
from projects.GLEE.glee.models.glee_model import GLEE_Model
ModuleNotFoundError: No module named 'projects.GLEE'
原码app.py部分如下：mport gradio as gr
import numpy as np
import cv2
import torch

from detectron2.config import get_cfg
import sys
#sys.path.insert(0, 'E:/tool/GLEE-main')
#sys.path.append('E:\tool\GLEE\projects\')
from projects.GLEE.glee.models.glee_model import GLEE_Model
from projects.GLEE.glee.config import add_glee_config

How can I export this GLEE model to onnx model?I only find the export2torchScript and export2caffe2 scripts.

Hugging Face demo not runnig

Try to test using the Hugging Face demo. The demo page is just showing a error:

The training code

Hi, Thanks for the solid work. Could you let me know when you'll release the training code?

the expression prompt mode only output one object?

in the hugging face demo, the expression prompt mode only output one object, even if there are multi same objects?

Open Source the test code and weight files about TAO dataset

Hello, thank you for the outstanding work. Could you please make the inference code and weight files about the TAO data set public? Thank you very much.

关于视频任务模型Plus版本

github界面只给了图片任务的R50和SwinL2个版本的模型，然后我在huggingface上demo的files里面看到了视频任务的R50版本（visual prompt，GLEE_vos_r50.pth），想问下作者能不能开源一下视频任务的SwinL版本，是不是因为huggingface上使用的GPU跑不动所以才没放SwinL版本？
此外，关于使用的体验，我发现模型对于没学过的语言提示词效果很差，比如用custom-list不认识人头（head），输入human head才有可能给出比较差的结果。

Tutorial for Zero-shot Testing or Fine-tuning GLEE on New Datasets

In the todo list you have a bullet point related to "zero-shot testing or fine-tuning new datasets".

Do you plan to also cover single modality dataset fine tuning?

Failure cases

关于coco指标复现的问题

您好，我使用了您的模型对 COCO 数据集进行测试，发现我得到的评估指标与paper的指标有差距。我想了解一下，现在提供的demo和完整测试过程之间是否存在某些差异？

Open-vocabulary segmentation Demo

Hi @wjf5203, thank you for this interesting work!

Can you please provide a "demo" script for open-vocabulary instance segmentation on videos? Currently, I only see a TEST.md file that describes evaluating on existing datasets.

I also find the code a bit hard to follow to be able to implement this on my own. So, it would be very helpful if you can provide such a demo script.

Thank you! :)
Yash

Detail about object detection decoder.

Hi, there. I believe GLEE is a great work, thanks for open source!
I have a question about object detection: what's the input to the decoder when used as a object detector?
Does it need to input object query including box position from anchor boxes?
If I'm not wrong, in MaskDINO, it will input box position from anchor and mask as object query.
So, what's the object query like in GLEE when used as object detector?
Looking forward for your reply, thanks a lot!

关于训练中数据采样的问题？

GLEE在训练过程中，每个iter采样的数据都是来自相同数据集的吗？我发现task总是取batched_inputs[0],

PANOPTIC video benchmark

Do you plan to add a VIDEO panoptic benchmark?
https://github.com/yoxu515/VIPOSeg-Benchmark

Open Vocabulary Multiple Object Tracking on a Custom Video

Hi,

Thank you for your great work on this project. I would like to perform open vocabulary multiple object tracking on a custom video using your model. Could you please provide guidance or some example code on how to achieve this?

Thank you in advance!

ImportError: cannot import name 'add_glee_config' from 'detectron2.projects'

TEST.md, To inderence on COCO,在Plus上按命令运行时报错，ImportError: cannot import name 'add_glee_config' from 'detectron2.projects' (/root/miniconda3/envs/myconda/lib/python3.10/site-packages/detectron2-0.6-py3.10-linux-x86_64.egg/detectron2/projects/init.py)
重新下载了detectron2文件也不行，请问该怎么解决？

DETRs with Hybrid Matching

In paper, "query denoising and "Hybrid matching" are kept to accelerate convergence and improve performance". Does Hybrid matching refer to the H-DETR paper: https://arxiv.org/pdf/2207.13080.pdf

What is the size of the model's parameter count?

How to run the demo locally and correctly?

Hi! I've read README and tried to run the demo on my server, but I think there're a lot of code that is out of sync or missing. And the guides are incomplete.

Here are my steps:

1. I installed the dependencies according to INSTALL.md
2. I ran the app.py in this repo
3. A lot of models are missing:
   • I need GLEE models so I downloaded them on huggingface demo directory
   • Misc models are missing, then I found the ones mentioned in TRAIN.md and downloaded them
4. There are bugs in app.py on lines like (outputs,_) = GLEEmodel(...), which should be ((outputs, _), _, _) = GLEEmodel(...)
5. Then I ran the app.py in this repo again, but the results are just random like below.

Did I do anything wrong? Should I just clone the huggingface repo instead?

本地运行报错

作者您好，非常感谢您的工作，我在本地运行时，报了缺少文件的错误：
Config '/mnt/yrfs/userdata/hsp/projects/GLEE/app/GLEE/configs/R50.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Traceback (most recent call last):
File "app.py", line 90, in
GLEEmodel_r50 = GLEE_Model(cfg_r50, None, device, None, True).to(device)
File "/mnt/yrfs/userdata/hsp/projects/GLEE/app/GLEE/glee/models/glee_model.py", line 67, in init
self.text_encoder = CLIPTextModel.from_pretrained('GLEE/clip_vit_base_patch32')
File "/home/hsp/anaconda3/envs/GLEE/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3206, in from_pretrained
raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory GLEE/clip_vit_base_patch32.
请问我需要在哪里下载对应的文件呢？

speed

请教一下GLEE-Pro和GLEE-Plus的运行速度是多少

Would it be possible to open up the source code of the inference script instead of the gradio script?

Hi， I reproduced the inference script via the gradio script (app.py). But I get different object detection results with the same parameter.

huggingface run error

ModuleNotFoundError: No module named 'detectron2'

是否能像SAM或CLIP那样直接调用

是否能像SAM或CLIP那样直接调用

例如：from sam import SAM
model = SAM(....)
.....

跨图检测

您好，我试用了GLEE，非常棒的工作！想请教一下GLEE是否支持跨图的检测呢，具体来说，就是在第一张图像上给出scribble或者bbox，然后在另一张图像上检测第一张图上的所指目标。我看到视频有类似功能，请问是否也支持静态图像呢

too many values to unpack (expected 2)

运行时出现如下问题，请问如何解决：
File "E:\hezt\vis\lib\site-packages\gradio\queueing.py", line 527, in process_events
response = await route_utils.call_process_api(
File "E:\hezt\vis\lib\site-packages\gradio\route_utils.py", line 270, in call_process_api
output = await app.get_blocks().process_api(
File "E:\hezt\vis\lib\site-packages\gradio\blocks.py", line 1847, in process_api
result = await self.call_function(
File "E:\hezt\vis\lib\site-packages\gradio\blocks.py", line 1433, in call_function
prediction = await anyio.to_thread.run_sync(
File "E:\hezt\vis\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "E:\hezt\vis\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "E:\hezt\vis\lib\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "E:\hezt\vis\lib\site-packages\gradio\utils.py", line 805, in wrapper
response = f(*args, **kwargs)
File "E:\tool\GLEE-new\app.py", line 169, in segment_image
(outputs,_) = GLEEmodel(infer_image, prompt_list, task="coco", batch_name_list=batch_category_name, is_train=False)
ValueError: too many values to unpack (expected 2)

RVOS inference code

Hello, GLEE is a wonderful work. I saw that the RVOS code part is not finished yet. Can you update the GLEE.py for RVOS inference?

VOS benchmark

Why you have excluded from the VOS benckmark at table 8 some top performing memory networks?

https://github.com/yoxu515/aot-benchmark
https://github.com/yamy-cheng/DMAOT-VOTS2023

你好， 请问 什么时候公开微调代码

非常感谢您的工作
但是由于设备有限，希望可以公开微调代码
可以告知大概时间吗？
万分感谢！！！

缺少这么多文件怎么破？OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory projects/GLEE/clip_vit_base_patch32.

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory projects/GLEE/clip_vit_base_patch32. 缺这么多文件怎么说

Where is 'get_tracking_contrastive_lossv3'?

projects\GLEE\glee\models\glee_model.py, line 364

track_loss = self.get_tracking_contrastive_lossv3(outputs[0], targets, task)

Where is get_tracking_contrastive_lossv3? And could you please give more details about how to train and inference with tracking data?

关于visual promt

您好，GLEE是一个很棒的工作。同时，关于算法的一些细节，我有一些疑问想像您请教，如果您有空了，可以回复一下，感谢！

1. 我使用points作为视觉提示词，GLEE是否支持负点击？能否像SAM一样使用多次点击来对一个目标进行微调？
2. 我看您的代码实现，似乎会将points变为一个box作为提示，为什么要这么做？我没有在您的论文中找到相关的解释。
3. 视觉提示词返回的topk_instance是否只能为1？它能否分割出一个被遮挡目标的多个部分？
   感谢！

COCO performance after finetuning

Have you tried to finetune the stage 2 model on COCO dataset and see the limit of the COCO performance?