amathislab/buctd

[ICCV 2023] "Rethinking pose estimation in crowds: overcoming the detection information-bottleneck and ambiguity"

Bottom-up conditioned top-down pose estimation (BUCTD)

This repository contains the official code for our paper: Rethinking pose estimation in crowds: overcoming the detection information-bottleneck and ambiguity. [YouTube Video] [Website]

• Sep 2023: We released the code :)
• July 2023: This work is accepted to ICCV 2023 🎉
• June 2023: BUCTD was also presented at the 2023 CV4Animals workshop at CVPR
• June 2023: An earlier version can be found on arxiv

• This code will also be integrated in DeepLabCut!

Installation

We developed and tested our models with python=3.8.10, pytorch=1.8.0, cuda=11.1. Other versions may also be suitable.

Instructions

1. Clone this repo, and in the following we will call the directory that you cloned ${BUCTD_ROOT}.

git clone https://github.com/amathislab/BUCTD.git
cd ${BUCTD_ROOT}

2. Install Pytorch and torchvision

Follow the instructions on https://pytorch.org/get-started/locally/.

# an example:
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge

3. Install additional dependencies

pip install -r requirements.txt

4. Install COCOAPI

# COCOAPI=/path/to/clone/cocoapi
git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
cd $COCOAPI/PythonAPI
# Install into global site-packages
make install
# Alternatively, if you do not have permissions or prefer
# not to install the COCO API into global site-packages
python setup.py install --user

5. Install CrowdPoseAPI exactly in the same way as COCOAPI.

6. Install NMS

cd ${BUCTD_ROOT}/lib
make

Training

Instructions

Generative sampling

You can use the script: train_BUCTD_synthesis_noise.sh.

Empirical sampling

You can match your own bottom-up (BU) models by updating the scripts in ./data_preprocessing/.

If you do not want to match your own BU models for training, we provide the training annotations. You can download the annotations here.

During inference, we use different BU/one-stage model's predictions (e.g. PETR, CID) as Conditions. The result files can be downloaded from the link above.

Testing

We also provide the best model per human dataset along with the testing scripts.

COCO

Model	Sampling strategy	Image Size	Condition	AP	Weights	Script
BUCTD-preNet-W48	Generative sampling	384x288	PETR	77.8	download	script

OCHuman

Model	Sampling strategy	Image Size	Condition	AP_val	AP_test	Weights	Script
BUCTD-CoAM-W48	Generative sampling (3x iterative refinement)	384x288	CID-W32	49.0	48.5	download	script

CrowdPose

Model	Sampling strategy	Image Size	Condition	AP	Weights	Script
BUCTD-CoAM-W48	Generative sampling	384x288	PETR	78.5	download	script

Code Acknowledgements

We are grateful to the authors of HRNet, MIPNet, and TransPose as our code builds on their excellent work.

Reference

If you find this code or ideas presented in our work useful, please cite:

Rethinking pose estimation in crowds: overcoming the detection information-bottleneck and ambiguity (ICCV) by Mu Zhou*, Lucas Stoffl*, Mackenzie W. Mathis and Alexander Mathis (arxiv)

@InProceedings{Zhou_2023_ICCV,
    author    = {Zhou, Mu and Stoffl, Lucas and Mathis, Mackenzie Weygandt and Mathis, Alexander},
    title     = {Rethinking Pose Estimation in Crowds: Overcoming the Detection Information Bottleneck and Ambiguity},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {14689-14699}
}

License

BUCTD is released under the Apache 2.0 license. Please see the LICENSE file for more information.

buctd's Issues

Looking forward to your open source！

Great work, looking forward to open source and roughly when it will be released!

How to get the "train_cond.json"？

When I run the "train_BUCTD_prenet_gen_sample.sh", I found that I couldn't simply use “person_keypoints_train2017.json”。So how to get the "train_cond.json"？Can you provide the json file for training CTD on coco dataset? Looking forward to your reply!

How to test on single image ?

Please give an example of given single image

Why the key point feature is a three-channel color map?

Hi, may I ask why three channels are used for the key point features entered here? Is there any basis for the selection of different keypoint colors?

Questions about the OCHuman dataset results

Hi! This is a splendid work and thanks for releasing the code.
I would like to know why your papers report low results on the OCHuman dataset? For example, MIPNet-W48 reported AP=74.1 in the original paper, but in your paper it becomes 42.0? Similar results were found for both CID and HRNet.

About code training

I am a beginner in human pose estimation, and I cannot find the script named "train_BUCTD_synthesis_noise.sh". What command should I use to train this code? Could you provide an example command?

Testing videos using pre-trained models

Hello,

I would like to run the BUTCD in videos using your pre-trained models. I just did not get the instructions for that.
Can you help me?

Regards,

Felipe.

install requirement packages lshashing==1.1.3 encounter error "NameError: name 'ext_modules' is not defined"

Hi, I encountered error when try to install packages inside my conda env (env created with python==3.8.10). I download numpy first then run the pip install.

Thanks for your help.

PC info:
OS: Win11, running code in WSL Ubuntu
CPU: Intel i7-10700F CPU 2.90GHz

After run pip install -r requirement.txt, error looks like this:

Full text version:
pip install -r requirements.txt
Collecting asttokens==2.0.5 (from -r requirements.txt (line 1))
Using cached asttokens-2.0.5-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting backcall==0.2.0 (from -r requirements.txt (line 2))
Using cached backcall-0.2.0-py2.py3-none-any.whl.metadata (2.0 kB)
Collecting certifi==2023.7.22 (from -r requirements.txt (line 3))
Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.3.0 (from -r requirements.txt (line 4))
Using cached charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (32 kB)
Collecting click==8.1.2 (from -r requirements.txt (line 5))
Using cached click-8.1.2-py3-none-any.whl.metadata (3.2 kB)
Collecting colour==0.1.5 (from -r requirements.txt (line 6))
Using cached colour-0.1.5-py2.py3-none-any.whl.metadata (18 kB)
Collecting cycler==0.11.0 (from -r requirements.txt (line 7))
Using cached cycler-0.11.0-py3-none-any.whl.metadata (785 bytes)
Collecting Cython==0.29.28 (from -r requirements.txt (line 8))
Using cached Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (2.8 kB)
Collecting debugpy==1.6.0 (from -r requirements.txt (line 9))
Using cached debugpy-1.6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (1.1 kB)
Collecting decorator==5.1.1 (from -r requirements.txt (line 10))
Using cached decorator-5.1.1-py3-none-any.whl.metadata (4.0 kB)
Collecting docopt==0.6.2 (from -r requirements.txt (line 11))
Using cached docopt-0.6.2.tar.gz (25 kB)
Preparing metadata (setup.py) ... done
Collecting easydict==1.7 (from -r requirements.txt (line 12))
Using cached easydict-1.7.tar.gz (6.2 kB)
Preparing metadata (setup.py) ... done
Collecting entrypoints==0.4 (from -r requirements.txt (line 13))
Using cached entrypoints-0.4-py3-none-any.whl.metadata (2.6 kB)
Collecting executing==0.8.3 (from -r requirements.txt (line 14))
Using cached executing-0.8.3-py2.py3-none-any.whl.metadata (8.6 kB)
Collecting Flask==2.1.1 (from -r requirements.txt (line 15))
Using cached Flask-2.1.1-py3-none-any.whl.metadata (3.9 kB)
Collecting fonttools==4.33.2 (from -r requirements.txt (line 16))
Using cached fonttools-4.33.2-py3-none-any.whl.metadata (125 kB)
Collecting h5py==3.6.0 (from -r requirements.txt (line 17))
Using cached h5py-3.6.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (1.9 kB)
Collecting idna==3.4 (from -r requirements.txt (line 18))
Using cached idna-3.4-py3-none-any.whl.metadata (9.8 kB)
Collecting imageio==2.17.0 (from -r requirements.txt (line 19))
Using cached imageio-2.17.0-py3-none-any.whl.metadata (4.8 kB)
Collecting importlib-metadata==4.11.3 (from -r requirements.txt (line 20))
Using cached importlib_metadata-4.11.3-py3-none-any.whl.metadata (4.0 kB)
Collecting ipykernel==6.13.0 (from -r requirements.txt (line 21))
Using cached ipykernel-6.13.0-py3-none-any.whl.metadata (2.4 kB)
Collecting ipython==8.2.0 (from -r requirements.txt (line 22))
Using cached ipython-8.2.0-py3-none-any.whl.metadata (4.9 kB)
Collecting itsdangerous==2.1.2 (from -r requirements.txt (line 23))
Using cached itsdangerous-2.1.2-py3-none-any.whl.metadata (2.9 kB)
Collecting jedi==0.18.1 (from -r requirements.txt (line 24))
Using cached jedi-0.18.1-py2.py3-none-any.whl.metadata (20 kB)
Collecting Jinja2==3.1.1 (from -r requirements.txt (line 25))
Using cached Jinja2-3.1.1-py3-none-any.whl.metadata (3.5 kB)
Collecting joblib==1.1.0 (from -r requirements.txt (line 26))
Using cached joblib-1.1.0-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting json-tricks==3.15.5 (from -r requirements.txt (line 27))
Using cached json_tricks-3.15.5-py2.py3-none-any.whl.metadata (17 kB)
Collecting jupyter-client==7.2.2 (from -r requirements.txt (line 28))
Using cached jupyter_client-7.2.2-py3-none-any.whl.metadata (5.4 kB)
Collecting jupyter-core==4.10.0 (from -r requirements.txt (line 29))
Using cached jupyter_core-4.10.0-py3-none-any.whl.metadata (1.4 kB)
Collecting kiwisolver==1.4.2 (from -r requirements.txt (line 30))
Using cached kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (6.4 kB)
Collecting lshashing==1.1.3 (from -r requirements.txt (line 32))
Using cached lshashing-1.1.3.tar.gz (122 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-1aamtb6d/lshashing_d75e976c9a8d40aa971e6d6fb7a2b29c/setup.py", line 42, in
ext_modules = ext_modules
NameError: name 'ext_modules' is not defined
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Regarding environment configuration

Hi, may I ask if this code can run on Windows and a CPU? When configuring the environment, I encounter many errors when installing dependencies, and it eventually shows "make: *** No rule to make target 'install'. Stop". Therefore, I am considering using a Windows CPU to run it and would like to know if that is possible.

visualization

How to implement visualization? Thank you !

Pretrained model keypoint number mismatch

Hi, I'm trying to test the code, and while the coco config file and weights work well, there is a mismatch between the ochuman and crowdpose weights.

The OcHuman weights says their missing weights with 17 keypoints on the coco yaml

The CrowdPose weights say they are missing weights with 14 keypoints on the crowdpose yaml

Also, will there be plans to release an inference code to perform single image inference?

Thank you, I appreciate your work!

Assumption regarding number of keypoint conditions

I notice the testing code requires condition values for all joints. Do the pretrained models still work if joints are omitted? (Assuming code is modified so that colors are still correct for the present joints).

Inquiry on Integration Timeline for BUCTD in DeepLabCut

Could you kindly share any available timelines or milestones regarding the integration of BUCTD into DeepLabCut? Understanding the expected rollout schedule will greatly benefit users like myself in planning upcoming research projects that could leverage this new functionality.

Thank you for your support of the research community. Looking forward to your response!

CTD-preNet-W32's model

I find that only BUCTD-preNet-W48's model Weights are available in the repository. Can you provide BUCTD-preNet-W32's model on COCO? I want to compare with your prenet-w32 model on the COCO dataset, but didn't have the time to train it again with your methods.