royalvane/mman

( ECCV2018 ) Macro-Micro Adversarial Network for Human Parsing

MMAN

This is the code for "Macro-Micro Adversarial Network for Human Parsing" in ECCV2018. Paper link

By Yawei Luo, Zhedong Zheng, Liang Zheng, Tao Guan, Junqing Yu^* and Yi Yang.

^* Corresponding Author: [email protected]

The proposed framework is capable of producing competitive parsing performance compared with the state-of-the-art methods, i.e., mIoU=46.81% and 59.91% on LIP and PASCAL-Person-Part, respectively. On a relatively small dataset PPSS, our pre-trained model demonstrates impressive generalization ability.

Prerequisites

• Python 3.6
• GPU Memory >= 4G
• Pytorch 0.3.1
• Visdom

Getting started

Clone MMAN source code

Download The LIP Dataset

The folder is structured as follows:

├── MMAN/
│   ├── data/                 	/* Files for data processing  		*/
│   ├── model/                 	/* Files for model    			*/
│   ├── options/          	/* Files for options    		*/
│   ├── ...			/* Other dirs & files 			*/
└── Human/
    ├── train_LIP_A/		/* Training set: RGB images		*/
    ├── train_LIP_B/		/* Training set: GT labels		*/
    ├── test_LIP_A/		/* Testing set: RGB images		*/
    └── test_LIP_B/		/* Testing set: GT labels		*/

Train

Open a visdom server

python -m visdom.server

Train a model

python train.py --dataroot ../Human --dataset LIP --name Exp_0 --output_nc 20 --gpu_ids 0 --pre_trained --loadSize 286 --fineSize 256

--dataroot The root of the training set.

--dataset The name of the training set.

--name The name of output dir.

--output_nc The number of classes. For LIP, it equals to 20.

--gpu_ids Which gpu to run.

--pre_trained Using ResNet101 model pretrained on Imagenet.

--loadSize Resize training images into 286 * 286.

--fineSize Randomly crop 256 * 256 patch from a 286 * 286 image.

Enjoy the training process in http://XXX.XXX.XXX.XXX:8097/ , where XXX is your server IP address.

Test

Use trained model to parse human images

python test.py --dataroot ../Human --dataset LIP --name Exp_0 --gpu_ids 0 --which_epoch 30 --how_many 10000 --output_nc 20 --loadSize 256

--dataroot The root of the testing set.

--dataset The name of the testing set.

--name The dir name of trained model.

--gpu_ids Which gpu to run.

--which_epoch Select the i-th model.

--how_many Total number of test images.

--output_nc The number of classes. For LIP, it equals to 20.

--loadSize Resize testing images into 256 * 256.

New! Pretrained models are available via this link:

Google Drive

Qualitative results

Trained on LIP train_set -> Tested on LIP val_set

Trained on LIP train_set -> Tested on Market1501

Citation

If you find MMAN useful in your research, please consider citing:

@inproceedings{luo2018macro,
	title={Macro-Micro Adversarial Network for Human Parsing},
	author={Luo, Yawei and 
		Zheng, Zhedong and 
		Zheng, Liang and 
		Guan, Tao and 
		Yu, Junqing and 
		Yang, Yi},
	booktitle ={ECCV},
	year={2018}
}

Related Repos

1. Pedestrian Alignment Network
2. pix2pix
3. Market-1501

mman's Issues

About test

Thanks for your excellent work and code.
I have a try to rewrite the code to test some data , but got some issue,

RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'weight'

Can you help me solve the problem? or do you have the code for testing single image?

The errors of multi-GPU training

Traceback (most recent call last):
File "train.py", line 27, in
model.optimize_parameters()
File "/home/o1/Royal/MMAN/models/pix2pix_model.py", line 287, in optimize_parameters
self.forward()
File "/home/o1/Royal/MMAN/models/pix2pix_model.py", line 170, in forward
self.fake_B_GAN = self.netG(self.real_A)['GAN'] #multi-channel label map--> target real_B_GAN
File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/o1/Royal/MMAN/models/networks.py", line 415, in forward
return nn.parallel.data_parallel(self.model, input, self.gpu_ids)
File "/usr/local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 119, in data_parallel
return gather(outputs, output_device, dim)
File "/usr/local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather
return gather_map(outputs)
File "/usr/local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 60, in gather_map
return type(out)(map(gather_map, zip(*outputs)))
ValueError: dictionary update sequence element #0 has length 30; 2 is required

Excuse me,I wonder how can I solve these errors?
The command is “python3 train.py --dataroot ../Human --dataset LIP --name Exp_0 --output_nc 20 --gpu_ids 0,1 --pre_trained --loadSize 286 --fineSize 256
”

Error training with PPSS

Training on the PPSS dataset, I get the following errors
/opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [225,0,0] Assertion t >= 0 && t < n_classes failed.
Is there anyway to avoid this ?
Thanks

RuntimeError: Error(s) in loading state_dict for UnetGenerator

Unexpected running stats buffer(s) "model.U4.2.U3.4.U2.4.U1.23.U0.6.running_mean" and "model.U4.2.U3.4.U2.4.U1.23.U0.6.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.
Unexpected running stats buffer(s) "model.U4.2.U3.4.U2.4.U1.26.running_mean" and "model.U4.2.U3.4.U2.4.U1.26.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.
Unexpected running stats buffer(s) "model.U4.2.U3.4.U2.7.running_mean" and "model.U4.2.U3.4.U2.7.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.
Unexpected running stats buffer(s) "model.U4.2.U3.7.running_mean" and "model.U4.2.U3.7.running_var" for InstanceNorm2d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm2d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm2d to enable them. See the documentation of InstanceNorm2d for details.

您好，我在测试训练数据时出现了上面的错误（market1501和LIP数据集均有这个错误），原模型中是训练200次，由于电脑配置较低我只训练了25次，并在which_epoch中改为25次。网上有说是pytorch版本的原因，可我电脑上安装的是0.4.0这个版本，请问我要如何解决这个错误？

ConnectionRefusedError: [Errno 111] Connection refused

Thank you for your reply. My previous problem has been solved. 
But now I meet a new problem and been stucked: ConnectionRefusedError: [Errno 111] Connection refused. this error occurred after prompt：model [Pix2PixModel] was created.
Do you have any tips? Be deeply grateful.

where is the plug-and-play module attention(attention to scale)

HI ,can you tell me where is the plug-and-play module attention(attention to scale) in your model?

Pre-trained weight

I would be help full if you provide pre-trained weights

confuse about weight printed in model.save

Hi, i wonder what's the meaning of 'weight L1' , 'weight L1 Attri' etc. printed in the function model:save() before saving the model state_dicts. It seems it has no relation with the lr scheduler, so I'm a little confused.

By the way, i want to know how do you train the baseline model with no discriminators? To be Specific, should the L1 loss be multiplied by lambda_A or not when no GAN loss applied?

Test on Market-1501

The number of classes for LIP is 20, and for Market is 8, how to use pre-trained model to generate Market dataset's map?

test model

What should I do if I want to test this model on a single dataset such as Market, what's parameter should be changed?
Thanks a lot!

The parameters training baseline

Hi, @RoyalVane , thanks for your excellent contribution!
I am confused about the baseline experiments, could you tell me the parameters(batch size, lr, optimizer, etc) you used when training deeplab-aspp and Do-DeepLab-ASPP? Are they the same with those training MMAN? If not, could you give me some details?
Thank you so much!

How to run the pretrained model?

hi, I want to use your pretrained model to generate the predicted images from market1501? How should I do？what's the command code?

label color map

Can you please provide the label color map in RGB.
for example background is (0, 0, 0) etc

concatenate the RGB image as a condition for the discriminator

You say "We concatenated the respect label map with RGB image of corresponding size along class channel as a strong condition to discriminators."

But I really don't understand why you concatenate the RGB images, and have you done any experiments to see how much improvement it brings?

Can you give me some hints and explanations? Thanks so much!

no mode train and eval?

Hello,
I find there is no model.eval() set or model.train() set in evaluation or training procedure. Is it a bug?
Best.

compute the mIoU

Your code in test.py can't compute the mIoU, so how do you get your result ?

Load the pretrained model

Hi, thank you for sharing. But how can I load the pretrained model from the googledisk? when i run the test.py, it always try to download resnet101 form pytorch.org

Where is the param 'hook' in function define_G

MMAN/models/test_model.py

Lines 17 to 21 in d464173

	self.netG = networks.define_G(opt.input_nc, opt.output_nc,
	opt.ngf, opt.which_model_netG,
	opt.norm, not opt.no_dropout,
	opt.init_type,
	self.gpu_ids)

Where is the param 'hook' there? Seems that the function 'define_G' needs a param 'hook'

MMAN/models/networks.py

Line 130 in d464173

def define_G(input_nc, output_nc, ngf, which_model_netG, hook, input_size, norm='batch', use_dropout=False, init_type='normal', gpu_ids=[]):

hook values for GAN and L1

Could you provide more details regarding the following statements:

input_B_GAN: original label image (multiple channels) for GAN loss calculation

input_B_L1: original label image (single channel) for L1 loss calculation

Where can I find these details in the paper? Thanks.

Test on my own data

Thanks for your excellent work and code!

I am working on testing the trained model you provide on my own data, but my data only have raw images without groundtruth segementation labels (when test on LIP, the groundtruth segementation labels are offered in the folder ./Human/test_LIP_B), how can I handle this？ Are the groundtruth segementation labels necessary in testing?

I only want to get the segementation results on my data by your trained model without training.

Waiting for you reply and thanks a lot!

Could you provide the pretrained models for pascal person part?

@RoyalVane