wisconsinaivision/mixnmatch

Pytorch implementation of MixNMatch

mixnmatch's Issues

Unable to download pretrained models

Hey !
Amazing work guys !!! I am able to download the data but when I try the pretrained models it says Error 404 page not found. Just wanted to ask if you guys have moved the pretrained models from the given link ?

Codes and pre-trained weights

Dear author, please upload the codes and pre-trained weights for the model.

Transfer Learning on MixnMatch

@Yuheng-Li Can you explain how transfer learning can be done on this model.

Change in output image size from 128 * 128 px to custom size

I am trying to train the model on custom dataset.
The train dataset is of the size 400 * 400 px. But while inferring the size of generated output is 128 * 128 px.

Is it possible to change the size of output image? If yes, is some modification required in the code?

where the trained modles?

for my understanding to run eval.py and create new image (from 4 images ) i need the generator (G.pth) and encoder (E.pth)
so after the first training stage i got folder with Model folder that contain the followings models :
BD_0,DO_O,D1_0,D2_0,E_0,G_0
after training the second stage i have folder EX_0.pth
my general questions what s should i need to do next for making new image?
i need path for 4 images that is ok
but i also need path to a folder with 3 pre trained models G.pth E.pth EX.pth - where are them?
do i need to take E_0,G_0 (from first training stage) and EX_0 (from second training stage) and put them together under the same folder and that folder is the models path?

few questions:
EX.pth that came from secound training stage is for feature mode?
for eval.py why do i need E.pth and EX.pth ?
how many epochs is needed (first and second stage) for getting the same resualts for birds datasets, the paper say nothing about that
thanks

why multiple 4 ??

why code_z *4 in Encoder Network ???

image resize

Hey,

Thanks for the great work and releasing the code. I was wondering how to set the ratio(76/64) at the resize part here https://github.com/Yuheng-Li/MixNMatch/blob/21095b3581c7d47f67ed1bb360ca8ac3db6c299f/code/datasets.py#L57 ,

To extend the work to other image size for training, which resize ratio would be recommended?

Thanks!

Parameter details

Hi,

What is the significance of these parameters SUPER_CATEGORIES and FINE_GRAINED_CATEGORIES.

Also, Is there a minimum number of recommended Image count for training. Do you have a Image preparation script to suit the model input for training.

Converting a reference image according to a reference video

is this keeping a video as pose images and then background, texture, and shape is the same image? Can you provide an example for Converting a reference image according to a reference video in eval.py?

question about eval result and re-train result

Hi, thanks your helpping , I have finish re-train all stage.
There are 6 pictures for result:

Format

-	-	-	-
pose_file	background_file	shape_file	color_file
retrain_gen_feature_mode	retrain_gen_code_mode	pretrain_gen_code_mode	pretrain_gen_feature_mode

Pictures:

pic name	pic
pic_1 (pose-1, backgroud-1, shape-1, color-1)
pic_2 (pose-2, backgroud-2, shape-2, color-2)
pic_3 (pose-3, backgroud-3, shape-3, color-3)
pic_4 (pose-4, backgroud-4, shape-4, color-4)
pic_5 (pose-5, backgroud-5, shape-5, color-5)
pic_6 (pose-6, backgroud-6, shape-6, color-6)

My quesions:

Q1: why object of pic_1 is out of picture in code mode ? such as both pic_1(retrain_gen_code_mode and pretrain_gen_code_mode ) ?

Q2: my retrain model weight maybe failed in capture texture feature ? Did you have some experience for this ? the reason is different random seed or something else? such as all retrain_gen pictures

Q3: Nothing to be generated except background, [such as pic_5(retrain_gen_code_mode)], or bad backgroud [such as pic_6(pretrain_gen_feature_mode)]

How can I train and create a new model for custom dataset?

I tested for pre-trained model and given dataset, but I want to create a new model for my custom dataset, how can I train and create a new model?

Hi, when will you release your code ??

About pre-training model

Related to the bounding box annotation.

I have used labelimg for creating bounding boxes for my custom dataset. But it is giving coordinate values ranging from (0 - 1). Due to this reason I am getting zeroDivisionError in the implementation of the code. So i want to know the correct method and model for creating these bounding box annotations.

Colab version please?

Hello. I love this repo and I'd like to use it. Colab notebooks tend to be easy to set up, I tried doing it just now and was unable to. Would you mind creating a colab notebook version of this repo? Thank you!

Requirements

Thanks for a nice package! Just wanted to point out that there are more requirements than what is stated in the README file; in addition to the ones already listed:

easydict
matplotlib
sklearn
tensorboardX
torchvision==0.4.2 (not sure exactly which versions work, but this one does and some others I tried don't)

why not initialization the weight of BD_Networks?

Hi, in load_networks functions only three networks have been initialized : netG, netDs, encoder

but only BD not intialization ? why ? and this is your experiment performance ?

Large amount of data is needed?

I took the top 6 categories in CUB dataset(323 pictures) and only changed SUPER_CATEGORIES to 3 and FINE_GRAINED_CATEGORIES to 6 to run train_first_stage.py but got bad results. In my opinion, deep learning model will easily fit the dataset which has little amount of data, it semms that your model need large amount of data to train? Expect to get your reply, Thanks.

What is the Bounding box？

I finished reading this paper . the paper said we only require a loose bounding box around the object to model background, But the author didn't explain it in detail. and i have some questions
1.How we define the bounding box，what is it used for？
2.what should i do to get the bounding box if i wanna train my own dataset?

i‘m looking forward to your reply。

How to print EG_Loss?

I tried the sample code and concernd about how can I print the EG_Loss?
Thank You.

Bounding Box

What implementation did you use to get the bounding box in that format?
I can see that (x, y, w, h) is the pixel distance from the top and left, I just don't know of a library that uses that format.

Unable to setting up the dataset

Hey, guys your work is so interesting, that I want to setting up this model but having issues in doing that, Can you please refer me some videos of it how to set up the dataset in the model. I also want to contribute it by training it on different dataset. Can you please help me?

mutual information discriminator

Hi, does the first p in the mutual information discriminator D(P|Pfm) used in the parent phase of FineGAN refer to the latent code p or the generated fake image p,I tried to read the source code of FineGAN, but I couldn't understand the idea of def train_Gnet in trainer.py about the implementation of the parent mutual information discriminator

Result is very bad on custom dataset

Hii @Yuheng-Li @utkarshojha @kkanshul @Johnson-yue

I am trying to generate full body human using this model but when we train this model on custom dataset result was bad after training completion, so can you suggest me how can we improve results on custom datasets.
I am share some details which you will be able to understand easily.

Model configuration :
SUPER_CATEGORIES = 1
FINE_GRAINED_CATEGORIES = 1
FIRST_MAX_EPOCH = 600
SECOND_MAX_EPOCH = 400

Here it's our model configuration as you can see above, now I am sharing two picture first one is ref image and another one is result of our model.

the Question about eval result!!

Hi, thank your sharing !!
It is very interesting, I have tested some birds picture with pre-trained model by using eval.py and I find some question

I run the command line code:
python eval.py --z pose/pose-2.png --b background/background-2.png --p shape/shape-2.png --c color/color-2.png --mode **_feature_** --models ../models/bird --out ./feature-2.png
got feature-2.png
and I run the:
python eval.py --z pose/pose-2.png --b background/background-2.png --p shape/shape-2.png --c color/color-2.png --mode **_code_** --models ../models/bird --out ./code-2.png
got code-2.png

the feature-2.png is the same as /code/result/0001.png
but when I check, pose-2.png , background-2.png, shape-2.png , color-2.png.
check image :

Conclusion:

1.the feature-2.png contain:

background of background-2.png, (checked )
shape of shape-2.png, (checked )
texture of color-2.png , (checked )
but different pose from pose-2.png (failed)

2.the code-2.png contain:

background of background-2.png, (checked )
shape of shape-2.png, (maybe checked )
texture of color-2.png , (checked )
but different pose from pose-2.png (checked)

My question is why it happened?

Wrong comment

background_stage network : this line comment

because ngf = cfg.GAN_GF_DIM = 64
So ngf*8 = 512 .
The output feature of self.fc is ngf*8*4*4 = 512 *4*4 , not 1024*4*4
So this function all comment is wrong