ktrk115/const_layout

Official implementation of the MM'21 paper "Constrained Graphic Layout Generation via Latent Optimization" (LayoutGAN++, CLG-LO, and Layout evaluation)

const_layout's Issues

Key error with constrained generation

Hey,
Thanks for releasing your code and environment. I was able to generate samples using pre-trained models.
However, when I try to generate samples with constraints, I get the following key error:

  File "generate_const.py", line 199, in <module>
    main()
  File "generate_const.py", line 148, in main
    for z in optimizer.generator(z, data):
  File "/home/sukrverm/const_layout/clg/auglag.py", line 96, in generator
    assert data.attr[0]['has_canvas_element']
KeyError: 0

I suppose this is some attribute of publaynet data that is False here.
Do I need to edit the get_dataset Publaynet class to remove such examples before hand?

How to keep aspect ratio during optimization?

How to preserve image aspect ratio while performing beautification constraints?
@ktrk115 Thanks.

LayoutNet - how to train?

Could you share a script to train LayoutNet for FID calculation?
I see the model architecture file, could you maybe share the code to train it or the loss functions used?

When I input ./download_model.sh, it runs like this...WHY

Why add "to_dense_batch" to process the location information x and category label y?

What does this mean, please? Why add "to_dense_batch" to process the location information x and category label y?

for i, data in enumerate(train_dataloader):
data = data.to(device)
label, mask = to_dense_batch(data.y, data.batch)#？？？？？？？？？？？？？？？
bbox_real, _ = to_dense_batch(data.x, data.batch)#？？？？？？？？？？？？？？？

about the train loss

thank for you work. I train the model with command:

python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8

and it will show me that. does it look normal ? the generator loss become large and Discriminator loss become small :

Quantitative comparison of unconstrained layout generation

I tried re-training LayoutGAN++ used magazine dataset.
But, I can't reappearance Quantitative comparison, especially overlap.
Results of multiple trials, overlap average 40 over.

ex.

Dataset: magazine
        FID: 13.88
        Max. IoU: 0.26
        Alignment: 0.82
        Overlap: 40.31

I think using the hyperparameters from the paper.
By any chance, cause in the version of os or library?

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"

pytorch-fid==0.2.1
torch==1.11.0+cu113
torch-cluster==1.6.0
torch-geometric==2.0.4
torch-scatter==2.0.9
torch-sparse==0.6.13
torch-spline-conv==1.2.1
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113

how to use the model with local dataset?

I want to train it with local dataset, how to train the LayoutFID model? Could you provide the training codes?

ModuleNotFoundError: No module named 'torch'，auther can add "torch" to requirement.txt

Unable to reproduce the CLG-LO

Thank you for source code.
I get an error when I run CLG-LO

If possible, please share the how to fix.

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/const_layout/generate_const.py", line 199, in <module>
    main()
  File "/home/vscode/const_layout/generate_const.py", line 182, in main
    save_gif(out_path, j, netG,
  File "/home/vscode/const_layout/generate_const.py", line 52, in save_gif
    subprocess.run(['convert', '-delay', '50',
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'convert'

Potential issue in max_iou computation

I have a question regarding the implementation of max_iou computation in the following file:

https://github.com/ktrk115/const_layout/blob/5287480505939345543fff0b9f2e5d541e6f84e2/metric.py#LL92C9-L92C21

For line 92: n = len(_bi)

In my understanding, if we simply take len(_bi), when _bj contains more elements than _bi and when in _bj there are elements beyond index n that match better with elements in _bi, these elements are ignored and thus producing sub-optimal max_iou calculations.

Would it be a better solution to create a len(_bi) by len(_bj) mesh, and handle non-square matrices as follows?

How to train the model on local PC

I have written all the code from the files into the google colab and it trained perfectly but when I tries training it on my PC using jupyter notebook it does not load any data from the dataloaders. What can be the solution.

for data in train_dataloader:
print(data)
break

this works in google colab but not in Jupyter notebook

How to change number of elements?

Hey,
I've been trying to train a model on Rico with more data
In rico.py, I can see that number of elements are capped at 9

I've been going through your generator and discriminator code and I would greatly appreciate some help in where I need to make changes so the models can accept in data samples with bboxes and labels > 9.

Thanks so much!
Sukriti

data link is in 404 status, how can i get the train data?

Download rico_dataset_v0.1_semantic_annotations.zip from "UI Screenshots and Hierarchies with Semantic Annotations" and decompress it.
but it is

python generate.py提示内存溢出，但内存确实是充足的，大家有遇到相似情况吗

（1）python generate.py
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

（2）内存情况

How to specify relation constraints?

Thank you for publishing your work! I want to ask how I can specify relational constraints (e.g. Image at top) because in the code now, it seems that all relational constraints are being used for optimisation.

Seeking for help about training code

Hello, could you provide the LayoutGan training code?(not LayoutGANpp)

7.Install ImageMagick for visualization can it be more detailed?

I am a green hand in AI model, after i got these pretrained models, i don't know the next step...Please help me

assert data.attr[0]['has_canvas_element']

furniture placement

I am on a project concerning furniture placement in a room，the target is place some furnitures in a room.The input is like
.The image includes wall, window and door line segments, the room shape not only rectangle but also polygon with right angle.

1. how can I change the input with any shape?
2. how can I consider the window and door into model，because window and door has a big connection with furniture placement

Thx a lot, look forward to your response

CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm

@ktrk115 hi, when i run the train code, I came across the problem as the title list. my environmet is :

torch 1.8.1+cu111
torch-geometric 1.7.2
torch-scatter 2.0.7
torch-sparse 0.6.10
torchaudio 0.8.1
torchvision 0.9.1+cu111

and I install pytorch with: pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

if i install pytorch with conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.3 -c pytorch -c conda-forge, I can not use gpu, for there is always cpu version pytorch

look forward to your reply.