ktrk115/const_layout

Official implementation of the MM'21 paper "Constrained Graphic Layout Generation via Latent Optimization" (LayoutGAN++, CLG-LO, and Layout evaluation)

const_layout's Issues

Unable to reproduce the CLG-LO

Thank you for source code.
I get an error when I run CLG-LO

If possible, please share the how to fix.

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/const_layout/generate_const.py", line 199, in <module>
    main()
  File "/home/vscode/const_layout/generate_const.py", line 182, in main
    save_gif(out_path, j, netG,
  File "/home/vscode/const_layout/generate_const.py", line 52, in save_gif
    subprocess.run(['convert', '-delay', '50',
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'convert'

Key error with constrained generation

Hey,
Thanks for releasing your code and environment. I was able to generate samples using pre-trained models.
However, when I try to generate samples with constraints, I get the following key error:

  File "generate_const.py", line 199, in <module>
    main()
  File "generate_const.py", line 148, in main
    for z in optimizer.generator(z, data):
  File "/home/sukrverm/const_layout/clg/auglag.py", line 96, in generator
    assert data.attr[0]['has_canvas_element']
KeyError: 0

I suppose this is some attribute of publaynet data that is False here.
Do I need to edit the get_dataset Publaynet class to remove such examples before hand?

ModuleNotFoundError: No module named 'torch'，auther can add "torch" to requirement.txt

CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm

@ktrk115 hi, when i run the train code, I came across the problem as the title list. my environmet is :

torch 1.8.1+cu111
torch-geometric 1.7.2
torch-scatter 2.0.7
torch-sparse 0.6.10
torchaudio 0.8.1
torchvision 0.9.1+cu111

and I install pytorch with: pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

if i install pytorch with conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.3 -c pytorch -c conda-forge, I can not use gpu, for there is always cpu version pytorch

look forward to your reply.

python generate.py提示内存溢出，但内存确实是充足的，大家有遇到相似情况吗

（1）python generate.py
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

（2）内存情况

Seeking for help about training code

Hello, could you provide the LayoutGan training code?(not LayoutGANpp)

How to specify relation constraints?

Thank you for publishing your work! I want to ask how I can specify relational constraints (e.g. Image at top) because in the code now, it seems that all relational constraints are being used for optimisation.

Potential issue in max_iou computation

I have a question regarding the implementation of max_iou computation in the following file:

https://github.com/ktrk115/const_layout/blob/5287480505939345543fff0b9f2e5d541e6f84e2/metric.py#LL92C9-L92C21

For line 92: n = len(_bi)

In my understanding, if we simply take len(_bi), when _bj contains more elements than _bi and when in _bj there are elements beyond index n that match better with elements in _bi, these elements are ignored and thus producing sub-optimal max_iou calculations.

Would it be a better solution to create a len(_bi) by len(_bj) mesh, and handle non-square matrices as follows?

data link is in 404 status, how can i get the train data?

Download rico_dataset_v0.1_semantic_annotations.zip from "UI Screenshots and Hierarchies with Semantic Annotations" and decompress it.
but it is

How to change number of elements?

Hey,
I've been trying to train a model on Rico with more data
In rico.py, I can see that number of elements are capped at 9

I've been going through your generator and discriminator code and I would greatly appreciate some help in where I need to make changes so the models can accept in data samples with bboxes and labels > 9.

Thanks so much!
Sukriti

about the train loss

thank for you work. I train the model with command:

python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8

and it will show me that. does it look normal ? the generator loss become large and Discriminator loss become small :

how to use the model with local dataset?

I want to train it with local dataset, how to train the LayoutFID model? Could you provide the training codes?

Why add "to_dense_batch" to process the location information x and category label y?

What does this mean, please? Why add "to_dense_batch" to process the location information x and category label y?

for i, data in enumerate(train_dataloader):
data = data.to(device)
label, mask = to_dense_batch(data.y, data.batch)#？？？？？？？？？？？？？？？
bbox_real, _ = to_dense_batch(data.x, data.batch)#？？？？？？？？？？？？？？？

LayoutNet - how to train?

Could you share a script to train LayoutNet for FID calculation?
I see the model architecture file, could you maybe share the code to train it or the loss functions used?

assert data.attr[0]['has_canvas_element']

furniture placement

I am on a project concerning furniture placement in a room，the target is place some furnitures in a room.The input is like
.The image includes wall, window and door line segments, the room shape not only rectangle but also polygon with right angle.

1. how can I change the input with any shape?
2. how can I consider the window and door into model，because window and door has a big connection with furniture placement

Thx a lot, look forward to your response

How to keep aspect ratio during optimization?

How to preserve image aspect ratio while performing beautification constraints?
@ktrk115 Thanks.

How to train the model on local PC

I have written all the code from the files into the google colab and it trained perfectly but when I tries training it on my PC using jupyter notebook it does not load any data from the dataloaders. What can be the solution.

for data in train_dataloader:
print(data)
break

this works in google colab but not in Jupyter notebook

When I input ./download_model.sh, it runs like this...WHY

Quantitative comparison of unconstrained layout generation

I tried re-training LayoutGAN++ used magazine dataset.
But, I can't reappearance Quantitative comparison, especially overlap.
Results of multiple trials, overlap average 40 over.

ex.

Dataset: magazine
        FID: 13.88
        Max. IoU: 0.26
        Alignment: 0.82
        Overlap: 40.31

I think using the hyperparameters from the paper.
By any chance, cause in the version of os or library?

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"

pytorch-fid==0.2.1
torch==1.11.0+cu113
torch-cluster==1.6.0
torch-geometric==2.0.4
torch-scatter==2.0.9
torch-sparse==0.6.13
torch-spline-conv==1.2.1
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113

7.Install ImageMagick for visualization can it be more detailed?

I am a green hand in AI model, after i got these pretrained models, i don't know the next step...Please help me