ktrk115/const_layout

Official implementation of the MM'21 paper "Constrained Graphic Layout Generation via Latent Optimization" (LayoutGAN++, CLG-LO, and Layout evaluation)

[MM'21] Constrained Graphic Layout Generation via Latent Optimization

This repository provides the official code for the paper "Constrained Graphic Layout Generation via Latent Optimization", especially the code for:

• LayoutGAN++: generative adversarial networks for layout generation
• CLG-LO: a framework for generating layouts that satisfy constraints
• Layout evaluation: measuring the quantitative metrics of Layout FID, Maximum IoU, Alignment, and Overlap for generated layouts

Installation

1. Clone this repository

   git clone https://github.com/ktrk115/const_layout.git
   cd const_layout

2. Create a new conda environment (Python 3.8)

   conda create -n const_layout python=3.8
   conda activate const_layout

3. Install PyTorch 1.8.1 and PyTorch Geometric 1.7.2. An example of the PyG installation command is shown below.

   pip install torch-scatter==2.0.7 -f https://data.pyg.org/whl/torch-1.8.1+cu111.html
   pip install torch-sparse==0.6.10 -f https://data.pyg.org/whl/torch-1.8.1+cu111.html
   pip install torch-geometric==1.7.2

4. Install the other dependent libraries

   pip install -r requirements.txt

5. Prepare data (see this instruction)

6. Download pre-trained models

   ./download_model.sh

7. Install ImageMagick for visualization

Development environment

• Ubuntu 18.04, CUDA 11.1

LayoutGAN++

Architecture

Training animation

Generate layouts with LayoutGAN++

python generate.py pretrained/layoutganpp_rico.pth.tar --out_path output/generated_layouts.pkl --num_save 5

Train LayoutGAN++ model

python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8

CLG-LO

w/ beautification constraints	w/ relational constraints

Generate layouts with beautification constraints

python generate_const.py pretrained/layoutganpp_publaynet.pth.tar --const_type beautify --out_path output/beautify/generated_layouts.pkl --num_save 5

Generate layouts with relational constraints

python generate_const.py pretrained/layoutganpp_publaynet.pth.tar --const_type relation --out_path output/relation/generated_layouts.pkl --num_save 5

Layout evaluation

Evaluate generated layouts

python eval.py rico output/generated_layouts.pkl

A pickle file should be a list of layouts, where each layout is a tuple of bounding boxes and labels. The bounding box is represented by [x, y, width, height] in normalized coordinates, and the label is represented by an index. An example is shown below.

In [x]: layouts
Out[x]:
[(array([[0.47403812, 0.11276676, 0.6250037 , 0.02210438],
         [0.49971417, 0.8550553 , 0.81388366, 0.03492427],
         [0.49919674, 0.47857162, 0.81024694, 0.7070079 ]], dtype=float32),
  array([0, 0, 3]),
  ...

Citation

If this repository helps your research, please consider citing our paper.

@inproceedings{Kikuchi2021,
    title = {Constrained Graphic Layout Generation via Latent Optimization},
    author = {Kotaro Kikuchi and Edgar Simo-Serra and Mayu Otani and Kota Yamaguchi},
    booktitle = {ACM International Conference on Multimedia},
    series = {MM '21},
    year = {2021},
    pages = {88--96},
    doi = {10.1145/3474085.3475497}
}

Licence

GNU AGPLv3

Related repositories

• https://github.com/JiananLi2016/LayoutGAN-Tensorflow
• https://github.com/alexandre01/deepsvg

const_layout's Issues

CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm

@ktrk115 hi, when i run the train code, I came across the problem as the title list. my environmet is :

torch 1.8.1+cu111
torch-geometric 1.7.2
torch-scatter 2.0.7
torch-sparse 0.6.10
torchaudio 0.8.1
torchvision 0.9.1+cu111

and I install pytorch with: pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

if i install pytorch with conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.3 -c pytorch -c conda-forge, I can not use gpu, for there is always cpu version pytorch

look forward to your reply.

ModuleNotFoundError: No module named 'torch'，auther can add "torch" to requirement.txt

Why add "to_dense_batch" to process the location information x and category label y?

What does this mean, please? Why add "to_dense_batch" to process the location information x and category label y?

for i, data in enumerate(train_dataloader):
data = data.to(device)
label, mask = to_dense_batch(data.y, data.batch)#？？？？？？？？？？？？？？？
bbox_real, _ = to_dense_batch(data.x, data.batch)#？？？？？？？？？？？？？？？

how to use the model with local dataset?

I want to train it with local dataset, how to train the LayoutFID model? Could you provide the training codes?

Quantitative comparison of unconstrained layout generation

I tried re-training LayoutGAN++ used magazine dataset.
But, I can't reappearance Quantitative comparison, especially overlap.
Results of multiple trials, overlap average 40 over.

ex.

Dataset: magazine
        FID: 13.88
        Max. IoU: 0.26
        Alignment: 0.82
        Overlap: 40.31

I think using the hyperparameters from the paper.
By any chance, cause in the version of os or library?

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"

pytorch-fid==0.2.1
torch==1.11.0+cu113
torch-cluster==1.6.0
torch-geometric==2.0.4
torch-scatter==2.0.9
torch-sparse==0.6.13
torch-spline-conv==1.2.1
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113

How to keep aspect ratio during optimization?

How to preserve image aspect ratio while performing beautification constraints?
@ktrk115 Thanks.

data link is in 404 status, how can i get the train data?

Download rico_dataset_v0.1_semantic_annotations.zip from "UI Screenshots and Hierarchies with Semantic Annotations" and decompress it.
but it is

Seeking for help about training code

Hello, could you provide the LayoutGan training code?(not LayoutGANpp)

assert data.attr[0]['has_canvas_element']

about the train loss

thank for you work. I train the model with command:

python train.py --dataset rico --batch_size 64 --iteration 200000 --latent_size 4 --lr 1e-05 --G_d_model 256 --G_nhead 4 --G_num_layers 8 --D_d_model 256 --D_nhead 4 --D_num_layers 8

and it will show me that. does it look normal ? the generator loss become large and Discriminator loss become small :

When I input ./download_model.sh, it runs like this...WHY

How to specify relation constraints?

Thank you for publishing your work! I want to ask how I can specify relational constraints (e.g. Image at top) because in the code now, it seems that all relational constraints are being used for optimisation.

furniture placement

I am on a project concerning furniture placement in a room，the target is place some furnitures in a room.The input is like
.The image includes wall, window and door line segments, the room shape not only rectangle but also polygon with right angle.

1. how can I change the input with any shape?
2. how can I consider the window and door into model，because window and door has a big connection with furniture placement

Thx a lot, look forward to your response

How to train the model on local PC

I have written all the code from the files into the google colab and it trained perfectly but when I tries training it on my PC using jupyter notebook it does not load any data from the dataloaders. What can be the solution.

for data in train_dataloader:
print(data)
break

this works in google colab but not in Jupyter notebook

Unable to reproduce the CLG-LO

Thank you for source code.
I get an error when I run CLG-LO

If possible, please share the how to fix.

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.python-2022.7.11371008/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vscode/const_layout/generate_const.py", line 199, in <module>
    main()
  File "/home/vscode/const_layout/generate_const.py", line 182, in main
    save_gif(out_path, j, netG,
  File "/home/vscode/const_layout/generate_const.py", line 52, in save_gif
    subprocess.run(['convert', '-delay', '50',
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'convert'

Potential issue in max_iou computation

I have a question regarding the implementation of max_iou computation in the following file:

https://github.com/ktrk115/const_layout/blob/5287480505939345543fff0b9f2e5d541e6f84e2/metric.py#LL92C9-L92C21

For line 92: n = len(_bi)

In my understanding, if we simply take len(_bi), when _bj contains more elements than _bi and when in _bj there are elements beyond index n that match better with elements in _bi, these elements are ignored and thus producing sub-optimal max_iou calculations.

Would it be a better solution to create a len(_bi) by len(_bj) mesh, and handle non-square matrices as follows?

python generate.py提示内存溢出，但内存确实是充足的，大家有遇到相似情况吗

（1）python generate.py
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

（2）内存情况

How to change number of elements?

Hey,
I've been trying to train a model on Rico with more data
In rico.py, I can see that number of elements are capped at 9

I've been going through your generator and discriminator code and I would greatly appreciate some help in where I need to make changes so the models can accept in data samples with bboxes and labels > 9.

Thanks so much!
Sukriti

LayoutNet - how to train?

Could you share a script to train LayoutNet for FID calculation?
I see the model architecture file, could you maybe share the code to train it or the loss functions used?

Key error with constrained generation

Hey,
Thanks for releasing your code and environment. I was able to generate samples using pre-trained models.
However, when I try to generate samples with constraints, I get the following key error:

  File "generate_const.py", line 199, in <module>
    main()
  File "generate_const.py", line 148, in main
    for z in optimizer.generator(z, data):
  File "/home/sukrverm/const_layout/clg/auglag.py", line 96, in generator
    assert data.attr[0]['has_canvas_element']
KeyError: 0

I suppose this is some attribute of publaynet data that is False here.
Do I need to edit the get_dataset Publaynet class to remove such examples before hand?

7.Install ImageMagick for visualization can it be more detailed?

I am a green hand in AI model, after i got these pretrained models, i don't know the next step...Please help me