serceman/fontogen

Hey, Computer, Make Me a Font

fontogen's Issues

Accidentially used Encoder instead of Decoder?

Hey there, great project!

I have just one thing that I am not sure if I misunderstood or you did. You wanted to reproduce the architecture of IconShop, but to my understanding, IconShop is a decoder-only transformer (like a language model).
Excerpt from their paper:

I first thought you had a minor mistake in your blog post, but also there you wrote encoder-only:

An encoder-only autoregressive transformer is self-contradictory, isn't it?

Also here, you took the xFormerEncoderConfig instead of the (imo) correct xFormerDecoderConfig.

fontogen/model/model.py

Lines 201 to 227 in de1e50b

	x_config = xFormerEncoderConfig(
	num_layers=num_layers,
	dim_model=d_model,
	feedforward_config={
	"name": "MLP",
	"dropout": 0.1,
	"activation": "gelu",
	"hidden_layer_multiplier": 4,
	},
	multi_head_config={
	"num_heads": nhead,
	"attention": BlockSparseAttention(
	num_heads=nhead,
	layout=BigBirdSparsityConfig(
	num_heads=nhead,
	block_size=block_size,
	num_global_blocks=1,
	num_random_blocks=2,
	num_sliding_window_blocks=int(3 * max_glyph_tokens / block_size)
	).make_layout(seq_len=self.seq_len),
	block_size=block_size,
	dropout=0.1,
	causal=True,
	)
	},
	)
	self.transformer = xFormer(x_config)

I haven't thought this through what the implications are. You correctly used causal attention, so there should not be issues on that front. There might not even be an issue at all, as decoders (without cross-attention) are kinda just encoders with causal attention masking, but I don't know the xformers library and its inner workings. Just wanted to set the mental model straight.

Thanks for releasing your code and project!

Publish dataset?

Hello, is there info about the dataset used for training the model? It's pretty important for some use cases (like publishing games on steam) to be sure that all training data is licensed appropriately, and I can't find any mentions of what kind of fonts were used during training. Were they all free?

Triton Issue?

Sorry, but I tried to run the last step of the install, but I get this error, any ideas how to resolve?

pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly==2.1.0.dev20230801015042 --no-deps
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/
ERROR: Could not find a version that satisfies the requirement triton-nightly==2.1.0.dev20230801015042 (from versions: none)
ERROR: No matching distribution found for triton-nightly==2.1.0.dev20230801015042

How can I do the inference?

Hello, I have done the Installation. I am wondering how can I do the Inference after training?

# the input fonts
ls ./example/dataset/
# prepare the dataset and start training
./train_example.sh

After I run those two instrcutions, Can I get some checkpoints or pretrained weights?

so after that, How can I do the Inference?

The model needs to be re-trained on a large dataset of OFL fonts. If anyone would like to contribute and re-train the model, please reach out and I'll be happy to help you set up the environment.

I'd like to contribute and re-train the model. I have a 4090 GPU.
btw, will you publish your pretrained weights in the future? I guess it is tricky to have a good training result because of many hyper parameters.

This project is awesome! Looking forward to your reply.

Blocksparse is not available: the current GPU does not expose Tensor cores

Attempting to run this using my nvidia RTX 2070 Super (Turing architecture), which can run other stuff like stable-diffusion-webui or koboldcpp just fine...

> python server.py
Blocksparse is not available: the current GPU does not expose Tensor cores
Traceback (most recent call last):
  File "path/fontogen/server.py", line 10, in <module>
    from sampler import create_sampler
  File "path/fontogen/sampler.py", line 11, in <module>
    from model.model import FontogenModule
  File "path/fontogen/model/model.py", line 11, in <module>
    from xformers.components.attention import BlockSparseAttention
ImportError: cannot import name 'BlockSparseAttention' from 'xformers.components.attention' (path/.virtualenvs/fontogen-Ir94UGpw/lib/python3.10/site-packages/xformers/components/attention/__init__.py)

Looking at the Blocksparse information, the nvidia Turing architecture isn't mentioned, so perhaps I am unable to try out fontogen?

Hello, may I ask do you reproduce Iconshop?

I came across the IconShop[2](https://serce.me/posts/02-10-2023-hey-computer-make-me-a-font#references) paper which achieved pretty impressive results. It took me some time to reproduce them by building a model based on the description in the paper.

Hello, Do you reproduce the result of Iconshop with the some code in this project(except the preprocess code)?

I saw your blog and find Iconshop is also quite interesting.

May I ask can I reproduce Iconshop using your code by changing the dataset in this project into FIGR-8-SVG(dataset mentioned in the paper Iconshop)?

Thanks and sorry for bothering.

Can I give it examples?

Would be awesome to document a way for me to give example fonts I want to emulate, rather than only a text-based prompt.

PS thanks for sharing the code for this!

Can this project generate lowercase letters and numbers?

Steps to train a model?

Can you lay out all the steps to train a model?

Thanks