dribnet/clipit_old

VQGAN+CLIP with some additional tuning. For notebooks and the command line.

clipit_old's Issues

static resources

sometimes it's useful to have static resources associated with the project. this is where they will go.

Repo is a fork, can't search

Currently clipit is a fork so the code can't be searched and it won't show up in searches on Github either. It's not even similar to VQGAN-CLIP so it's definitely beyond the usual idea of a fork.

Before things progress too much further it'd be for the best if this was remade so it can be searched properly, it's making it difficult trying to find what controls the color of diffvg and where it is, and it'll only get worse.

missing requirements

braceexpand
perlin_numpy

Build error on Windows

With windows 10 + VS2017/2019 (I tried both), I got the folling error preventing me from installing diffvg.

 diffvg.cpp
c:\[...]\diffvg\pybind11\include\pybind11\cast.h(1405): error : too few arguments for template template pa
rameter "Tuple" [C:[...]\diffvg\build\temp.win-amd64-3.9\Release\diffvg.vcxproj]
            detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=
  std::pair, Ts=<T1, T2>]"
  (1483): here

Is there anyway to make this work...?

fyi - init_image

#@title Parameters
prompts = "trending on artstation" #@param {type:"string"}

#@markdown you can trade off speed for quality

quality = "best" #@param ["draft", "normal", "better", "best"]

#@markdown some people actually like square? ¯\(ツ)/¯

aspect = "square" #@param ["widescreen", "square"]
init_image = "1.jpeg"
#@markdown When you have the settings you want, press the play button on the left.
#@markdown The system will start drawing and show you results as it goes along.

#@markdown (I can get "better" running on the colabs, but not "best".
#@markdown Sometimes it is necessary to "Restart Runtime" to clear memory.)

Simple setup from menu choices

import clipit
clipit.reset_settings()
clipit.add_settings(prompts=prompts, aspect=aspect, quality=quality,init_image=init_image)

Optional: you could put extra settings here...

Apply these settings and run

settings = clipit.apply_settings()
clipit.do_init(settings)
clipit.do_run(settings)

diffvg RuntimeError on render

Hey! I'm trying to run clipit on my own hardware (with an RTX A5000). I've got the environment set up more or less like the Colab notebook. I'm running this:

import sys
sys.path.append("clipit")
import clipit
clipit.reset_settings()
clipit.add_settings(prompts="Beirut skyline", aspect="widescreen")
clipit.add_settings(quality="better", scale=2.5)
clipit.add_settings(use_pixeldraw=True)
clipit.add_settings(num_cuts=10)
settings = clipit.apply_settings()
clipit.do_init(settings)
clipit.do_run(settings)

And getting a mysterious error from when calling loss.backward() in the train function:

(lightning) root@pytorch-6dc4c57d8d-l2h26:/app# python tmp.py
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/comet_ml/monkey_patching.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/pkg_resources/__init__.py:1131: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
  self, resource_name
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/mlflow/types/schema.py:49: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  binary = (7, np.dtype("bytes"), "BinaryType", np.object)
Using device: cuda:0
Optimising using: AdamP
Using text prompts: ['Beirut skyline']
Using seed: 6415327989839999698
0it [00:00, ?it/s]/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode)
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/kornia/utils/helpers.py:96: UserWarning: torch.solve is deprecated in favor of torch.linalg.solveand will be removed in a future PyTorch release.
torch.linalg.solve has its arguments reversed and does not return the LU factorization.
To get the LU factorization see torch.lu, which can be used with torch.lu_solve or torch.lu_unpack.
X = torch.solve(B, A).solution
should be replaced with
X = torch.linalg.solve(A, B) (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448265233/work/aten/src/ATen/native/BatchLinearAlgebra.cpp:760.)
  out1, out2 = torch.solve(input.to(dtype), A.to(dtype))
iter: 0, loss: 2.97693, losses: 1.04082, 0.961298, 0.974818
0it [00:01, ?it/s]Oops: runtime error:
Try reducing --num-cuts to save memory
0it [00:01, ?it/s]
Traceback (most recent call last):
  File "tmp.py", line 11, in <module>
    clipit.do_run(settings)
  File "clipit/clipit.py", line 933, in do_run
    raise e
  File "clipit/clipit.py", line 925, in do_run
    train(args, cur_iteration)
  File "clipit/clipit.py", line 845, in train
    loss.backward()
  File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
  File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/autograd/function.py", line 87, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore[attr-defined]
  File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/diffvg-0.0.1-py3.7-linux-x86_64.egg/pydiffvg/render_pytorch.py", line 709, in backward
    eval_positions.shape[0])
RuntimeError

I'm kind of at a loss, here (pun intended), since the RuntimeError doesn't have any message associated with it, and the function in question seems to be a compiled C function. Any pointers would be appreciated.

PixelDrawer support for limited colors

The PixelDrawer currently has no support for limiting the colors - it only support full three channel RGB. It would be preferable to be able to limit the colorspace to either subsets of RGB or to a fixed color palette.

As an initial step, I would like to create a version of the PixelDrawer class that supports a grayscale image where R=G=B. In principle this should be possible by optimizing only one tensor per pixel and then having that tensor be connected to the R, G, and B channels.

The hope is that the PixelDrawer would be able to evolve from there to next support a discrete set of fixed colors, and ultimately also to optionally optimize the color lookup table as it also optimizes the color assignments of the pixels.

Note also that this issue is related up upstream issue BachiLi/diffvg#23