pixray/pixray_notebooks

pixray demo notebooks

pixray

Pixray is an image generation system. It combines previous ideas including:

• Perception Engines which uses image augmentation and iteratively optimises images against an ensemble of classifiers
• CLIP guided GAN imagery from Ryan Murdoch and Katherine Crowson as well as modifictions such as CLIPDraw from Kevin Frans
• Useful ways of navigating latent space from Sampling Generative Networks
• (more to come)

pixray it itself a python library and command line utility, but is also friendly to running on line in Google Colab notebooks.

There is currently some documentation on options. Also checkout THE DEMO NOTEBOOKS or join in the discussion on discord.

Usage

Be sure to git clone --recursive to also get submodules.

You can install pip install -r requirements.txt and then pip install basicsr manually in a fresh python 3.8 environment (eg: using conda). After that you can use the included pixray.py command line utility:

python pixray.py --drawer=pixel --prompt=sunrise --outdir sunrise01

pixray can also be run from within your own python code, like this

import pixray
pixray.run("an extremely hairy panda bear", "vdiff", custom_loss="aesthetic", outdir="outputs/hairout")

Examples of pixray colab notebooks can be found in this separate repo.

running in a Docker using Cog is also possible. First, install Docker and Cog, then you can use cog run to run Pixray inside Docker. For example:

cog run python pixray.py --drawer=pixel --prompt=sunrise --outdir sunrise01

pixray_notebooks's Issues

Swap Model Attribute Error

When trying to run the Swap Model notebook, on the final code section titled 'Now Run', the following error pops up:

AttributeError: module 'PIL.TiffTags' has no attribute 'IFD'

error: `return_transform` is deprecated.

I installed everything in requirements.txt according to your instructions and ran the demo command:

python pixray.py --prompt=sunrise

and got this error:

...
Restored from models/vqgan_imagenet_f16_16384.ckpt
100%|███████████████████████████████████████| 338M/338M [00:56<00:00, 6.25MiB/s]
Loaded CLIP ViT-B/32: 224x224 and 151.28M params
100%|███████████████████████████████████████| 335M/335M [00:55<00:00, 6.29MiB/s]
Loaded CLIP ViT-B/16: 224x224 and 149.62M params

Traceback (most recent call last):
  File "pixray.py", line 2135, in <module>
    main()
  File "pixray.py", line 2129, in main
    do_init(settings)
  File "pixray.py", line 648, in do_init
    make_cutouts = MakeCutouts(cut_size, args.num_cuts, cut_pow=args.cut_pow)
  File "pixray.py", line 414, in __init__
    augmentations.append(MyRandomPerspective(distortion_scale=0.40, p=0.7, return_transform=True))
  File "/opt/anaconda3/envs/autothread/lib/python3.8/site-packages/kornia/augmentation/_2d/geometric/perspective.py", line 71, in __init__
    super().__init__(p=p, return_transform=return_transform, same_on_batch=same_on_batch, keepdim=keepdim)
  File "/opt/anaconda3/envs/autothread/lib/python3.8/site-packages/kornia/augmentation/base.py", line 183, in __init__
    raise ValueError(
ValueError: `return_transform` is deprecated. Please access the transformation matrix with `.transform_matrix`. For chained matrices, please use `AugmentationSequential`.

This doesn't seem like an install error, but a version issue. Can you clarify?
Here's my pip environment list

(autothread) /Volumes/LEGX/autothread/pixray$ pip3 list
Package                                           Version
------------------------------------------------- --------------
absl-py                                           1.0.0
addict                                            2.4.0
aiohttp                                           3.8.1
aiosignal                                         1.2.0
antlr4-python3-runtime                            4.8
appdirs                                           1.4.4
appnope                                           0.1.3
asttokens                                         2.0.5
async-timeout                                     4.0.2
attrs                                             21.4.0
audioread                                         2.1.9
backcall                                          0.2.0
basicsr                                           1.3.5
braceexpand                                       0.1.7
cachetools                                        5.0.0
certifi                                           2021.10.8
cffi                                              1.15.0
charset-normalizer                                2.0.12
clip                                              1.0
cmake                                             3.22.4
coloredlogs                                       15.0.1
colorthief                                        0.2.1
cssutils                                          2.4.0
cycler                                            0.11.0
decorator                                         4.4.2
deepspeech                                        0.9.3
einops                                            0.4.1
executing                                         0.8.3
fonttools                                         4.33.3
frozenlist                                        1.3.0
fsspec                                            2022.3.0
ftfy                                              6.1.1
future                                            0.18.2
google-auth                                       2.6.6
google-auth-oauthlib                              0.4.6
greenlet                                          1.1.2
grpcio                                            1.44.0
humanfriendly                                     10.0
idna                                              3.3
imageio                                           2.18.0
imageio-ffmpeg                                    0.4.7
importlib-metadata                                4.11.3
ipython                                           8.3.0
jedi                                              0.18.1
Jinja2                                            3.1.2
joblib                                            1.1.0
jsonpatch                                         1.32
jsonpointer                                       2.3
kiwisolver                                        1.4.2
kornia                                            0.6.4
librosa                                           0.9.1
llvmlite                                          0.38.0
lmdb                                              1.3.0
Markdown                                          3.3.6
MarkupSafe                                        2.1.1
matplotlib                                        3.5.1
matplotlib-inline                                 0.1.3
moviepy                                           1.0.3
multidict                                         6.0.2
networkx                                          2.8
numba                                             0.55.1
numpy                                             1.22.3
oauthlib                                          3.2.0
omegaconf                                         2.1.2
opencv-python                                     4.5.5.64
packaging                                         21.3
pandas                                            1.4.2
parso                                             0.8.3
perlin-numpy                                      0.0.0
pexpect                                           4.8.0
pickleshare                                       0.7.5
Pillow                                            9.1.0
pip                                               21.2.4
pooch                                             1.6.0
proglog                                           0.1.9
prompt-toolkit                                    3.0.29
protobuf                                          3.20.1
ptyprocess                                        0.7.0
pure-eval                                         0.2.2
pyaml                                             21.10.1
pyasn1                                            0.4.8
pyasn1-modules                                    0.2.8
pycparser                                         2.21
pyDeprecate                                       0.3.2
pydub                                             0.25.1
Pygments                                          2.12.0
pymongo                                           4.1.1
pyobjc                                            8.5
pyobjc-core                                       8.5
pyobjc-framework-Accessibility                    8.5
pyobjc-framework-Accounts                         8.5
pyobjc-framework-AddressBook                      8.5
pyobjc-framework-AdServices                       8.5
pyobjc-framework-AdSupport                        8.5
pyobjc-framework-AppleScriptKit                   8.5
pyobjc-framework-AppleScriptObjC                  8.5
pyobjc-framework-ApplicationServices              8.5
pyobjc-framework-AppTrackingTransparency          8.5
pyobjc-framework-AudioVideoBridging               8.5
pyobjc-framework-AuthenticationServices           8.5
pyobjc-framework-AutomaticAssessmentConfiguration 8.5
pyobjc-framework-Automator                        8.5
pyobjc-framework-AVFoundation                     8.5
pyobjc-framework-AVKit                            8.5
pyobjc-framework-BusinessChat                     8.5
pyobjc-framework-CalendarStore                    8.5
pyobjc-framework-CallKit                          8.5
pyobjc-framework-CFNetwork                        8.5
pyobjc-framework-ClassKit                         8.5
pyobjc-framework-CloudKit                         8.5
pyobjc-framework-Cocoa                            8.5
pyobjc-framework-Collaboration                    8.5
pyobjc-framework-ColorSync                        8.5
pyobjc-framework-Contacts                         8.5
pyobjc-framework-ContactsUI                       8.5
pyobjc-framework-CoreAudio                        8.5
pyobjc-framework-CoreAudioKit                     8.5
pyobjc-framework-CoreBluetooth                    8.5
pyobjc-framework-CoreData                         8.5
pyobjc-framework-CoreHaptics                      8.5
pyobjc-framework-CoreLocation                     8.5
pyobjc-framework-CoreMedia                        8.5
pyobjc-framework-CoreMediaIO                      8.5
pyobjc-framework-CoreMIDI                         8.5
pyobjc-framework-CoreML                           8.5
pyobjc-framework-CoreMotion                       8.5
pyobjc-framework-CoreServices                     8.5
pyobjc-framework-CoreSpotlight                    8.5
pyobjc-framework-CoreText                         8.5
pyobjc-framework-CoreWLAN                         8.5
pyobjc-framework-CryptoTokenKit                   8.5
pyobjc-framework-DataDetection                    8.5
pyobjc-framework-DeviceCheck                      8.5
pyobjc-framework-DictionaryServices               8.5
pyobjc-framework-DiscRecording                    8.5
pyobjc-framework-DiscRecordingUI                  8.5
pyobjc-framework-DiskArbitration                  8.5
pyobjc-framework-DVDPlayback                      8.5
pyobjc-framework-EventKit                         8.5
pyobjc-framework-ExceptionHandling                8.5
pyobjc-framework-ExecutionPolicy                  8.5
pyobjc-framework-ExternalAccessory                8.5
pyobjc-framework-FileProvider                     8.5
pyobjc-framework-FileProviderUI                   8.5
pyobjc-framework-FinderSync                       8.5
pyobjc-framework-FSEvents                         8.5
pyobjc-framework-GameCenter                       8.5
pyobjc-framework-GameController                   8.5
pyobjc-framework-GameKit                          8.5
pyobjc-framework-GameplayKit                      8.5
pyobjc-framework-ImageCaptureCore                 8.5
pyobjc-framework-IMServicePlugIn                  8.5
pyobjc-framework-InputMethodKit                   8.5
pyobjc-framework-InstallerPlugins                 8.5
pyobjc-framework-InstantMessage                   8.5
pyobjc-framework-Intents                          8.5
pyobjc-framework-IntentsUI                        8.5
pyobjc-framework-IOSurface                        8.5
pyobjc-framework-iTunesLibrary                    8.5
pyobjc-framework-KernelManagement                 8.5
pyobjc-framework-LatentSemanticMapping            8.5
pyobjc-framework-LaunchServices                   8.5
pyobjc-framework-libdispatch                      8.5
pyobjc-framework-LinkPresentation                 8.5
pyobjc-framework-LocalAuthentication              8.5
pyobjc-framework-LocalAuthenticationEmbeddedUI    8.5
pyobjc-framework-MailKit                          8.5
pyobjc-framework-MapKit                           8.5
pyobjc-framework-MediaAccessibility               8.5
pyobjc-framework-MediaLibrary                     8.5
pyobjc-framework-MediaPlayer                      8.5
pyobjc-framework-MediaToolbox                     8.5
pyobjc-framework-Metal                            8.5
pyobjc-framework-MetalKit                         8.5
pyobjc-framework-MetalPerformanceShaders          8.5
pyobjc-framework-MetalPerformanceShadersGraph     8.5
pyobjc-framework-MetricKit                        8.5
pyobjc-framework-MLCompute                        8.5
pyobjc-framework-ModelIO                          8.5
pyobjc-framework-MultipeerConnectivity            8.5
pyobjc-framework-NaturalLanguage                  8.5
pyobjc-framework-NetFS                            8.5
pyobjc-framework-Network                          8.5
pyobjc-framework-NetworkExtension                 8.5
pyobjc-framework-NotificationCenter               8.5
pyobjc-framework-OpenDirectory                    8.5
pyobjc-framework-OSAKit                           8.5
pyobjc-framework-OSLog                            8.5
pyobjc-framework-PassKit                          8.5
pyobjc-framework-PencilKit                        8.5
pyobjc-framework-Photos                           8.5
pyobjc-framework-PhotosUI                         8.5
pyobjc-framework-PreferencePanes                  8.5
pyobjc-framework-PushKit                          8.5
pyobjc-framework-Quartz                           8.5
pyobjc-framework-QuickLookThumbnailing            8.5
pyobjc-framework-ReplayKit                        8.5
pyobjc-framework-SafariServices                   8.5
pyobjc-framework-SceneKit                         8.5
pyobjc-framework-ScreenCaptureKit                 8.5
pyobjc-framework-ScreenSaver                      8.5
pyobjc-framework-ScreenTime                       8.5
pyobjc-framework-ScriptingBridge                  8.5
pyobjc-framework-SearchKit                        8.5
pyobjc-framework-Security                         8.5
pyobjc-framework-SecurityFoundation               8.5
pyobjc-framework-SecurityInterface                8.5
pyobjc-framework-ServiceManagement                8.5
pyobjc-framework-ShazamKit                        8.5
pyobjc-framework-Social                           8.5
pyobjc-framework-SoundAnalysis                    8.5
pyobjc-framework-Speech                           8.5
pyobjc-framework-SpriteKit                        8.5
pyobjc-framework-StoreKit                         8.5
pyobjc-framework-SyncServices                     8.5
pyobjc-framework-SystemConfiguration              8.5
pyobjc-framework-SystemExtensions                 8.5
pyobjc-framework-UniformTypeIdentifiers           8.5
pyobjc-framework-UserNotifications                8.5
pyobjc-framework-UserNotificationsUI              8.5
pyobjc-framework-VideoSubscriberAccount           8.5
pyobjc-framework-VideoToolbox                     8.5
pyobjc-framework-Virtualization                   8.5
pyobjc-framework-Vision                           8.5
pyobjc-framework-WebKit                           8.5
pyparsing                                         3.0.8
python-dateutil                                   2.8.2
pytorch-lightning                                 1.6.2
pytorch-ranger                                    0.1.1
pyttsx3                                           2.90
pytz                                              2022.1
PyWavelets                                        1.3.0
PyYAML                                            6.0
pyzmq                                             22.3.0
regex                                             2022.4.24
requests                                          2.27.1
requests-oauthlib                                 1.3.1
resampy                                           0.2.2
resmem                                            1.1.4
rsa                                               4.8
scikit-image                                      0.19.2
scikit-learn                                      1.0.2
scipy                                             1.8.0
seaborn                                           0.11.2
setuptools                                        61.2.0
six                                               1.16.0
SoundFile                                         0.10.3.post1
sox                                               1.4.1
SpeechRecognition                                 3.8.1
SQLAlchemy                                        1.4.36
stack-data                                        0.2.0
svgpathtools                                      1.4.4
svgwrite                                          1.4.2
taming-transformers                               0.0.1
tb-nightly                                        2.9.0a20220502
tensorboard                                       2.9.0
tensorboard-data-server                           0.6.1
tensorboard-plugin-wit                            1.8.1
threadpoolctl                                     3.1.0
tifffile                                          2022.4.28
timm                                              0.5.4
torch                                             1.11.0
torch-optimizer                                   0.3.0
torch-tools                                       0.1.5
torchfile                                         0.1.0
torchmetrics                                      0.8.1
torchvision                                       0.12.0
tornado                                           6.1
tqdm                                              4.64.0
traitlets                                         5.1.1
typing_extensions                                 4.2.0
urllib3                                           1.26.9
visdom                                            0.1.8.9
wcwidth                                           0.2.5
websocket-client                                  1.3.2
Werkzeug                                          2.1.2
wheel                                             0.37.1
yapf                                              0.32.0
yarl                                              1.7.2
zipp                                              3.8.0