Comments (3)
You are not able to provide your own input image. You can just sample some latent z and then pass it through the (pretrained) generator. Changing the expressions/appearance is then just simply done via perturbing the latent z in the correct direction.
Isn' that net a half of an autoencoder, so if we had an encoder, it should have been possible to create a latent vector corresponding to the image? Anyway, even if it isn't, restoring the latent vector corresponding to the image with backprop should work, though not as efficient, as if we had an encoder.
from pychubby.
So the StyleGAN only contains the "decoder" path - the generator. Assuming that one wants to use their pretrained model then the only possibility left is to find a z
such that G(z) ~ x
. As you pointed out it can be done through backprop and in general it is not clear whether suchz
even exists. Sure, one can just hope that we are close enough but I wonder what artifacts it is going to create.
I actually did some research and found one project that tries to achieve something very similar https://github.com/Puzer/stylegan-encoder. It's great and I totally commend the author, however, let me point out a few problems:
- Requires gpus
- Not very user friendly (no setup.py -> not on PyPI + requires some background in ML). It's a fork of the official StyleGAN so I understand there are inherent limitations
- Only provides 3 latent directions (smile, age and gender)
- Aligns the original face and then only works with the aligned version of the image. That means that additional logic for changing the initial image would have to be implemented + logic for multiple faces
- I haven't tested it but I would guess that in some cases it creates artifacts that are either due to imperfections in the pretrained model, encoding approximation (x -> z) or the latent space interpolations. For all of these there is no quick fix.
I am actually really interested in extending some of the ideas from that project. However, for now I just wanted to list some things the "Neural network-based programs" might struggle with.
Some links
from pychubby.
Hey @KOLANICH, thank you for your interest.
pychubby
is purely based on geometric transformations (warping) of the input image. It cannot create new textures, objects, etc. So just from this point of view it is fundamentally different from pix2pix GANs and similar.
See below some comments on the two links:
Transparent Latent GAN
-
You are not able to provide your own input image. You can just sample some latent
z
and then pass it through the (pretrained) generator. Changing the expressions/appearance is then just simply done via perturbing the latentz
in the correct direction. -
The number of actions is limited by the attributes the feature extractor network was trained on Dataset Celebrities
Style GAN
This is a brilliant paper however again I would like to point out that the goal of it is fundamentally different from pychubby
.
- Same as for the first link, you are not able to provide your own input image.
I would be more than happy to discuss further if you want to!
Cheers
from pychubby.
Related Issues (19)
- just trying to store a readme image HOT 1
- Better error message when estimate class method fails
- Cmake a required dependency on osx HOT 2
- Define new actions HOT 5
- "shape_predictor_68_face_landmarks.dat" looks like under non-commercial license. HOT 1
- Error on performing action HOT 1
- TypeError: tuple indices must be integers or slices, not str HOT 4
- how to move landmark points.. pixel unit? HOT 1
- Make warping faster for big images HOT 3
- .
- When i apply a transform to an image, it shrinks the image. HOT 6
- Realtime warping HOT 2
- Use pychubby in android using Chaquopy sdk HOT 21
- Implement inversion of DF HOT 1
- Set python_requires in setup.py HOT 3
- Show a default in CLI
- Better error message when standard action applied on LandmarkFaces
- Fix code typos in docs (visualize -> visualization)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pychubby.