henzler / neuraltexture Goto Github PK
View Code? Open in Web Editor NEWLearning a Neural 3D Texture Space from 2D Exemplars [CVPR 2020]
Home Page: https://geometry.cs.ucl.ac.uk/projects/2020/neuraltexture/
License: MIT License
Learning a Neural 3D Texture Space from 2D Exemplars [CVPR 2020]
Home Page: https://geometry.cs.ucl.ac.uk/projects/2020/neuraltexture/
License: MIT License
Hi @henzler,
Have you tested training new models on PyTorch versions newer than 1.4?
I am doing a follow-up work based on this project and I have noticed that the model training performance degrades noticeably when I train in PyTorch 1.5 or later. Inference on models trained using PyTorch 1.4, via recent versions of PyTorch works just fine. I am guessing this issue is related to the noise operator implementation I obtained from this codebase.
hi I was trying to download pretrained models that you guys provided but the access was denied.
I was wondering if you guys limited the downloading on purpose and not providing it anymore
thank you
Hi there @henzler @madhawav
I cloned the repo as is and only made the changes mentioned in this pull request
I also got the weights as mentioned in the shell script.
I ran
But the results are horrible. I think maybe the weights were updated or some structure changed?
I have kept the versions as mentioned in the requirements.txt
Input:-
Outputs:-
Hi,
When I try to train on a new dataset, it fails with the following error.
[PYTHON_ENV_PATH]/neuraltexture/bin/python -u [PROJECT_ROOT]/code/train_neural_texture.py
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Use pytorch 1.4.0
Load config: configs/neural_texture/config_default.yaml
INFO:lightning:GPU available: True, used: True
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:23: RuntimeWarning: You have defined a `val_dataloader()` and have defined a `validation_step()`, you may also want to define `validation_epoch_end()` for accumulating stats.
warnings.warn(*args, **kwargs)
[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:23: RuntimeWarning: You have defined a `test_dataloader()` and have defined a `test_step()`, you may also want to define `test_epoch_end()` for accumulating stats.
warnings.warn(*args, **kwargs)
Validation sanity check: 0it [00:00, ?it/s]Traceback (most recent call last):
File "[PROJECT_ROOT]/neuraltexture/code/train_neural_texture.py", line 47, in <module>
trainer.fit(system)
File "[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 765, in fit
self.single_gpu_train(model)
File "[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 492, in single_gpu_train
self.run_pretrain_routine(model)
File "[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 896, in run_pretrain_routine
eval_results = self._evaluate(model,
File "[PYTHON_ENV_PATH]/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 322, in _evaluate
eval_results = model.validation_end(outputs)
File "[PROJECT_ROOT]/neuraltexture/code/systems/s_core.py", line 33, in validation_end
for key in outputs[0].keys():
IndexError: list index out of range
Process finished with exit code 1
My "config_default.yml" Is shown below:
version_name: neuraltexture_all_2d_single
device: cuda
n_workers: 8
n_gpus: 1
dim: 2
noise:
octaves: 8
logger:
log_files_every_n_iter: 1000
log_scalars_every_n_iter: 100
log_validation_every_n_epochs: 1
image:
image_res: &image_res 128 # (height, width)
texture:
e: &texture_e 64 # encoding size
dataset:
name: datasets.images
path: '../datasets/all'
use_single: -1 # -1 = all, 0,1,2 for single
system:
block_main:
model_texture_encoder:
model_params:
name: models.neural_texture.encoder
type: 'ResNet'
shape_in: [[3, *image_res, *image_res]]
bottleneck_size: 8
model_texture_mlp:
model_params:
name: models.neural_texture.mlp
type: 'MLP'
n_max_features: 128
n_blocks: 4
dropout_ratio: 0.0
non_linearity: 'relu'
bias: True
encoding: *texture_e
optimizer_params:
name: 'adam'
lr: 0.0001
weight_decay: 0.0001
scheduler_params:
name: 'none'
loss_params:
style_weight: 1.0
style_type: 'mse'
train:
epochs: 3
bs: 16
accumulate_grad_batches: 1
seed: 41127
Your help is much appreciated.
Hi, I read the paper, and it is said you can generate 3D texture.
But in the code, I only found the normal 2D image, stripe and zoom and interpolation. Did the 3D texture not implemented yet?
First of all, this is great work, amazing!
While playing around with your code I encountered an issue relating to the synthesis of interpolated samples.
First of all, s_neural_texture.py line 254: z_texture_interpolated = z_texture_interpolated[:, :-2]
does not work, since it causes a dimension mismatch later on. I don't know that it is supposed to do so I commented it out.
Next, s_neural_texture.py line 68+69
only make sense in non-interpolation mode, I therefore prepended the line if z_encoding.shape[2]==1:
to mitigate this.
Finally, transform_coord()
in neural_texture_helper.py
is also not handling interpolation mode properly. I replaced the lines 105-107 with the following:
inter = (t_coeff.shape[2] != 1)
if inter:
t_coeff = t_coeff.reshape(bs, octaves, dim, dim, h, w)
t_coeff = t_coeff.permute(0, 1, 4, 5, 2, 3)
else:
t_coeff = t_coeff.reshape(bs, octaves, dim, dim).unsqueeze(2).unsqueeze(2)
t_coeff = t_coeff.expand(bs, octaves, h, w, dim, dim)
An unrelated question: is it possible to somehow run interference on cpu? I am a tensorflow guy and not familiar with pytorch, but it seems that your custom noise sampler is not cpu capable. Is there a way to run it on the cpu?
Cheers,
tiziano
Hi,
Whenever I run the script "test_neural_texture.py", it fails on trained_models that has the term "space" in the filename (such as ../trained_models/neural_texture/version_468753_neuraltexture_rust_paint_2d_space/
). Thus, I had to remove those directories from the trained_model
directory to completely run the script. I believe these "space" models synthesize textures that encode style statistics from multiple input images.
Use pytorch 1.4.0
Load config: ../trained_models/neural_texture/version_468753_neuraltexture_rust_paint_2d_space/logs/config.txt
INFO:lightning:GPU available: True, used: True
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
checkpoint loaded ../trained_models/neural_texture/version_468753_neuraltexture_rust_paint_2d_space/checkpoints/neural_texture_ckpt_epoch_1.ckpt
[PATH TO CONDA ENV]/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:23: UserWarning: Checkpoint directory ../trained_models/neural_texture/version_468753_neuraltexture_rust_paint_2d_space/checkpoints exists and is not empty with save_top_k != 0.All files in this directory will be deleted when a checkpoint is saved!
warnings.warn(*args, **kwargs)
Testing: 33%|████████████ | 1/3 [00:01<00:02, 1.49s/it]Traceback (most recent call last):
File "[PROJECT ROOT]/code/systems/s_neural_texture.py", line 261, in test_step
image_out_inter = self.forward(z_texture_interpolated, position, seed)
File "[PROJECT ROOT]/code/systems/s_neural_texture.py", line 66, in forward
transform_coeff, z_encoding = torch.split(weights, [self.p.texture.t, self.p.texture.e], dim=1)
File "[PATH TO CONDA ENV]/lib/python3.8/site-packages/torch/functional.py", line 77, in split
return tensor.split(split_size_or_sections, dim)
File "[PATH TO CONDA ENV]/lib/python3.8/site-packages/torch/tensor.py", line 377, in split
return super(Tensor, self).split_with_sizes(split_size, dim)
RuntimeError: start (32) + length (64) exceeds dimension size (94).
I wonder what I should do to overcome this issue.
P.S.: I am using the pre-trained models and test images provided by you. No train images are placed.
Hello, I am wondering if it is possible to train separate directions in 3D. I.e., you have a 3D texture that is different in three major axes, and you have different 2d pictures corresponding to each direction. Is there any way I can do that?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.