The pytti-notebook from pytti-tools

no interpolated frames folder?

I am not seeing an interpolated frames that contains results created in my input folder in the drive? Anyone know whats up with this?

Last update seems to require input_audio

I'm not using or trying to use audio, I am however trying to use a video for animation....

ConfigAttributeError                      Traceback (most recent call last)
[<ipython-input-15-8e8b96d52965>](https://localhost:8080/#) in <module>()
      6   params.seed = random.randint(-0x8000_0000_0000_0000, 0xffff_ffff_ffff_ffff)
      7 
----> 8 render_frames(params)

10 frames
[/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py](https://localhost:8080/#) in _get_node(self, key, validate_access, throw_on_missing_value, throw_on_missing_key)
    468         if value is None:
    469             if throw_on_missing_key:
--> 470                 raise ConfigKeyError(f"Missing key {key}")
    471         elif throw_on_missing_value and value._is_missing():
    472             raise MissingMandatoryValue("Missing mandatory value: $KEY")

ConfigAttributeError: Missing key input_audio
    full_key: input_audio
    object_type=dict

Fails to download AdaBins

After rendering a couple frames, it dies with:

2022-02-17 01:32:25.422 | DEBUG    | pytti.workhorse:update:503 - Time: 0.0000 seconds
2022-02-17 01:32:25.435 | DEBUG    | pytti.LossAug.DepthLoss:init_AdaBins:18 - Loading AdaBins...
2022-02-17 01:32:25.439 | DEBUG    | adabins.infer:__init__:81 - /root/.cache/adabins/AdaBins_nyu.pt
2022-02-17 01:32:25.444 | DEBUG    | adabins.model_io:dl_adabins:8 - Attempting to fetch AdaBins pretrained weights...
2022-02-17 01:32:25.447 | DEBUG    | adabins.model_io:dl_adabins:15 - using destination path: /root/.cache/adabins/
2022-02-17 01:32:25.449 | DEBUG    | adabins.model_io:dl_adabins:24 - downloading from: https://drive.google.com/uc?id=1lvyZZbC9NLcS8a__YPcUP7rDiIpbRpoF

 	Cannot retrieve the public link of the file. You may need to change
	the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:

	 https://drive.google.com/uc?id=1lvyZZbC9NLcS8a__YPcUP7rDiIpbRpoF 

2022-02-17 01:32:25.859 | DEBUG    | adabins.model_io:dl_adabins:27 - gdown response: None
2022-02-17 01:32:25.860 | DEBUG    | adabins.model_io:dl_adabins:8 - Attempting to fetch AdaBins pretrained weights...
2022-02-17 01:32:25.866 | DEBUG    | adabins.model_io:dl_adabins:15 - using destination path: /root/.cache/adabins/
2022-02-17 01:32:25.869 | DEBUG    | adabins.model_io:dl_adabins:24 - downloading from: https://drive.google.com/uc?id=1zgGJrkFkJbRouqMaWArXE4WF_rhj-pxW

 	Cannot retrieve the public link of the file. You may need to change
	the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:

	 https://drive.google.com/uc?id=1zgGJrkFkJbRouqMaWArXE4WF_rhj-pxW 

2022-02-17 01:32:26.055 | DEBUG    | adabins.model_io:dl_adabins:27 - gdown response: None
Access denied with the following error:
Access denied with the following error:
Loading base model ()...Downloading: "https://github.com/rwightman/gen-efficientnet-pytorch/archive/master.zip" to /root/.cache/torch/hub/master.zip
Downloading: "https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ap-9e82fae8.pth
Done.
Removing last two layers (global_pool & classifier).
Building Encoder-Decoder model..Done.
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-8-55db5d020140>](https://localhost:8080/#) in <module>()
      4 
      5 # function wraps step 2.3 of the original p5 notebook
----> 6 _main(cfg)

13 frames
[/usr/local/lib/python3.7/dist-packages/torch/serialization.py](https://localhost:8080/#) in __init__(self, name, mode)
    209 class _open_file(_opener):
    210     def __init__(self, name, mode):
--> 211         super(_open_file, self).__init__(open(name, mode))
    212 
    213     def __exit__(self, *args):

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/adabins/AdaBins_nyu.pt'

Defaults should activate fewer features

Add patch for missing settings keys to pyttipanna and Run It! cells

Add cell for consuming parameter dict from old pytti colab

something like this:

# duct tape to permit posting raw JSON into the cell and interpreting it as python
false=False
true=True
null=None

colab_dict = {}

# User populates this value
MY_SETTINGS ={}

colab_dict.update(MY_SETTINGS)
cfg = OmegaConf.create(colab_dict)

Encounter NameError: name 'embedder' is not defined using pyttitools-PYTTI.ipynb

First, thanks for updating the tools.

When I use the pyttitools-PYTTI.ipynb I encounter following error message:

[/usr/local/lib/python3.7/dist-packages/pytti/LossAug/LossOrchestratorClass.py](https://localhost:8080/#) in configure_init_image(init_image_pil, restore, img, params, loss_augs)
     70         if params.semantic_init_weight not in ["", "0"]:
     71             semantic_init_prompt = parse_prompt(
---> 72                 embedder,
     73                 f"init image [{params.init_image}]:{params.semantic_init_weight}",
     74                 init_image_pil,

NameError: name 'embedder' is not defined

Here is the setting dict and hope it help you to define where I did wrong:

SETTINGS:
{'defaults': ['_self_', {'conf': '???'}], 'scenes': '[mj0.jpg] || greatest of all time | dynasty | I can fly | unstoppable | enemies| basketball | hoop score | fadeaway shots | dunk | teammates', 'scene_prefix': 'satellite image:-1:-.95 | text:-1:-.95 | anime:-1:-.95 | watermark:-1:-.95 | backyard telescope:-1:-.95 | map:-1:-.95', 'scene_suffix': 'surrealism | abstract', 'direct_image_prompts': 'mj0.jpg', 'init_image': 'mj0.jpg', 'direct_init_weight': '3', 'semantic_init_weight': '3', 'image_model': 'VQGAN', 'vqgan_model': 'imagenet', 'animation_mode': '3D', 'width': 1280, 'height': 720, 'steps_per_scene': 20000, 'steps_per_frame': 50, 'interpolation_steps': 0, 'learning_rate': None, 'reset_lr_each_frame': True, 'seed': '${now:%f}', 'cutouts': 40, 'cut_pow': 2, 'cutout_border': 0.25, 'border_mode': 'clamp', 'field_of_view': 60, 'near_plane': 10000, 'far_plane': 10000, 'translate_x': '-1700*sin(radians(1.5))', 'translate_y': '0', 'translate_z_3d': '(50+10*t)*sin(t/10*pi)**2', 'rotate_3d': '[cos(radians(1.5)), 0, -sin(radians(1.5))/sqrt(2), sin(radians(1.5))/sqrt(2)]', 'rotate_2d': '5', 'zoom_x_2d': '0', 'zoom_y_2d': '0', 'sampling_mode': 'bicubic', 'infill_mode': 'wrap', 'pre_animation_steps': 100, 'lock_camera': True, 'pixel_size': 1, 'smoothing_weight': 0.02, 'random_initial_palette': False, 'palette_size': 6, 'palettes': 9, 'gamma': 1, 'hdr_weight': 0.01, 'palette_normalization_weight': 0.2, 'show_palette': False, 'target_palette': '', 'lock_palette': False, 'frames_per_second': 12, 'direct_stabilization_weight': '', 'semantic_stabilization_weight': '', 'depth_stabilization_weight': '', 'edge_stabilization_weight': '', 'flow_stabilization_weight': '', 'video_path': '', 'frame_stride': 1, 'reencode_each_frame': False, 'flow_long_term_samples': 1, 'ViTB32': True, 'ViTB16': False, 'ViTL14': False, 'RN50': False, 'RN101': False, 'RN50x4': False, 'RN50x16': False, 'RN50x64': False, 'file_namespace': 'default', 'allow_overwrite': False, 'display_every': 50, 'clear_every': 0, 'display_scale': 1, 'save_every': 50, 'backups': 5, 'show_graphs': False, 'approximate_vram_usage': False, 'models_parent_dir': '.', 'gradient_accumulation_steps': 1, 'use_tensorboard': False, 'input_audio': '', 'ViTL14_336px': False, 'base_name': 'default'}

adabins symlink fails for mounted google drive

trivial fix, just change ln to cp in the shell script cell. should auto-detect users who have google drive mounted and give them the option to specify the location of the adabins file or whatever.

Last frame doesn't save/output (colab notebook)

is this limited to the colab? if this was an issue with the local notebook too I'm surprised It got past me for so long...

steps_per_scene is a non-obvious cap on video length

I was doing a long video render (6 minutes) and found my runs were arbitrarily stopping at around 2000 frames. After digging through the code it looks like they stopped because the steps_per_scene value was arbitrarily set to 60000 or so and I was doing 30 steps per frame.

This steps_per_scene value seems kind of non obvious to me. Consider dropping this limit at least for Video Source?

Replace "batch settings" with hydra multirun demonstration

`!pip` -> `%pip`

hasn't caused issues that I'm aware of, but would be good to help people who are using the colab notebook offline for setup to make sure everything gets installed into the correct env

Use CLI to emit runs

from pytti.pytti_cli_w_clearml import _main
from omegaconf import OmegaConf

cfg = OmegaConf.create(dict(params))
_main(cfg) # yeah baby

Import error in video display cell of pyttitools-PYTTI_local.ipynb

Running the cell to embed a rendered video in pyttitools-PYTTI_local.ipynb yields an import error for me

# Optionally display your video in the notebook

from Ipython.display import Video

output_video_filename = "default.mp4" # you'll probably need to change this after the first run

Video(output_video_filename)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/home/naka/code/side/pytti-notebook/pyttitools-PYTTI_local.ipynb Cell [1](vscode-notebook-cell:/home/naka/code/side/pytti-notebook/pyttitools-PYTTI_local.ipynb#ch0000015?line=0)6' in <cell line: 3>()
      1[ # Optionally display your video in the notebook
----> ]()[3](vscode-notebook-cell:/home/naka/code/side/pytti-notebook/pyttitools-PYTTI_local.ipynb#ch0000015?line=2)[ from Ipython.display import Video
      ]()[5](vscode-notebook-cell:/home/naka/code/side/pytti-notebook/pyttitools-PYTTI_local.ipynb#ch0000015?line=4)[ output_video_filename = "default.mp4" # you'll probably need to change this after the first run
      ]()[7](vscode-notebook-cell:/home/naka/code/side/pytti-notebook/pyttitools-PYTTI_local.ipynb#ch0000015?line=6)[ Video(output_video_filename)

ModuleNotFoundError: No module named 'Ipython']()

Happy to take a stab at it

Notebook not working

Hi,

When I run the notebook often it fails to detect any images, and when it does it runs and displays 'interpolation complete' without saving any images or displays any of the logging from the output function.

m

deprecate bunch from the notebook

Recursive Interpolation Passes Setting Has No Effect

I'm running Pytti Tools FILM on CoLab and the referenced setting has no effect on number of frames generated. Leaving the integer at "1" results in frame output on the order of 10X original frame count.

Resume checkbox not working on Google colab

Hi, I am trying to animate a video using this notebook on google Colab.
After running for a while the run usually stops at some point due to the runtime timing out or other reasons.
When I run the notebook again with the resume checkbox checked a new Video is generated, rather than the notebook continuing from where it left off.

Trying to run PyTTI-Tools: FiLM.ipynb on Google Colab, but there is an error

2022-08-18 12:58:05.658194: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.659580: W ./tensorflow/stream_executor/stream.h:2119] attempting to perform DNN operation using StreamExecutor without DNN support
2022-08-18 12:58:05.691343: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.696118: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNIMPLEMENTED: DNN library is not found.
2022-08-18 12:58:05.697546: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.698416: W ./tensorflow/stream_executor/stream.h:2119] attempting to perform DNN operation using StreamExecutor without DNN support
2022-08-18 12:58:05.699909: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.700836: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNIMPLEMENTED: DNN library is not found.
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 572, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 162, in process
input_frames, _TIMES_TO_INTERPOLATE.value, self.interpolator))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 120, in interpolate_recursively_from_files
times_to_interpolate, interpolator)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 88, in _recursive_generator
np.expand_dims(frame1, axis=0), np.expand_dims(frame2, axis=0), time)[0]
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 104, in interpolate
result = self._model(inputs, training=False)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/load.py", line 686, in _call_attribute
return instance.call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 55, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'AvgPool' defined at (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 457, in get
self.data_channel_factory)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 865, in init
op.setup()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 141, in setup
_MODEL_PATH.value)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 82, in init
self._model = tf.compat.v2.saved_model.load(model_path)
Node: 'AvgPool'
dnn PoolForward launch failed
[[{{node AvgPool}}]] [Op:__inference_restored_function_body_13284]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 996, in process_bundle
element.data)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 221, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 352, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 354, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1300, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1396, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1300, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1396, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1316, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 572, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 162, in process
input_frames, _TIMES_TO_INTERPOLATE.value, self.interpolator))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 120, in interpolate_recursively_from_files
times_to_interpolate, interpolator)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 88, in _recursive_generator
np.expand_dims(frame1, axis=0), np.expand_dims(frame2, axis=0), time)[0]
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 104, in interpolate
result = self._model(inputs, training=False)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/load.py", line 686, in _call_attribute
return instance.call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 55, in quick_execute
inputs, attrs, num_outputs)
RuntimeError: tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'AvgPool' defined at (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 457, in get
self.data_channel_factory)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 865, in init
op.setup()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 141, in setup
_MODEL_PATH.value)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 82, in init
self._model = tf.compat.v2.saved_model.load(model_path)
Node: 'AvgPool'
dnn PoolForward launch failed
[[{{node AvgPool}}]] [Op:__inference_restored_function_body_13284] [while running 'Process directories']
2022-08-18 12:58:07.798 | INFO | main::14 - Interpolation comlpete.

No frames_dir directory

When running PyTTI-Tools: FiLM.ipynb from https://github.com/google-research/frame-interpolation, can't find "frames_dir" directory.

Configure repo to facilitate notebook diffing

Candidate options:

Error when generate video

i got below error at final step to generate video

install libraries from pytti-core/requirements.txt rather than specifying in local cell

In-notebook sketcher for prompting init images

Good candidate here: https://dash.plotly.com/canvas

Where'd the logging messages go?

misc advanced techniques

can even be just snippets

warped init image
np.where as a binary toggle

document parameter options in default config

remove config folder

let the warmup script be responsible for building the local config.

Link "resources" appendix to wiki

https://github.com/pytti-tools/pytti-core/wiki/Resources

Minor bugs in main setup cell

foce_install =False should be default, not True
pytti-tools/pytti-core clone ops should clone into pytti folder

Separate notebook for dummy-proof local support

Need to add "restore" to notebook params

For longer video work, the "restore" feature is often necessary. However, it appears the toggle has been pulled out of the notebook params so there's no way to restore a run without going into workhorse.py and setting restore to True.

Push application logic to pytti-core

Link git submodules

pytti-core setup currently recommends installing everything in pytti-notebook as root directory. We can do this for the users.

Add git submodules: AdaBins, GMA, CLIP, taming-transformers, pytti-core
Add folders with .gitkeep: ./pretrained, ./images_out, ./videos

Desired structure:

 ├── pytti-notebook
 │   ├── AdaBins
 │   ├── CLIP
 │   ├── GMA
 │   ├── images_out
 │   ├── pretrained
 │   ├── pytti-core
 │   ├── taming-transformers
 │   └── videos

Missing images

Hi
It runs and reads "interpolation complete" but there are no interpolated images on
my drive/interpolation/frame-interpolation/photos
Where are they?

Thanks so much
Ron

Install dependencies rather than just cloning them

Will need to update pytti-core to be pip installable
Pretty sure CLIP is already installable, but likely not the case for other models
- Where this is an issue, we can fork those codebases to pytti-tools and make installable versions that suit our needs
several directory change operations will need to be addressed. If not in the notebook, then they're in pytti-core

add tensorboard

Add a setup script to support "I don't even have python installed" users

download and install miniconda
create and activate new environment
install conda dependencies
- pytorch
- opencv
- pillow
- ffmpeg
install pip dependencies
install jupyter
shortcut script/command for activating environment and starting jupyter
emit necessary git clone and pip install operations
delete existing folders (with prompt) where they need to be overwritten
end state should result in notebook's setup cell skipping because installed dependencies detected

use pytti-docker as a guide, refactor dockerfile setup into shell scripts for separate windows and unix support. Windows support is priority, linux users probably less likely to experiences challenges with setup (and dockerfile steps can be copied basically at face)

open notebook to a rotoscoped demo

serves multiple purposes:

Demonstrates some of the flashier things these tools are capable of, (i.e. hook more "just curious" users)
Forces the user to learn a little about the basic settings to exit the "Tutorial" (specifically changing animation mode from video_source or providing their own video file/url)

if there're other settings I can "game tutorial"-ify, that'd be great.

clear out stuff that's redundant to documentation in pytti-book (e.g. settings)

don't want to have to maintain instruction sets in multiple places, just direct users to the appropriate page in the pytti-book

Auto install can lead to PIL version conflicts

pip requirements installs PIL, conflicts with conda installed PIL. need to remove from pip requirements, rely on conda install

circular import in notebook that didn't come up in pytest end-to-end runs

ImportError Traceback (most recent call last)
/tmp/ipykernel_2058627/2608867653.py in
4 from loguru import logger
5 from omegaconf import OmegaConf
----> 6 from pytti.workhorse import _main as render_frames
7
8 CONFIG_BASE_PATH = "config"

~/miniconda3/envs/sandbox/lib/python3.9/site-packages/pytti/workhorse.py in
29
30 logger.info("Loading pytti...")
---> 31 from pytti.Notebook import (
32 change_tqdm_color, # why though?
33 get_last_file,

~/miniconda3/envs/sandbox/lib/python3.9/site-packages/pytti/Notebook.py in
16 from PIL import Image
17
---> 18 from pytti.LossAug import Loss as LossType
19
20 # https://stackoverflow.com/questions/15411967/how-can-i-check-if-code-is-executed-in-the-ipython-notebook

~/miniconda3/envs/sandbox/lib/python3.9/site-packages/pytti/LossAug/init.py in
44
45 from pytti.LossAug.TVLoss import TVLoss
---> 46 from pytti.LossAug.MSELoss import MSELoss
47 from pytti.LossAug.OpticalFlowLoss import OpticalFlowLoss, TargetFlowLoss
48 from pytti.LossAug.DepthLoss import DepthLoss

~/miniconda3/envs/sandbox/lib/python3.9/site-packages/pytti/LossAug/MSELoss.py in
4 from torch.nn import functional as F
5 from pytti.LossAug import Loss
----> 6 from pytti.Notebook import Rotoscoper
7 from pytti import DEVICE, fetch, parse, vram_usage_mode
8 import torch

ImportError: cannot import name 'Rotoscoper' from partially initialized module 'pytti.Notebook' (most likely due to a circular import) (/home/dmarx/miniconda3/envs/sandbox/lib/python3.9/site-packages/pytti/Notebook.py)

infer image size/ratio from init image if provided

this is complicated by the current configuration scheme which requires height and width be specified in the config file. Does hydra support optional config options? I guess maybe that's just defaulting to Null?

PyTTI-Tools: FiLM.ipynb Error

Mounted at /content/drive/
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Transport endpoint is not connected
ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.

-----[PYTTI-TOOLS]-------

If you received a scary OSError and your drive was already mounted, ignore it.

-----[PYTTI-TOOLS]-------

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 36, in
get_ipython().magic('cd {gdrive_fpath}')
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2160, in magic
return self.run_line_magic(magic_name, magic_arg_s)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2081, in run_line_magic
result = fn(*args,**kwargs)
File "", line 2, in cd
File "/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py", line 188, in
call = lambda f, *a, **k: f(*a, **k)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/magics/osm.py", line 288, in cd
oldcwd = py3compat.getcwd()
OSError: [Errno 107] Transport endpoint is not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 1823, in showtraceback
stb = value.render_traceback()
AttributeError: 'OSError' object has no attribute 'render_traceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 1132, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 313, in wrapped
return f(*args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 358, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/usr/lib/python3.7/inspect.py", line 1502, in getinnerframes
frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
File "/usr/lib/python3.7/inspect.py", line 1460, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File "/usr/lib/python3.7/inspect.py", line 696, in getsourcefile
if getattr(getmodule(object, filename), 'loader**', None) is not None:
File "/usr/lib/python3.7/inspect.py", line 725, in getmodule
file = getabsfile(object, _filename)
File "/usr/lib/python3.7/inspect.py", line 709, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File "/usr/lib/python3.7/posixpath.py", line 383, in abspath
cwd = os.getcwd()
OSError: [Errno 107] Transport endpoint is not connected

Migrate params to OmegaConf.DictConfig

direct_image_prompts

From what I can derive from the docs, direct image prompts should behave like a "target_image". Or maybe I don't understand, but it certainly does not behave like that...
given:

init image: "car.jpg"
scenes: 'a green bike || a long bus'
direct_image_prompts: "bike.jpg || bus.jpg"

I expect a video starting from my car.jpg morphing into bike.jpg using the 'a green bike prompt' into the bus.jpg via the 'a long bus' prompt...

am I misunderstanding this?

/usr/local/lib/python3.7/dist-packages/pytti/workhorse.py:186: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="config", config_name="default")
100%|███████████████████████████████████████| 338M/338M [00:05<00:00, 65.2MiB/s]

Working with z of shape (1, 256, 16, 16) = 65536 dimensions.

/usr/local/lib/python3.7/dist-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
  f"The parameter '{pretrained_param}' is deprecated since 0.13 and will be removed in 0.15, "
/usr/local/lib/python3.7/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /root/.cache/torch/hub/checkpoints/vgg16-397923af.pth

100%
528M/528M [00:02<00:00, 229MB/s]

loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
VQLPIPSWithDiscriminator running with hinge loss.

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

[<ipython-input-18-8e8b96d52965>](https://localhost:8080/#) in <module>
      6   params.seed = random.randint(-0x8000_0000_0000_0000, 0xffff_ffff_ffff_ffff)
      7 
----> 8 render_frames(params)

7 frames

[/usr/local/lib/python3.7/dist-packages/torch/serialization.py](https://localhost:8080/#) in __init__(self, name_or_buffer)
    240 class _open_zipfile_reader(_opener):
    241     def __init__(self, name_or_buffer) -> None:
--> 242         super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
    243 
    244 

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

revert cell order to mount gdrive first and give option for users to specify custom mount point

need to make sure mount point is consistent in cd call of setup cell as well

pytti-tools / pytti-notebook Goto Github PK

pytti-notebook's People

Contributors

Stargazers

Watchers

Forkers

pytti-notebook's Issues

Recommend Projects

Recommend Topics

Recommend Org