orange-opensource/aivc

AIVC is a fully-learned video codec. It is able to code video sequences at different rates and it features tunable coding configurations.

aivc's Issues

training code

Hi Dr. Theo,

Is it possible to share also the training code in the near future?
I really want to learn your training code.

Best regards

Python error using docker

First of all I want to congratulate for your outstanding results on IEEE CLIC Challenge! Impressive work using machine learning with video compression.

I tried to run the code using docker as per instructions on README but stumbled upon this error using the sanity script:

root@eb31c26fd94a:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
Traceback (most recent call last):
File "encode.py", line 103, in
model = load_model(prefix='0_', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
Traceback (most recent call last):
File "decode.py", line 56, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting evaluation
/home/ubuntu/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in double_scalars
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/home/ubuntu/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in double_scalars
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined
root@eb31c26fd94a:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
Traceback (most recent call last):
File "encode.py", line 103, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
Traceback (most recent call last):
File "decode.py", line 56, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting evaluation
/home/ubuntu/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in double_scalars
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/home/ubuntu/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in double_scalars
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined

With the latest update bitstream.bin files are now huge in size

Trying to compress an 8.4GB HD 1080 uncompressed yuv file resulted in an 2.9GB bitstream.bin file with model number 1 which is the highest rate, before the update, rates were much lower.

LFS data quota

Hi, I have been trying to clone a repo with all large files there, but got an error from git lfs
This repository is over its data quota.
Account responsible for LFS bandwidth should purchase more data packs to restore access.

Is there are any chances how to download all necessary files to get it running?

Training code

Hi Dr. Theo,

Is it possible to share also the training code in the near future?

Best regards

请问代码什么时候开源

请问代码什么时候开源

compressed.yuv can not open

I have configured the code according to your instructions, and the code has finished running, but the final decoded file cannot be opened and played by YUV software. The original video can be played. Where is the problem
$ ./sanity_script.sh
********************************************************************************
Starting encoding
Using /home/ganzhi/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/ganzhi/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/ganzhi/.local/lib/python3.8/site-packages/torch/include -isystem /home/ganzhi/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ganzhi/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ganzhi/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/ganzhi/.local/lib/python3.8/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o
[2/2] c++ torchac_backend.o -shared -L/home/ganzhi/.local/lib/python3.8/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o torchac_backend.so
Loading extension module torchac_backend...
[STATE] Start processing: yuv -> png
Input file: ../raw_videos/BlowingBubbles_416x240_50_420.yuv
Output directory: ../raw_videos/BlowingBubbles_416x240_50_420/
Number of frames in the video: 101
Frame: 101 / 101
[STATE] End processing: yuv -> png
[STATE] End processing
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
model path == ./0_model.pt
rm: 无法删除 '../tmp/3697412327659087746_0/tmp_bitstream_working_dir/': 没有那个文件或目录
rm: 无法删除 '../tmp/3697412327659087746_0/tmp_bitstream_working_dir': 没有那个文件或目录
[INFO] | Start encoding | |
[INFO] | infer_one_sequence | sequence name | BlowingBubbles_416x240_50_420
[INFO] | infer_one_sequence | GOP_struct_name | 2_GOP_16
[INFO] | infer_one_sequence | idx_rate | 0
[DEBUG] | infer_one_sequence | nb_GOP | 4
[INFO] | Encoding done | |
[INFO] | Bitstream path | | ../bitstream.bin
[INFO] | Frame resolution | [H x W] | 240 x 416
[INFO] | First coded frame | [frame] | 0
[INFO] | Last coded frame | [frame] | 100
[INFO] | Number of frames to code | [frame] | 101
[INFO] | Number of frames coded | [frame] | 132
[INFO] | Intra-period | [frame] | 33
[RESULT] | Number of frames | [frame] | 101
[RESULT] | Encoding/decoding time | [s] | 20.4
[RESULT] | Encoding/decoding FPS | [frame/s] | 5.0
[RESULT] | Estimated PSNR | [dB] | 26.7213
[RESULT] | Estimated MS-SSIM | [dB] | 11.8853
[RESULT] | Estimated rate | [byte] | 23382.6
[RESULT] | Real rate | [byte] | 28448
[RESULT] | Estimated rate overhead | [%] | 21.66
********************************************************************************
Starting decoding
Using /home/ganzhi/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/ganzhi/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
model path == ./0_model.pt
[INFO] | | | Found CUDA device
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
[INFO] | Bitstream path | | ../bitstream.bin
[INFO] | Bitstream processing dir. | | ../tmp/3697412327659087746_0/tmp_out_bitstream/
[INFO] | Decoded frames directory | | ../compressed/
[INFO] | Number of GOPs in bitstream | | 4
[INFO] | Index first frame | | 0
[INFO] | Index last frame | | 100
[INFO] | Start decoding | |
[INFO] | Decoding done | |
[RESULT] | Number of frames | [frame] | 101
[RESULT] | Decoding time | [s] | 15.9
[RESULT] | Decoding FPS | [frame/s] | 6.4
[STATE] Starting png -> yuv conversion
[INFO] | Final decoded video | | ../compressed.yuv
********************************************************************************
Starting evaluation
PSNR [dB]: 26.72129
MS-SSIM : 0.93531
MS-SSIM [dB]: 11.89145
Size [bytes]: 28448

model checkpoints

Hello,

I tried to clone the repo, but I got the following error.

Downloading models/ms_ssim-2021cc-1/0_model.pt (164 MB)
Error downloading object: models/ms_ssim-2021cc-1/0_model.pt (43f4625): Smudge error: Error downloading models/ms_ssim-2021cc-1/0_model.pt (43f4625ad0294be5ac745e2410f2c0c4e6e2f08dafcf94999564e5cf306c93ff): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

I think I am not able to fetch the model checkpoints. Is it possible to share the checkpoints another way?

=========================below is the error================================
********************************************************************************
Starting encoding
Using /homes/husseini/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
Emitting ninja build file /homes/husseini/.cache/torch_extensions/py310_cu116/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
[STATE] Start processing: yuv -> png
Input file: ../raw_videos/BlowingBubbles_416x240_50_420.yuv
Output directory: ../raw_videos/BlowingBubbles_416x240_50_420/
Number of frames in the video: 101
Frame: 101 / 101
[STATE] End processing: yuv -> png
[STATE] End processing
[INFO] | Seed | | 666
Traceback (most recent call last):
File "/medias/Sahar/AIVC/src/encode.py", line 103, in
model = load_model(prefix='0_', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/medias/Sahar/AIVC/src/model_mngt/model_management.py", line 347, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "//medias/Sahar/env/image_compression/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "//medias/Sahar/env/image_compression/lib/python3.10/site-packages/torch/serialization.py", line 1002, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting decoding
Using /homes/husseini/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
Emitting ninja build file /homes/husseini/.cache/torch_extensions/py310_cu116/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
Traceback (most recent call last):
File "/medias/Sahar/AIVC/src/decode.py", line 56, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/medias/Sahar/AIVC/src/model_mngt/model_management.py", line 347, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "//medias/Sahar/env/image_compression/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "//medias/Sahar/env/image_compression/lib/python3.10/site-packages/torch/serialization.py", line 1002, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting evaluation
/medias/Sahar/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in scalar divide
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/medias/Sahar/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in scalar divide
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "/medias/Sahar/AIVC/src/evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined

git LFS bandwidth issue

Looks like data limits have been hit for this repository. When attempting to clone using git-lfs, I am getting the following error:

batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
error: failed to fetch some objects from 'https://github.com/Orange-OpenSource/AIVC.git/info/lfs'