vb000/neuricam

Deep learning based video sensing method for low-power IoT cameras (Smart glasses, GoPro, Blink etc.).

neuricam's Issues

about dataset

hello,do you provide the dataset?

Welcome update to OpenMMLab 2.0

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

	OpenMMLab 1.0 branch	OpenMMLab 2.0 branch
MMEngine		0.x
MMCV	1.x	2.x
MMDetection	0.x 、1.x、2.x	3.x
MMAction2	0.x	1.x
MMClassification	0.x	1.x
MMSegmentation	0.x	1.x
MMDetection3D	0.x	1.x
MMEditing	0.x	1.x
MMPose	0.x	1.x
MMDeploy	0.x	1.x
MMTracking	0.x	1.x
MMOCR	0.x	1.x
MMRazor	0.x	1.x
MMSelfSup	0.x	1.x
MMRotate	0.x	1.x
MMYOLO		0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

Any hope for making a GUI or offical google collab for this code?

I have little or to understanding of how to use Python, but would greatly benefit from using this code for my work, anyway we can see an official port to google collab for this project?

Run real time on less powerful devices?

Hi,

In the paper you mentioned that the inference was made on an Nvidia RTX GPU and that the latency wasn't so great.
Do you think that there could be some tweaks that could be done in order to execute inference in real time on a low power device like a smartphone or a laptop with a small iGPU?

Using a higher input resolution than 160x120

Hi,

The code provides great results but at times it feels like the quality of the upscale and colorization of finer details is limited by the small input resolution of 160x120. Evaluation using evalute.py only seems to allow for an input resolution of 160x120 for our low-resolution greyscale video. For example, using frames with a size of 320x240 returns 'RuntimeError: stack expects a non-empty TensorList'. I also encountered at one point 'runtimeerror: Sizes of tensors must match except in dimension 1' when attempting to use a frame size above 160x120. Is there any way to use a higher resolution frames for our lr-set video set folders?

Training with Vimeo90K

Hello, I am trying to train a new model to take advantage of your great color propagation algorithm.

I have the following questions:

1. I keep getting the following assertion error when trying to train the model with Vimeo90K (the same data as you have used).
   Given groups=1, weight of size [64, 196, 3, 3], expected input[4, 136, 68, 68] to have 196 channels, but got 136 channels instead
   I am sending vimeo frames as input for the lr_set with the upscale_factor changed to 1 from params.json.

Could you please provide me with a reason for this error? Were any changes made to the code for training for the Vimeo90K dataset?

2. How did you change the directory structure for Vimeo90K? Seems like the preferred directory structure is very different to that of the dataset provided from the link you've provided in one of the closed issues.

Thank you in advance!

Running evaluation code without training

I am trying to perform evaluation task for your code without doing the trainig process for myself. I am using pre-trained model ([pretrained.pth.tar]) & spynet weights available in the Evaluation section of this GitHub page. But I am not getting results (for evaluation) as described in the paper. I have genereated LR images using cv2.resize() with cv2.INTER_CUBIC as the interpolation parameter (image alreday converted to LAB color scheme using cv2.COLOR_BGR2LAB ). Should i have to re-train the model myself or should i generate LR frames using MatlaB resize(as told in the paper) or I am doing some other mistake. Please guide.

My results example: For the 'walk' video frames of the dataset Vid4 i am getting PSNR-RGB of about 21.08.

Evalution command i used is : !python evaluate.py --lr_dir=lr-set-lab --key_dir=key-set --target_dir=hr-set --output_dir=sr-set --model_dir=experiments/bix4_keyvsrc_attn --restore_file=pretrained --file_fmt="frame%d.png"

dataset to test the model, and evaluate the model.

To run this command ,
python evaluate.py --lr_dir= --key_dir= --target_dir= --model_dir=experiments/bix4_keyvsrc_attn --restore_file=pretrained --file_fmt=<file format eg., "%08d.png">

i want these paths right -path of LR,Key and ground-truth .
can you please share the dataset link from where i download .

Some Doubts

I was looking, and I wanted to know what the "lr-set", "key-set" and "hr-set" folders would be

about training

Hi, how long does it take you to train once？Under the condition that the effect will not be greatly reduced, how much can the training epoch be reduced？

RuntimeError: stack expects a non-empty TensorList

Hi,

First of all I wanted to compliment this great work, it's very impressive. 👍

I am having a issue when trying to evaluate using the pre-trained model.

Am I missing any directories or files here that could cause this error, or does any code need to be adjusted?
Thanks.

python evaluate.py --lr_dir=/content/NeuriCam/lrvideo --key_dir=/content/NeuriCam/key --target_dir=/content/NeuriCam/hrvideo  --model_dir=/content/NeuriCam/experiments/bix4_keyvsrc_attn --restore_file=pretrained --file_fmt=%06d.png
Creating the dataset...
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
- done.
load checkpoint from local path: /content/NeuriCam/model/keyvsrc/spynet_20210409-c6c1bd09.pth
Evaluating keyvsrc
Starting evaluation
  0% 0/756 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "evaluate.py", line 182, in <module>
    args.output_dir, args.file_fmt, args.profile)
  File "evaluate.py", line 71, in evaluate
    for i, (train_batch, target, sample_ids) in enumerate(tqdm(dataloader)):
  File "/usr/local/lib/python3.7/dist-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 652, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1347, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.7/dist-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/NeuriCam/model/dataset.py", line 150, in __getitem__
    self.frame_fmt, start_num, end_num, grayscale=self.grayscale)
  File "/content/NeuriCam/model/dataset.py", line 77, in load_video
    video = torch.stack(video, dim=0) # [t, c, h, w]
RuntimeError: stack expects a non-empty TensorList