Video can't generate problems

[kaiyangzhou]

Video can't generate problems

[SinDongHwan]

Did you solve the problem?
I solved!!
I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

• you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

• i executed 'videos2frames.sh' in 'SumMe/videos' directory.
  ( you should make each directories about .mp4 as 'SumMe/videos/frames/
  For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

• in my case, i made json file.
  ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }.
  you can see the information about mapping following code.

import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))

if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

[ShaoDu]

Yes, I do. I made the format of video frame according to your format requirements. So do you mean save all video frames as.h5 instead of saving them in a folder and reading them? If that's the case, I'll try again.Because I tried in the folder to save.

…

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"<[email protected]>; 发送时间: 2019年1月29日(星期二) 下午5:38 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"<[email protected]>; 抄送: "邵 都"<[email protected]>; "Author"<[email protected]>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15) Did you see this line in summary2video.py? we have to inform filename "000001.jpg, 000002.jpg, ..." and i found about number of video frames. import h5py h5_file_name = "datasets/eccv16_dataset_summe_google_pool5.h5" f = h5py.File(h5_file_name,'r') print(f['video_1']['n_frames'].value) i can see "4494". we can see more information about each h5 files on datasets/readme.txt . i will try to find fps to convert frames from video correctly. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ShaoDu]

Thank you very much for your help! Because I am busy with something else now, I will try according to your steps after I finish my work.

…

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"<[email protected]>; 发送时间: 2019年1月30日(星期三) 上午9:58 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"<[email protected]>; 抄送: "邵 都"<[email protected]>; "Author"<[email protected]>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15) Did you solve the problem? I solved!! I will explain from 'convert frames' to 'summary2video' step by step. step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory. you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to 'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg' ( if extract Summe dataset, there is 'SumMe' directory. ) step 2. convert summe dataset to frames. i executed 'videos2frames.sh' in 'SumMe/videos' directory. ( you should make each directories about .mp4 as 'SumMe/videos/frames/ For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' ) step 3. have to map h5.file key and frame dirctory name in my case, i made json file. ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }. you can see the information about mapping following code. import h5py import sys h5_file_name = (sys.argv[1]) f = h5py.File(h5_file_name,'r') for key in f.keys(): print("{} : {}".format(key, f[key]['video_name'].value)) if you edit properly "summary2video.py", script is performed. if you need to my help, ask me. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Swati640]

you can generate the frames, the way its feasible for you. I used ffmpeg and "https://github.com/SinDongHwan/pytorch-vsumm-reinforce/blob/master/utils/generate_dataset.py" , for generation of frames and feature extraction on my own dataset. I hope i understood your question properly.

[PedroJSousa]

I went through this whole comment section, and this is super confusing. This is how I managed to do it, I hope it saved some time.

Note: Make sure you have the correct dataset downloaded, because it is not on github, you can download it here
Note2: In that dataset the videos2frames.sh file is actually named videos2frames.sh~
You have to change the name of the file by running in the terminal:
mv videos2frames.sh~ videos2frames.sh

Step 1: Create a folder called "frames" inside the SumMe/videos folder
Step 2:

• Install ffmpeg
• Edit 2 lines in the videos2frames.sh file: Add a new Line before line 10: mkdir frames/"$name". This will create inside the frames folder a folder with the name of each file
• Edit line 11 to this: ffmpeg -i "$f" -f image2 -vf fps=30 frames/"$name"/%6d.jpg. This will generate the frames and save them into correspondent folder. FPS needs to be set to 30 because the results.h5 file is also at 30 fps.

Step3: Run summary2video.py

• Check which videos are your test videos. You can see it when you output the results of your training. In my case image below. Note that the videos are numbered. This means e.g. video 25 is the last video in alphabetical order from the dataset: Valparaiso_Downhill
  
• Run the function with the inputs:
  -p path of the results file ;
  -d path of where the frames are stored;
  -i is the index of your training videos matrix. So in my case 0 = video 12, 1 = video 25, 2 = video 4 etc
  Choose a video from the 5 test videos and generate the highlight. I chose video 25 and ran

summary2video.py -p log/summe-split0/result.h5 -d datasets/videos/frames/Valparaiso_Downhill -i 1 --fps 30 --save-dir log --save-name summary.mp4

[SinDongHwan]

on README,
Visualize summary

You can use summary2video.py to transform the binary machine_summary to real summary video.

You need to have a directory containing video frames.

The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a .mp4 video.

maybe, it seems that there aren't frames on "video_frames" directory. but i don't know which frames yet.

Did you solve this problem?

[ShaoDu]

Thank you very much for your reply! I haven't solved this problem for the moment, I just tried to split the video to some frames and save them in a folder, but I don't know how the summary2video-py uses these frames to generate the MP4 video summary, and the result is always that the corresponding frame is not found.

…

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"<[email protected]>; 发送时间: 2019年1月29日(星期二) 下午4:39 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"<[email protected]>; 抄送: "邵 都"<[email protected]>; "Author"<[email protected]>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15) on README, Visualize summary You can use summary2video.py to transform the binary machine_summary to real summary video. You need to have a directory containing video frames. The code will automatically write summary frames to a video where the frame rate can be controlled. Use the following command to generate a .mp4 video. maybe, it seems that there aren't frames on "video_frames" directory. but i don't know which frames yet. Did you solve this problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[SinDongHwan]

Did you see this line in summary2video.py?

we have to formate filename as "000001.jpg, 000002.jpg, ..."
and i found about number of video frames.

import h5py
h5_file_name = "datasets/eccv16_dataset_summe_google_pool5.h5"
f = h5py.File(h5_file_name,'r')
print(f['video_1']['n_frames'].value)

i can see "4494".

we can see more information about each h5 files on datasets/readme.txt .

i will try to find fps to convert frames from video correctly.

[SinDongHwan]

No, we should save frames in folder.
I told you that read dataset/readme.txt to convert frames from video correctly.
Parameter "summary" of frm2video function consists of [0,1,0,1,0,0,0,...]. this length of list( [0,1,0,1,0,0,0,...] ) is maybe number of frames.

I'm not good at English, so it could be difficult for you to understand about my write.

[ShaoDu]

Thank you for your reply! I'll have a good look. I'll try again

…

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"<[email protected]>; 发送时间: 2019年1月29日(星期二) 晚上6:10 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"<[email protected]>; 抄送: "邵 都"<[email protected]>; "Author"<[email protected]>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15) No, we should save frames in folder. I told you that read dataset/readme.txt to convert frames from video correctly. Parameter "summary" of frm2video function consists of [0,1,0,1,0,0,0,...]. this length of list( [0,1,0,1,0,0,0,...] ) is maybe number of frames. I'm not good at English, so it could be difficult for you to understand about my write. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[SinDongHwan]

if you have dataset video, could you share that? i don't have videos.

[ShaoDu]

All my video data is MP4 video segmented by ffmpeg, but now there is no video data, because the previous deleted data, now there is no operation

…

------------------ 原始邮件 ------------------ 发件人: "Shin Donghwan"<[email protected]>; 发送时间: 2019年1月29日(星期二) 晚上6:17 收件人: "KaiyangZhou/pytorch-vsumm-reinforce"<[email protected]>; 抄送: "邵 都"<[email protected]>; "Author"<[email protected]>; 主题: Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generateproblems (#15) if you have dataset video, could you share that? i don't have videos. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[SinDongHwan]

okay. i will try to do summary2video, too. if you can convert, please reply.
if i will be converted, i reply, too.

[Swati640]

Did you solve the problem?
I solved!!
I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

* you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

* i executed 'videos2frames.sh' in 'SumMe/videos' directory.
  ( you should make each directories about _.mp4 as 'SumMe/videos/frames/_
  For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

* in my case, i made json file.
  ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }.
  you can see the information about mapping following code.

import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))

if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

Hello
i tried to follow the procedure you told above and made the required changes in shell file as well. But, while running a shell file I am getting "Conversion failed error". This is the error "[image2 @ 0x5654ab990a80] Could not open file : SumMe/videos/frames/Valparaiso_Downhill/%6d.jpg
av_interleaved_write_frame(): Input/output error"
frame= 1 fps=0.0 q=8.9 Lsize=N/A time=00:00:00.03 bitrate=N/A speed=1.58x
video:79kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

Please help!! I am stuck with this conversion process.

[SinDongHwan]

@Swati640
Hi, I think can't open file(image).
Did you try to edit shell script?

* you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )

If you are okay, i can see your shell on Teamviewer?

[Swati640]

yes, I edited the shell file completely as you told above, but it is not working.

[SinDongHwan]

@Swati640
hmm...
If you are okay, could i see your situation on Teamviewer?

[Swati640]

Hey.. Thanks for the help. It worked for me. Had to do some changes in the shell file :)

[Harryjun]

Hi, @Swati640
FFMPEG generate frames slowly,Can I take video source as input directly,and generate video summary

[Harryjun]

Hi, @SinDongHwan
FFMPEG generate frames slowly,Can I take video source as input directly,and generate video summary

[SinDongHwan]

@Harryjun
Thank you for kindly answer, instead of me^^

[ghagareomkar]

@SinDongHwan I created a folder named video_frames which contains folder :Air_Force_One,Base_jumping.... and others containing video frames. But still i am getting the same error.Anyone who can help me out. ThankYou

[SinDongHwan]

@ghagareomkar
HI, Could you tell me a error message?

[ghagareomkar]

@SinDongHwan Thank You for your reply
python summary2video.py -p log/summe-split0/result.h5 -d video_frames/ -i 0 --fps 30 --save-dir log --save-name summary.mp4
this occured：
OpenCV: FFMPEG: tag 0x5634504d/'MP4V' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v'
Traceback (most recent call last):
File "summary2video.py", line 43, in
frm2video(args.frm_dir, summary, vid_writer)
File "summary2video.py", line 27, in frm2video
frm = cv2.resize(frm, (args.width, args.height))
cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

This above error i was getting when created a folder named video_frames which contains folder :Air_Force_One,Base_jumping.... and others containing video frames.
But when i put just the frames of only one video in video_frames folder .got no error, working properly got summary for the corresponding video .Is my approach correct? ThankYou

[SinDongHwan]

@ghagareomkar
please check data of frm before cv2.resize()
print(frm)

[ghagareomkar]

@SinDongHwan ThankYou ,will try this

[sky-16]

I went through this whole comment section, and this is super confusing. This is how I managed to do it, I hope it saved some time.

Note: Make sure you have the correct dataset downloaded, because it is not on github, you can download it here
Note2: In that dataset the videos2frames.sh file is actually named videos2frames.sh~
You have to change the name of the file by running in the terminal:
mv videos2frames.sh~ videos2frames.sh

Step 1: Create a folder called "frames" inside the SumMe/videos folder
Step 2:

• Install ffmpeg
• Edit 2 lines in the videos2frames.sh file: Add a new Line before line 10: mkdir frames/"$name". This will create inside the frames folder a folder with the name of each file
• Edit line 11 to this: ffmpeg -i "$f" -f image2 -vf fps=30 frames/"$name"/%6d.jpg. This will generate the frames and save them into correspondent folder. FPS needs to be set to 30 because the results.h5 file is also at 30 fps.

Step3: Run summary2video.py

• Check which videos are your test videos. You can see it when you output the results of your training. In my case image below. Note that the videos are numbered. This means e.g. video 25 is the last video in alphabetical order from the dataset: Valparaiso_Downhill
  
• Run the function with the inputs:
  -p path of the results file ;
  -d path of where the frames are stored;
  -i is the index of your training videos matrix. So in my case 0 = video 12, 1 = video 25, 2 = video 4 etc
  Choose a video from the 5 test videos and generate the highlight. I chose video 25 and ran

summary2video.py -p log/summe-split0/result.h5 -d datasets/videos/frames/Valparaiso_Downhill -i 1 --fps 30 --save-dir log --save-name summary.mp4

I was currently using a dataset of 165 MB mentioned in #40. I was getting error in that dataset during generating summary2video.py and I found out I need to use original Summe dataset of 2.2 GB. Can anyone guide me how to start from beginning.
Thanks

[SinDongHwan]

Hi, @sky-16.
If you can run teamviewer, send to me email.
i will run a code.

[wangsen-sen]

I want to ask, is it necessary to extract all the video frames in the dataset to make a frame dictionary? Or just a few of them? Is it OK to name them in order？

[wangsen-sen]

Did you solve the problem?
I solved!!
I will explain from 'convert frames' to 'summary2video' step by step.

step 1. edit 'videos2frames.sh' script in 'SumMe/videos' directory.

• you edit 'ffmpeg -i "$f" -f image2 frames/"$name"/%5d.png' to
  'ffmpeg -i "$f" -f image2 frames/"$name"/%6d.jpg'
  ( if extract Summe dataset, there is 'SumMe' directory. )

step 2. convert summe dataset to frames.

• i executed 'videos2frames.sh' in 'SumMe/videos' directory.
  ( you should make each directories about .mp4 as 'SumMe/videos/frames/
  For example, if Air_Force_One.mp4, you should make directory 'SumMe/videos/frames/Air_Force_One' )

step 3. have to map h5.file key and frame dirctory name

• in my case, i made json file.
  ( For example, {"video_1":"Air_Force_One", "video_2":"...", ... }.
  you can see the information about mapping following code.

import h5py
import sys
h5_file_name = (sys.argv[1])
f = h5py.File(h5_file_name,'r')
for key in f.keys():
    print("{} : {}".format(key, f[key]['video_name'].value))

if you edit properly "summary2video.py", script is performed.

if you need to my help, ask me.^^

excuse me，I want to ask, when I execute 'videos2frames. sh'.
'Processing ./Air_Force_One.webm file...
./Air_Force_One.webm
bash: ffmpeg: command not found' What should I do?

[SinDongHwan]

Hi, @wangsen-sen.
Install ffmpeg.
ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

[wangsen-sen]

Hello，Thank you for your reply. I use the windows system. It shows that the installation has been completed in the CMD. I don't know why there is a problem during the operation.Do you know why？

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "KaiyangZhou/pytorch-vsumm-reinforce" ***@***.***&gt;; 发送时间:&nbsp;2021年6月1日(星期二) 中午12:34 ***@***.***&gt;; ***@***.******@***.***&gt;; 主题:&nbsp;Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generate problems (#15) Hi, @wangsen-sen. Install ffmpeg. ex) if your OS is ubuntu, "sudo apt-get install ffmpeg". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[wangsen-sen]

Hi, @wangsen-sen.
Install ffmpeg.
ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

Thank you very much for your reply. This problem has just been solved. If you have any questions, I'll ask you again.Good luck to you.

[wangsen-sen]

Hi, @wangsen-sen.
Install ffmpeg.
ex) if your OS is ubuntu, "sudo apt-get install ffmpeg".

Hello, excuse me again. I want to ask if the final video summary can be obtained through the file, which frame is included in the original video？

[SinDongHwan]

Hi, @wangsen-sen.

you can check which frames is selected in result.h5 file.
selected frames are assigned '1', not selected frames are assigned '0'.

[wangsen-sen]

Hi, @wangsen-sen.

you can check which frames is selected in result.h5 file.
selected frames are assigned '1', not selected frames are assigned '0'.

Hello,I processed the result.h5 file and got the final videos tested, but I want to know which frame of the video summary is composed of. Where should I look?

[SinDongHwan]

Hi, @wangsen-sen.

you can check selected frames using following code.

pytorch-vsumm-reinforce/summary2video.py

Line 41 in 77f7c93

summary = h5_res[key]['machine_summary'][...]

[wangsen-sen]

Ok，thank you very much.

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "KaiyangZhou/pytorch-vsumm-reinforce" ***@***.***&gt;; 发送时间:&nbsp;2021年6月3日(星期四) 上午10:21 ***@***.***&gt;; ***@***.******@***.***&gt;; 主题:&nbsp;Re: [KaiyangZhou/pytorch-vsumm-reinforce] Video can't generate problems (#15) Hi, @wangsen-sen. you can check selected frames using following code. https://github.com/KaiyangZhou/pytorch-vsumm-reinforce/blob/77f7c93249240d6fa297818c1eefd5c2b51b1747/summary2video.py#L41 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.