Comments (9)
How many videos are you processing roughly? Does it occur if all of the paths in video_lists
are the same video? What OS/environment are you running this under, and how are you running the script?
Thank you.
Edit: I was able to verify on Windows x64 at least on my system there is no memory leak with the code pattern you outlined above, this may be something due to the environment.
from pyscenedetect.
How many videos are you processing roughly? Does it occur if all of the paths in
video_lists
are the same video? What OS/environment are you running this under, and how are you running the script?Thank you
Thanks for your reply.
- I use the script to process 30k videos.
- If all of the paths in video_lists are the same video, it still occurs.
- I find this situation on Ubuntu and MacOS.
- I just use 'python3 xxx.py' in a single thread.
from pyscenedetect.
Could you install and try another backend like 'pyav' or 'moviepy'? e.g. run pip install av
and open video with video = open_video(video_path, 'pyav')
PySceneDetect is a pure Python library and offloads all video processing to either OpenCV, PyAV, or MoviePy. The next steps would be to narrow down if the memory leak, so trying a different backend will help greatly with that. In the meantime I will setup a VM to test this with locally on Ubuntu.
Can you create and upload a small script that causes the issue? A video clip as well would be helpful to ensure we are testing the same things and I want to see if we can use the exact same script/video. Thanks.
from pyscenedetect.
Using the same package versions as you outlined above, on Ubuntu 22.04, I'm running all 3 different backends in a loop the same way you described above. From what I can see memory usage for PySceneDetect+OpenCV and PySceneDetect+MoviePy are steady. As for PyAV, the memory does seem to climb at first, but it is eventually reclaimed and drops again. I don't think PySceneDetect and most of the backends in use have any memory leaks, but am happy to look further into this if you can provide a reproduction.
from pyscenedetect.
I have previously had this issue with moviepy, but that was a couple years ago and I haven't run things in a loop like this since then. I fixed it by writing things out to file occasionally and invoking the del
function on a bunch of variables every X number of iterations.
from pyscenedetect.
Using the same package versions as you outlined above, on Ubuntu 22.04, I'm running all 3 different backends in a loop the same way you described above. From what I can see memory usage for PySceneDetect+OpenCV and PySceneDetect+MoviePy are steady. As for PyAV, the memory does seem to climb at first, but it is eventually reclaimed and drops again. I don't think PySceneDetect and most of the backends in use have any memory leaks, but am happy to look further into this if you can provide a reproduction.
Hi, I find that the code uses cv2.VideoCapture to open a video, but without to release it. When I run a for loop, it may cause the memory raising?
from pyscenedetect.
Hi, I find that the code uses cv2.VideoCapture to open a video, but without to release it. When I run a for loop, it may cause the memory raising?
The destructor of a VideoCapture object will call release()
which applies to Python as well. That will happen automatically at the end of each loop iteration. Are you able to reproduce this with a different versions of OpenCV?
I recently came across https://github.com/bloomberg/memray which might be useful to find the cause. A memory flame graph would be really helpful to identify what part of code is using the memory.
from pyscenedetect.
Hello, I recently wrote a synchronous stripping service, and I also have this problem. The free memory keeps decreasing.
from pyscenedetect.
‘ scene_manager.detect_scenes ’ Immediately following the previous answer, the execution of splitting will become slower and slower.
from pyscenedetect.
Related Issues (20)
- Fix Github license detection
- How run it with asyncio and fastapi
- The performance of detecting shot changes is poor HOT 5
- Change `list-scenes` defaults: don't output file, skip cuts
- Ignore time intervals HOT 5
- Remove this issue please
- How to detect entire scenes and not only shot changes? HOT 1
- Add StreamJoiner to concatenate multiple videos
- Inconsistent error handling when start time exceeds video length
- url input ,1080p video much slower HOT 8
- Use csv as input for scene splitting HOT 1
- No output to list-scenes HOT 3
- horizontal wipe cuts
- ffmpeg segment_times options HOT 4
- QP file output HOT 4
- Can't get save_images to work HOT 1
- 'Error splitting video (ffmpeg returned 1) ' occurs while using 'split_video_ffmpeg' HOT 6
- Define minimum and maximum number of splits in split-video HOT 1
- Extend image_name_template to allow timestamp HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyscenedetect.