Recreate any audio track by rearranging the frames of another video
License: MIT License
striving to make audio software that is playful and surprising.
sometimes i break mp3s.
pip install Pillow is required (at least on my pyenv install of 3.9.16) to run stammer.
add a requirements.txt:
scipy
numpy
Pillow
i get this error
python : The term 'python' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ CategoryInfo : ObjectNotFound: (python:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
I'm trying to use a quite long carrier video with multiple sections of music and talking. Unfortunately the linux /tmp folder fills up very easily when it's being used like this when it is on a separate partition from /home (which is a fairly common practice). It might be beneficial for longer inputs to select a different temp folder.
As the title says, say's that i don't have ffmpeg installed , even though i have it installed
The error it gave:
ERROR: 'ffmpeg' not found. Please install it.
Not sure if you would want to fold in additional options for word-by-word matching. If so then this might be a useful ancillary tool. It uses whisper to extract a transcription and corresponding timestamps for each word. I'd have to put it into a TOML layout (or similar legacy action of some kind) for usage to be a straightforward pip/conda affair. As a standalone project it is still a work in progress as I haven't added the ffmpeg subprocess calls to splice the video back together again.
I installed all three dependencies, I put stammer.py in the same directory as ffmpeg, but every time I try to run Stammer, it gives me this message:
"Traceback (most recent call last):
File "C:\ffmpeg\stammer.py", line 17, in
import image_tiling
ModuleNotFoundError: No module named 'image_tiling'"
and then it closes. What's happening?
I've noticed that the audio band arrays can be very memory-intensive. I've seen my RAM easily eaten up just from the "analyzing audio" phase.
We could have array data on disk using numpy.memmap.
btw i installed ffmpeg
The command python stammer.py <carrier>.mp4 <modulator> <output>.mp3 should produce an audio output, but results in "Invalid audio stream" error from ffmpeg. Maybe we need to determine if the extension of the output is a video or audio file? This could be hard-coded for common formats.
Here is the error I got:
westonfleck@Westons-iMac ~ % python3 stammer.py steamedhams.mp3 ohthemisery.mp3 question.mp3
Traceback (most recent call last):
File "/Users/westonfleck/stammer.py", line 254, in
main()
File "/Users/westonfleck/stammer.py", line 247, in main
process(carrier_path, modulator_path, output_path)
File "/Users/westonfleck/stammer.py", line 143, in process
carrier_type = file_type(carrier_path)
File "/Users/westonfleck/stammer.py", line 58, in file_type
return subprocess.run(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffprobe'
line 171 causes an issue where it tries to find a frame that doesn't exist
shutil.copy(frames_dir / f'frame{match_num+1:06d}.png', outframes_dir / f'frame{i:06d}.png')
it's the "match_num+1" part. When it builds the output video-- if it points to the last frame in the frames temp folder, it tries to copy over a frame after that last frame, but it doesn't exist, and so it aborts the whole operation
I tried just removing the +1 from right there, but then it looks for frame000000 in the frames folder. So, I modified it to just copy frame000001 as frame000000
at line 169 I modified the code to be
shutil.copy(frames_dir / f'frame000001.png', frames_dir / f'frame000000.png') //added this line to copy the frame
if type(best_matches) == list:
for i, match_num in enumerate(best_matches):
shutil.copy(frames_dir / f'frame{match_num:06d}.png', outframes_dir / f'frame{i:06d}.png')
I don't know Python like at all but this seems to have fixed the error for me
brilliant program, by the way. I love this thing. This is great
When running python3 stammer.py SteamedHams.mp4 OnJulySecond1988.wav onjuly.mp4, the video is processed to have it's frames dumped into memory, but when it comes time to read the audio (of what?) and analyzing it, the process just gets stuck doing nothing using up one core and is unkillable by doing Ctrl+C.
Calculating video length
Separating video frames
frame= 0 fps=0.0 q=0.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kb
frame= 4047 fps=245 q=-0.0 Lsize=N/A time=00:02:48.75 bitrate=N/A speed=10.2x
reading audio
analyzing audio
This is the current console output.
https://files.catbox.moe/79n9ks.wav
And this is the file "OnJulySecond1988.wav", am I doing something wrong?
Are there any tutorial vids on this to make it easier? I'm reading the instructions but failing to get it working.
I agree that it will save a lot of disk space and allow faster writing of the frames.
One of the reasons you said to use it is because modern computers have plenty of RAM. This makes sense for short videos such as "Steamed Hams". However, it makes less sense for longer ones.
For example, I made a video with this script from a ~2.5 GB 720p copy of "Bee Movie". Uncompressed and lossless image/video formats take up a lot of space, and even with the frames being in the compressed PNG format, the frames totalled around 126 GB (127 GB on disk). For perspective, the computer with which I generated the video has 32 GB of RAM.
How will this problem be addressed? I suspect that a mix of using the actual RAM and writing to disk (maybe via a page file?) could be used. I also believe that once the frames needed to build the video are chosen, the useless ones can be immediately deleted before the build starts.
Whenever I try to run stammer, this happens. I can run other python files just fine, but only when I run stammer does this happen. I have ffmpeg and the requirements installed, so I don't know why this error is happening.
running into an issue with the program. i think the issue may be the length of my audio file modulator file(25min), but I am unsure. (Im an absolutely illiterate idiot with technical code such as this)
If I've done something dumb I apologise
problem usually occurs 7099 frames through the "building output video" segment.
Traceback (most recent call last):
File "C:\Users*name*\Downloads\stammer-main\stammer-main\stammer.py", line 309, in
main()
File "C:\Users*name*\Downloads\stammer-main\stammer-main\stammer.py", line 305, in main
process(vars(args))
File "C:\Users*name\Downloads\stammer-main\stammer-main\stammer.py", line 265, in process
build_output_video(handler, matcher)
File "C:\Users*name\Downloads\stammer-main\stammer-main\stammer.py", line 144, in build_output_video
video_handler.write_frame(video_frame_i,video_handler.get_frame(carrier_video_frame+1))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users*name*\Downloads\stammer-main\stammer-main\video_out.py", line 90, in get_frame
super().get_frame(idx)
File "C:\Users*name*\Downloads\stammer-main\stammer-main\video_out.py", line 41, in get_frame
assert(idx > self.framecount)
^^^^^^^^^^^^^^^^^^^^^
AssertionError
edit: i got a temp fix working by setting the output as .mp3
Traceback (most recent call last):
File "E:\Users\Acer2019\Desktop\[my real name;;.;]\stammer-main\stammer.py", line 254, in
main()
File "E:\Users\Acer2019\Desktop\[my real name;;.;]\stammer-main\stammer.py", line 247, in main
process(carrier_path, modulator_path, output_path)
File "E:\Users\Acer2019\Desktop\[my real name;;.;]\stammer-main\stammer.py", line 143, in process
carrier_type = file_type(carrier_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Users\Acer2019\Desktop\[my real name;;.;]\stammer-main\stammer.py", line 58, in file_type
return subprocess.run(
^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\subprocess.py", line 1024, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\Python311\Lib\subprocess.py", line 1493, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
I need some help plz, I just started the project, And it's now confusing me.
Pictures would've been better though.
this is on a larger video keep that in mind the audio and video I used for input is ~30min long
here is the console output
[image2 @ 000001fa64223580] Could not open file : C:\Users\kamil\AppData\Local\Temp\tmpkvu1s3ml\frames\frame027618.png av_interleaved_write_frame(): I/O error Calculating video length Separating video frames Traceback (most recent call last): File "G:\odds and ends\stammer-main\stammer.py", line 285, in <module> main() File "G:\odds and ends\stammer-main\stammer.py", line 281, in main process(**vars(args)) File "G:\odds and ends\stammer-main\stammer.py", line 207, in process subprocess.run( File "C:\Users\kamil\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ffmpeg', '-loglevel', 'error', '-i', WindowsPath('Visions.mp4'), WindowsPath('C:/Users/kamil/AppData/Local/Temp/tmpkvu1s3ml/frames/frame%06d.png')]' returned non-zero exit status 1.
I have tried this script with smaller videos and it has worked but for this larger project it ends up failing with this error
Traceback (most recent call last):
File "/home/user/Code/python/stammer/stammer.py", line 309, in <module>
main()
File "/home/user/Code/python/stammer/stammer.py", line 305, in main
process(**vars(args))
File "/home/user/Code/python/stammer/stammer.py", line 222, in process
subprocess.run(call,check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ffmpeg', '-v', 'quiet', '-stats', '-i', '/home/user/Videos/carrier.mp4', '/tmp/tmpfx2yx0my/frames/frame%06d.png']' returned non-zero exit status 228.
I tried to use this, but it didn't work. It worked the first time, but now it's showing this:
Traceback (most recent call last):
File "/content/stammer.py", line 309, in
main()
File "/content/stammer.py", line 305, in main
process(**vars(args))
File "/content/stammer.py", line 265, in process
build_output_video(handler, matcher)
File "/content/stammer.py", line 157, in build_output_video
video_handler.complete()
File "/content/video_out.py", line 111, in complete
subprocess.run(call,check=True)
File "/usr/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ffmpeg', '-v', 'quiet', '-y', '-framerate', '29.931093884582257', '-stats', '-i', '/tmp/tmp82rrbr7g/outframes/frame%06d.png', '-i', '/tmp/tmp82rrbr7g/out.wav', '-c:a', 'aac', '-c:v', 'libx264', '-crf', '20', '-pix_fmt', 'yuv420p', '-shortest', '/content/outputs/srwyh.mp4']' returned non-zero exit status 1.
I am inputting the following
stammer.py "C:\Users\teamn\OneDrive\Documents\Steamed_Hams.mp4" "C:\Users\teamn\OneDrive\Documents\Disciple.mp3" "C:\Users\teamn\OneDrive\Documents\stammer-main\outputs\test.mp4"
and everytime i have inputted this i get back this
Traceback (most recent call last):
File "C:\Users\teamn\OneDrive\Documents\stammer-main\stammer.py", line 372, in
main()
File "C:\Users\teamn\OneDrive\Documents\stammer-main\stammer.py", line 368, in main
process(**vars(args))
File "C:\Users\teamn\OneDrive\Documents\stammer-main\stammer.py", line 264, in process
raise FileNotFoundError(f"Carrier file {carrier_path} not found.")
FileNotFoundError: Carrier file C:\Users\teamn\OneDrive\Documents\Steamed_Hams.mp4 not found."
am i doing something wrong here?
Edit: Moving the files to different folders also produces the same error
Edit 2: I do in fact have ffmpeg installed
log (no that is not my real name that would be a stupid real name anyways)
Traceback (most recent call last):
File "C:\Users\snear\Desktop\videotest\stammer-main\stammer.py", line 254, in
main()
File "C:\Users\snear\Desktop\videotest\stammer-main\stammer.py", line 247, in main
process(carrier_path, modulator_path, output_path)
File "C:\Users\snear\Desktop\videotest\stammer-main\stammer.py", line 143, in process
carrier_type = file_type(carrier_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snear\Desktop\videotest\stammer-main\stammer.py", line 58, in file_type
return subprocess.run(
^^^^^^^^^^^^^^^
File "C:\Users\snear\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\snear\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1024, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\snear\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1493, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
This project works very well, but it temporarily uses a lot of disk space. ffmpeg can write all images of a video to pipe, which Python can read. It's then possible to build output frames in memory too.
As computers today have many GBs of memory, this could save ffmpeg the trouble of unnecessarily writing image files which will be soon deleted anyway.
I have a fork of this at home which does the read-into-memory, but as of writing, not the output-building.
Thoughts?
The title explains above.
A few users understandably don't have the ffmpeg prerequisite, so STAMMER should check if they don't and warn them.
This is a proposal from @mayhair in #11:
I also believe that once the frames needed to build the video are chosen, the useless ones can be immediately deleted before the build starts.
This is kind of the idea behind the new "mem_decay" mode, but it could also work for the default "write frames to disk" mode.
I would implement this as a list of ranges generated from the best matches. We sort the best matches, then create range() objects, then call ffmpeg for each range of frames we need generated.
C:\Users*myname*\Downloads\stammer-main\stammer-main>python stammer.py C:\Users*myname*\Videos\return_of_the_kipz360P.mp4 C:\Users*myname*\Music\BGMs\badapel.mp3 C:\Users*myname*\Downloads\stammer-main\stammer-main\outputs\cccc.mp4
Traceback (most recent call last):
File "C:\Users*myname*\Downloads\stammer-main\stammer-main\stammer.py", line 254, in
main()
File "C:\Users*myname*\Downloads\stammer-main\stammer-main\stammer.py", line 247, in main
process(carrier_path, modulator_path, output_path)
File "C:\Users*myname*\Downloads\stammer-main\stammer-main\stammer.py", line 143, in process
carrier_type = file_type(carrier_path)
File "C:\Users*myname*\Downloads\stammer-main\stammer-main\stammer.py", line 58, in file_type
return subprocess.run(
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ffprobe', '-loglevel', 'error', '-show_entries', 'stream=codec_type', '-of', 'csv=p=0', WindowsPath('\u202aC:/Users/myname/Videos/return_of_the_kipz360P.mp4')]' returned non-zero exit status 1.
I'm not sure what is causing this but it errors out no matter if I use MP4 or MP3 files, seems to be related to when it rearranges the audio maybe?
Calculating video length
Separating video frames
reading audiops= 63 q=0.0 Lsize=N/A time=00:00:02.50 bitrate=N/A dup=1 drop=0
c:\Games\stammer-main\stammer.py:240: WavFileWarning: Chunk (non-data) not understood, skipping it.
_, carrier_audio = wavfile.read(get_audio_as_wav_bytes(carrier_path))
c:\Games\stammer-main\stammer.py:241: WavFileWarning: Chunk (non-data) not understood, skipping it.
_, modulator_audio = wavfile.read(get_audio_as_wav_bytes(modulator_path))
analyzing audio
Traceback (most recent call last):
File "c:\Games\stammer-main\stammer.py", line 307, in
main()
File "c:\Games\stammer-main\stammer.py", line 303, in main
process(**vars(args))
File "c:\Games\stammer-main\stammer.py", line 246, in process
matcher = BasicAudioMatcher(carrier_audio, modulator_audio, INTERNAL_SAMPLERATE, frame_length)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Games\stammer-main\audio_matching.py", line 19, in init
self.make_best_matches()
File "c:\Games\stammer-main\audio_matching.py", line 62, in make_best_matches
self.carrier_frames = self.make_frames(self.carrier, self.samples_per_frame)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Games\stammer-main\audio_matching.py", line 30, in make_frames
frames = np.zeros((num_frames, frame_length * 2), dtype=input_audio.dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: negative dimensions are not allowed
Using long frame-lengths for stammer results in a more YTP-style output, compared to the static-like output at short frame-lengths. Making frame-length an argument would increase usability.
I implemented this on my copy, can make a pull request if this is judged to be a good idea.
C:\Users[USER]>python C:\Users[USER]\Downloads\stammer-main\stammer.py Descargas\steamed-hams.mp4 C:\Users[USER]\Downloads\street-fighter-iii-3rd-strike-lets-get-it-on.mp4 C:\Users[USER]\Downloads\stammer-main\outputs\thirdstrike.mp4
Traceback (most recent call last):
File "C:\Users[USER]\Downloads\stammer-main\stammer.py", line 20, in
import video_out
File "C:\Users[USER]\Downloads\stammer-main\video_out.py", line 2, in
from decay_cache import DecayCache
File "C:\Users[USER]\Downloads\stammer-main\decay_cache.py", line 13, in
class DecayCache:
File "C:\Users[USER]\Downloads\stammer-main\decay_cache.py", line 14, in DecayCache
array: dict[Frame] = {}
TypeError: 'type' object is not subscriptable
I don't know what I'm doing wrong sorry this is the first python code that I run
I Installed all the things the readme tells me
i think ffprobe is detecting a single image (album art) as video in my carrier.mp3, causing this line in process:
if 'video' in carrier_type:
to be true. i ran ffprobe with debug loglevel and got this output:
Duration: 00:05:53.31, start: 0.000000, bitrate: 256 kb/s
Stream #0:0, 50, 1/14112000: Audio: mp3, 44100 Hz, stereo, fltp, 256 kb/s
Stream #0:1, 1, 1/90000: Video: mjpeg (Baseline), 1 reference frame, yuvj420p(pc, bt470bg/unknown/unknown, center), 1280x720 [SAR 1:1 DAR 16:9], 0/1, 90k tbr, 90k
tbn (attached pic)
Metadata:
comment : Other
audio
video
i tried this:
ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames -print_format csv=p=0 carrier.mp3
and it returned 1
i get this error when i run stammer:
File "C:\Users\thoma\source\stammer\stammer.py", line 61, in make_frames
frames = np.zeros((num_frames, frame_length * 2), dtype=input_audio.dtype)
ValueError: negative dimensions are not allowed
after removing the album art and running again:
ffprobe -loglevel error -show_entries stream=codec_type -of csv=p=0 carrier.mp3
returns only 'audio' and stammer works fine.
maybe stammer can use the file extension rather than ffprobe to determine the file type?
Traceback (most recent call last):
File "C:\Users\jspar\Downloads\stammer-main\stammer-main\stammer.py", line 20, in
import video_out
File "C:\Users\jspar\Downloads\stammer-main\stammer-main\video_out.py", line 2, in
from decay_cache import DecayCache
File "C:\Users\jspar\Downloads\stammer-main\stammer-main\decay_cache.py", line 13, in
class DecayCache:
File "C:\Users\jspar\Downloads\stammer-main\stammer-main\decay_cache.py", line 14, in DecayCache
items: dict[DecayItem] = {}
TypeError: 'type' object is not subscriptable
This is my console output:
Separating video frames
copying audio
Error while decoding stream #0:0: Invalid data found when processing input
reading audio
analyzing audio
finding best matches
creating output audio
building output video
Traceback (most recent call last):
File "/Users/myname/Desktop/folder/stammer/stammer.py", line 254, in
main()
File "/Users/myname/Desktop/folder/stammer/stammer.py", line 247, in main
process(carrier_path, modulator_path, output_path)
File "/Users/myname/Desktop/folder/stammer/stammer.py", line 225, in process
build_output_video(frames_dir, outframes_dir, best_matches, 1/frame_length, output_path)
File "/Users/myname/Desktop/folder/stammer/stammer.py", line 105, in build_output_video
shutil.copy(frames_dir / f'frame{match_num+1:06d}.png', outframes_dir / f'frame{i:06d}.png')
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/shutil.py", line 419, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/shutil.py", line 256, in copyfile
with open(src, 'rb') as fsrc:
^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'temp/frames/frame017558.png'
It never creates the file frame017558.png when creating frames.
the terminal is saying no module named numpy even though I have it. Here is my terminal
C:\Users\westo> pip install numpy
Requirement already satisfied: numpy in c:\users\westo\appdata\local\programs\python\python311\lib\site-packages (1.25.2)
C:\Users\westo>python3 C:\Users\westo\Downloads\stammer-main\stammer-main\stammer.py C:\Users\westo\Downloads\tpot6.mp4 C:\Users\westo\Downloads\bfdi1a.mp4 C:\Users\westo\Downloads\outputBFDI.mp4
Traceback (most recent call last):
File "C:\Users\westo\Downloads\stammer-main\stammer-main\stammer.py", line 5, in
import numpy as np
ModuleNotFoundError: No module named 'numpy'
If the audio match frames are a linear combination of the best matches, then the audio output could be tuned to be more or less like the source audio--truncation at one element is effectively the current implementation.
This also leaves open the possibility of interesting combinations of the visual frames (i.e, the visual frames corresponding to the audio matches [basis vectors]).
One straightforward visualization would be vertical slices with the width assigned as a fraction of the frame corresponding to its coefficient in the linear combination.
Another possibility is the use of a graphic equalizer-like effect--but which would only work well if there are time periods with relatively moderate changes to the basis vectors/frames. Each bar in the equalizer then represents a frame and the coefficient is its height. A bar could drop out as the influence of the particular frame fell to zero.
I was trying to run STAMMER on some mp3 files, but instead got this error:
Calculating video length
reading audio
analyzing audio
Traceback (most recent call last):
File "C:\Users\tyryry\Desktop\STAMMER\stammer.py", line 307, in <module>
main()
File "C:\Users\tyryry\Desktop\STAMMER\stammer.py", line 303, in main
process(**vars(args))
File "C:\Users\tyryry\Desktop\STAMMER\stammer.py", line 246, in process
matcher = BasicAudioMatcher(carrier_audio, modulator_audio, INTERNAL_SAMPLERATE, frame_length)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyryry\Desktop\STAMMER\audio_matching.py", line 19, in __init__
self.make_best_matches()
File "C:\Users\tyryry\Desktop\STAMMER\audio_matching.py", line 66, in make_best_matches
self.modulator_bands = self.make_normalized_bands(self.modulator_frames)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyryry\Desktop\STAMMER\audio_matching.py", line 40, in make_normalized_bands
spectra = abs(transforms[:,1:len(transforms[0])//2])
~~~~~~~~~~^^^
IndexError: index 0 is out of bounds for axis 0 with size 0
It seems to happen everytime I'm using an mp3 file as the carrier track.
However, I found out that if I were to delete the first few milliseconds of silence at the start of the mp3 track (as mp3 files are) the program worked as expected.
Based on the error
IndexError: index 0 is out of bounds for axis 0 with size 0
I think the error might be because of the silence (which have empty frames in them), and the empty frames cause issues when trying to create an array (hence out of bounds for an empty array of 0). I tried creating a file with ONLY silence to test this, but STAMMER seemed to work just fine with the silent file, even when it was a .mp3.
I think when I tried this with .oggs the program also worked just fine.
In YTPs, it's common to use sped up and slowed down audio/video clips of the source. We could do the same thing, which would add variety to the output, especially for short carriers, and increase how much the output matches the modulator. The benefit would be more noticeable at larger frame-lengths.
The specific process I'm envisioning is that copies of the carrier are made at something like 70%, 200%, and 300% speed, then a new file is created that's the original and the copies all stitched together end-to-end. I assume all that can be done with ffmpeg. After that, the new file is analyzed like normal.
I feel like this is within the scope of the project in a way just pitching up/down frames or vocoding the output wouldn't be, but I don't think it should be default behavior.
D:\steamer\stammer-main>python stammer.py D:\steamer\stammer-main\hams.mpeg D:\steamer\stammer-main\applevocal.wav D:\steamer\stammer-main\outputs\applehamfull.mp4
Calculating video length
Traceback (most recent call last):
File "D:\steamer\stammer-main\stammer.py", line 291, in
main()
File "D:\steamer\stammer-main\stammer.py", line 287, in main
process(**vars(args))
File "D:\steamer\stammer-main\stammer.py", line 200, in process
carrier_framecount = float(get_framecount(carrier_path))
ValueError: could not convert string to float: '5105,\n\n'
Trying to use the program results in this:
Ok, I have the models and the files, but what should I do with the ffmpeg and how do I write the Output file? Do I simply add the format I want or do I have to do something else?
Putting these links here because I think they're interesting previous approaches to this and similar problems, and I mean to study them in more detail:
There are definitely some really interesting things that we're doing here that these other apps aren't doing and vice versa. I think there's a lot of potential to make a really great utility that learns from the strengths of these other programs, and that explores new ground with breathing the life of one sample into another.
I just wanna give a huge THANK YOU to everyone who's contributing to this project. I am amazed and humbled by all of your great work.
For reference, I am running python 3.8.6, is there another python version I need to install?
building output video
Traceback (most recent call last):
File "C:\Users\drewd\stammer-main\stammer.py", line 291, in <module>
main()
File "C:\Users\drewd\stammer-main\stammer.py", line 287, in main
process(**vars(args))
File "C:\Users\drewd\stammer-main\stammer.py", line 255, in process
build_output_video(frames_dir, outframes_dir, matcher, real_frame_length, frame_length, carrier_framecount, output_path)
File "C:\Users\drewd\stammer-main\stammer.py", line 117, in build_output_video
shutil.copy(frames_dir / f'frame{carrier_video_frame+1:06d}.png', outframes_dir / f'frame{video_frame_i:06d}.png')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Unknown format code 'd' for object of type 'float'
this is seems to be in the same place where the program was trying to generate frames that were beyond the end of the video
My program has read and generated 87,211 frames, but it keeps trying to access frame 87,235. I don't know anything of why it's trying to do that.
Tested with carrier as video and modulator as audio. Not sure exactly what the problem is.
Traceback:
File "C:\stammer\audio_matching.py", line 74, in make_output_audio
output_audio = self.build_output_audio()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\stammer\audio_matching.py", line 169, in build_output_audio
output_audio[i*self.samples_per_frame : i*self.samples_per_frame + self.samples_per_frame*2] += composed_frame
numpy.core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'add' output from dtype('O') to dtype('float64') with casting rule 'same_kind'`
C:\Users\*me*>python D:\stammer-main\stammer.py D:\*folder\*carrier*.mp4 D:\*other folder*\*modulator*.mp3 D:\stammer-main\outputs\test.mp4
ERROR: 'ffprobe' not found. Please install it.
Traceback (most recent call last):
File "D:\stammer-main\stammer.py", line 372, in <module>
main()
File "D:\stammer-main\stammer.py", line 357, in main
test_command(['ffprobe', '-version'])
File "D:\stammer-main\stammer.py", line 29, in test_command
raise error
File "D:\stammer-main\stammer.py", line 26, in test_command
subprocess.run(cmd, capture_output=True)
File "C:\Users\cleme\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 466, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\*me*\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 769, in __init__
restore_signals, start_new_session)
File "C:\Users\*me*\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1172, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
is there an issue with ffprobe here? also I did put ffmpeg in my Path
I'm confused.. I'm inputting "python stammer.py boot adventures.mp4 futurama.mp4 C:\Users\User\Desktop\Folders\stammer-main\boot.mp4" and it's saying it's an unrecognized argument.. what do I do
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.