Comments (10)
Also I'm getting ready to make an entire lip syncing system based off your EXE also using open AI in my blender add on so if you could supply that source code... I wouldn't mind also... Giving you whatever I come up with As far as AI integration I don't know if you're a blender User but if you are I make the biggest blender add on in the world code wise it has tons of awesome features so I wouldn't mind throwing that in to sweeten the deal...$50 value
from rhubarb-lip-sync.
Actually I do apologize I just realized that I wasn't signed in to Github and I think as a result it didn't show your source code I just looked back and it was in there...
from rhubarb-lip-sync.
But all that being said I just looked how many files that you have and this is a pretty big repo as far as what you put into it you know about the code can you let me know what file your phonetic logic is in and you created all this so you're really good at all this stuff what do you think of my approach to possibly using Google in place of phonetic do you think it might get better results I'm sure you probably use pocket sphinx so you can use exe in more situations and with more applications.... But also you know about the code if I just went in and replaced the phonetic logic would it work or is your Code all over the place meaning what I have to like search around in all these files and figure stuff out and learn or can I just link it up to whatever variable in that one file and it's done
from rhubarb-lip-sync.
OKI learn more about the code I just ported it over as an update to my add on for using rhubarb with shape keys and I just have a feature request kinda that may help out your end product you know what I like to do to keep a good workflow is import you know my entire character's dialog at once like I imported her saying like three different things so I had three different clips that had you know 10 to 15 seconds of silence between them that separated these clips I created them in Ableton live from replica software....but... the way you do the adding the key frames you only add a mouthe closed at the very end if you would add a mouthe clothes or a rest pose at the very beginning then this would be strengthened for people like me that do this and you know when you're doing animation in the phonetic is only one I really know cuz that's what I just used but when the character you know finished saying a word you know it's common for them to close their mouths like one to two frames after and then open their mouths a lot of times I see a little bit more Keyframes not being closed properly if you know what I'm meaning like the characters done talking and the mouse should be closed but it's just merging into the next keyframe so just saying I would look over the mouse rest I really think with this functionality if the logic is fine tuned it's possible to get a perfect result every time but thank you very much for this awesome software....
from rhubarb-lip-sync.
I'll send a video example if you need it
from rhubarb-lip-sync.
Wow, that's a lot of thoughts at once! (And very little punctuation, which I personally found tough to read. 😉) To answer your questions:
- I agree that PocketSphinx as a speech recognizer is rather outdated, with other libraries giving much better results. However, replacing the recognizer isn't straightforward since there is some interplay between the speech recognition and alignment part. That being said, I'm actively working on Rhubarb 2.0 (#121), which will be a full rewrite based on modern technology. In particular, I plan on using a modern offline STT system like Whisper to combine the quality of Google STT with the offline support of PocketSphinx. However, given my very limited time, I can't make any promises as to when (if ever) it will be done.
- For much the same reason, I'm not planning to make any more changes to Rhubarb 1 except for bug fixes.
- Feel free to embed Rhubarb in your Blender plugin, as long as you properly attribute it.
- I'm not sure whether I understand you correctly regarding closed mouth shapes. Rhubarb does support X as a closed mouth, and it will use this mouth shape whenever the character isn't saying anything. Only for very short silences, it will leave the mouth open for a more natural result. If you experience a different behavior, please report it as a bug, preferably in a separate issue.
from rhubarb-lip-sync.
from rhubarb-lip-sync.
OK Yeah I just noticed what you said about the punctuation yeah I am very very busy I have the biggest blender add on in the world code wise which I can provide to add a little power to my idea about a collaboration.... I'll try and keep these messages concise that'll make them easier to read.... But if you're interested you could look up fast animation studio toolz on blender market.... I don't have a huge amount of sales yet because I've been working on the developing for the last year but I'm just starting once I get these AI feature set up to go full boar into advertising.... This add on really is epic it's like almost two gigabytes of pure code ... It's a tool that makes television production in blender a possibility for one person... And adding the auto lip sync functionality will remove the dependency my add on has on the FACEIT addon which will make it a little bit easier for my end users to approach than having to buy another $79 add some of which which are important for my workflow because other blender add-ons like extreme P B R can't easily be replicated and are necessary...... In my vision of the end result it did include a linking up to ARKIT shape keys... I don't know if you would consider that to be on my end or if you would like to dip your toes in that section the only other issue would be to how to create the arkit shape keys like the FACEIT add-on does... As it is now it costs $79 to get their addon and create the arkit shape keys easily so we at least need to find a cheaper solution...But the end result we could share as far as the blender functionality whatever functionality that we create as a team you could have that part as your own separate blender add-on and sell it separately and I could implement it into my own add on as an extra feature... Just some ideas we both seem to be in a similar situation....
from rhubarb-lip-sync.
I'm sorry, but I'm busy, too, and I refuse to spend my time trying to decipher this kind of spat-out mess.
from rhubarb-lip-sync.
No problem I'm going to create this soon...and if it becomes necessary anywhere in the process to update your code i'll send whatever fixes I make. Don't worry I won't send anything that isn't functional well written and definite benefit... I just realized that it's probably going to be necessary to upgrade it to use open AI whisper so if you want you could just hold off on that and I'll send the end result when I get it done.
from rhubarb-lip-sync.
Related Issues (20)
- Question: can it run at real time? HOT 2
- " is not a valid Shape value HOT 6
- [FEATURE REWUEST] Live printing of visemes HOT 1
- Rhubarb for Spine "Error performing speech recognition via PocketSphinx tools. Error creating speech decoder." HOT 2
- Script stucks After effects MacOS
- Feature Request - Support for OpenAI Whisper speech recognition model
- Error message saying "Check executable path in addon preferences" HOT 1
- Error performing speech recognition via PocketSphinx tools. HOT 4
- [Fatal] Application terminating with error: Error processing file audio.wav. HOT 1
- How to integrate this library in react or nodejs? HOT 4
- Real time animation for tts output HOT 1
- Build Rhubarb as shared library
- Rhubarb - Lip - Sync for 4.0 and above?? HOT 1
- Rhubarb crashes out of memory HOT 2
- Rhubarb crashes on MacOs x86_64
- Rhubarb for adobe animate? HOT 1
- Rhubarb-Lip-Sync can't work in linux HOT 1
- Installing on MacOS 14 Sonoma After Effects
- can I use it to put together an animation command line? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rhubarb-lip-sync.