Feature Request - Speech / Phonetics automatic generation/ alignment

[morevnaproject-org]

Feature Request - Speech / Phonetics automatic generation/ alignment

[steveway]

If you downloaded from my fork then it's best to use the master branch. That is the most complete one.

[steveway]

Something more related to the original topic.
I just found Vosk, for which there also seems to be some support to get phonemes out of.
alphacep/vosk-api#528
While the results from Allosaurus are very good, this might be an interesting alternative.

[aziagiles]

@Hunanbean Ok. I get your point. Best wishes as you fix the issue. I believe you can do this.

[Hunanbean]

@aziagiles Ok, i think i've got it worked out.. I will post the fix to the Git as long as it works for you too.
to fix it real quick, replace
from bpy.props import *
with
from bpy.props import EnumProperty, FloatProperty, StringProperty, PointerProperty, IntProperty

[Hunanbean]

@Hunanbean Ok. will do just that and give you feedback.

incase it is easier, i've gone ahead and updated the git at https://github.com/Hunanbean/Papagayo-NGLipsyncImporterForBlender
I will try again, if that does not work

[aziagiles]

@aziagiles Ok, the biggest issues is, i use Shape Keys which got fixed by the change in that line. I am not set up to even try Pose Libraries.. Is there a way you can send me a generic file with a pose library i can use to test with?

Ok. Let me prepare a file and send. Me I use grease pencil, and my phonemes are controlled by a bone.

[Hunanbean]

I've just replied you on your email box, with the blend file attached.

Ok, i just got it, thanks. I will see what i can do. This is going to take some time though, so probably will not have much until tomorrow or the day after as they are working on the power here, and it is scheduled to be down for several hours :/

[morevnaproject]

I was thinking about that too, and started to investigate. This is what I found - https://cmusphinx.github.io/wiki/phonemerecognition/

Frequently, people want to use Sphinx to do phoneme recognition. In other words, they would like to convert speech to a stream of phonemes rather than words. This is possible, although the results can be disappointing. The reason is that automatic speech recognition relies heavily on contextual constraints (i.e. language modeling) to guide the search algorithm.

For now, I think integrating with RhubarbLipSync (#44) is a way to go.

[morevnaproject]

We've got Rhubarb feature merged just now. - #50 ^__^

[Hunanbean]

Montreal Forced Aligner may be something to look into, but that would be more for automatic alignment from the text, rather than the full shebang.

[steveway]

As mentioned we currently have Rhubarb integrated.
But I just found an interesting project for this call Allosaurus.
It seems to be pretty easy to use and here on Windows 10 it was very easy to pip install it.
The only problem is that it outputs IPA Phonemes and that it does not provide any timestamps (yet).
We should be able to create a mapping for the phonemes to the ones we support already.
And for the timestamps there is already an open issue: xinjli/allosaurus#24
But even without timestamps it might already be usable with some conversion of the phonemes.

[steveway]

Here is a very simple conversion dict from IPA to CMU:
{ "b": "B", "ʧ": "CH", "d": "D", "ð": "DH", "f": "F", "g": "G", "h": "HH", "ʤ": "JH", "k": "K", "l": "L", "m": "M", "n": "N", "ŋ": "NG", "p": "P", "r": "R", "s": "S", "ʃ": "SH", "t": "T", "θ": "TH", "v": "V", "w": "W", "j": "Y", "z": "Z", "ʒ": "ZH", "ɑ": "AA2", "æ": "AE2", "ə": "AH0", "ʌ": "AH2", "ɔ": "AO2", "ɛ": "EH2", "ɚ": "ER0", "ɝ": "ER2", "ɪ": "IH2", "i": "IY2", "ʊ": "UH2", "u": "UW2", "aʊ": "AW2", "aɪ": "AY2", "eɪ": "EY2", "oʊ": "OW2", "ɔɪ": "OY2" }
It's based on this mapping from CMU to IPA.
https://github.com/margonaut/CMU-to-IPA-Converter/blob/master/cmu_ipa_mapping.rb

[Hunanbean]

I must have underestimated what Rhubarb actually does. I will take a look at it now.
In the CMU phoneme set i did, i purposely simplified it to remove the specific variants, such as AO1, AO2 would both become just AO. But, i am pretty sure i still have the full setup before i truncated it if you want me to post that on my git. But due to the imperceivable differences between AO, AO1, AO2 in action, perhaps it makes more sense to just have the conversion dictionary truncate to the existing set of 39

[steveway]

Yes, Rhubarb is quite nice, it would be awesome if it could also output text besides phonemes.
With our language dictionaries we could try to convert phonemes back to words too for that.
The results from Rhubarb are not as exact as our manual methods, but I guess for most animations it's enough.
I don't think we need the untruncated list for CMU.
I just quickly generated that list up there based on that little converter from @margonaut.
If we really want to integrate Allosaurus then we should make a fitting conversion table for our phoneme list.
We can use that information to create a new phoneme_set and phoneme_conversion dictionary for IPA.
And we should add some code to use these to convert between different phoneme sets, that should already kinda be possible in a limited way.

[steveway]

I now have an allosaurus branch: https://github.com/steveway/papagayo-ng/tree/allosaurus
This currently uses pydub to prepare the sound files for allosaurus.
This works very well with our Tutorial Files, even the spanish ones.
The results seem to be better than what Rhubarb provides.
Here is a quick test showing the result for running it on the lame.wav file:
https://youtu.be/4hqHaEXo9xU
The phonemes are partially overlapping, so some pruning needs to be done for animation purposes.
But as you can see the results are quite good.

[Hunanbean]

That is very cool! Thank you I just ditched windows and went back to linux last night, so it is going to take me a little while before i can test it. Seems like now would be a good time to start making some noise on the forums. That looks like a Patreon magnet to me!

…

On Wed, Jun 2, 2021 at 1:03 AM Stefan Murawski ***@***.***> wrote: I now have an allosaurus branch: https://github.com/steveway/papagayo-ng/tree/allosaurus This currently uses pydub to prepare the sound files for allosaurus. This works very well with our Tutorial Files, even the spanish ones. The results seem to be better than what Rhubarb provides. Here is a quick test showing the result for running it on the lame.wav file: https://youtu.be/4hqHaEXo9xU The phonemes are partially overlapping, so some pruning needs to be done for animation purposes. But as you can see the results are quite good. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMNVR7D3RQZHLOXBOMRFTILTQXQUTANCNFSM4FYFKTXQ> .

[steveway]

Alright, I made some more progress.
Based on the time between phonemes from the automatic recognition I chunk the phonemes together to single "words".
This should make editing the results a bit easier.
Also I added some simple logic to convert between different phoneme sets, but for that we likely need some good hand-made phoneme conversion dicts.
Also at the moment I convert first to CMU39 and then from that to the desired ones.
For the best results we should create conversions between each set manually.

I think the direct next step would be to add some GUI option to change the selected phonemes/words/phrases to belong to a different voice.
That way after auto recognition did it's work we just need to separate the parts to the different voices, if there are any and then it's pretty much done.

And for the future some automatic speaker diarization would be awesome, that way we could automate almost everything.

[steveway]

Ok, it's now available.
I've created a first pull request for this: #94
The results are pretty good, I would recommend everyone to test this.
You should download FFmpeg and the allosaurus AI model using the actions at the top of Papagayo-NG first, then restart Papagayo-NG and it should all work.

[Hunanbean]

It is working and Very impressive! Thank you very much!

[aziagiles]

@steveway Everything is working except the 'Convert Phonemes' function which doesn't really function well. I think it still needs some fixing. I'm on a windows 10 platform.

[aziagiles]

@steveway Also for the default 'cmu_39' automatic breakdown, the breakdown goes well for all the earlier parts of the input audio but it doesn't breakdown the end of the audio.

[steveway]

Yes, the conversion needs some work, as I mentioned:

Also I added some simple logic to convert between different phoneme sets, but for that we likely need some good hand-made phoneme conversion dicts.
Also at the moment I convert first to CMU39 and then from that to the desired ones.
For the best results we should create conversions between each set manually.

Can you show what files it does not break down all the way?
The test files worked pretty well, the automatic breakdown is being done by https://github.com/xinjli/allosaurus so there might not be much we can do depending on the cause.

[Hunanbean]

Yes, there appears to be a problem with it truncating the last about .5 seconds of the audio file. I will see if it works if i add some empty time at the end of the audio file.

[aziagiles]

@steveway Ok. I made a video of the 2 worries I had. That of the conversion from CMU39 to Preston Blair not working properly as alot of missing phonemes are notified. The second worry was that the last phrase or words in the audio are not broken down.

phoneme.conversion.mp4

[Hunanbean]

Ok, after i added 1 second of silence to the end of the audio file, it now picks up the last phrase

Edit: I was mistaken, it still truncates the end. It was just the audio lining up to the end, not the actual conversion. The last word/words are still truncated

[aziagiles]

@Hunanbean I just added a second silence at the end of my audio, and after the breakdown, it ended exactly where my audio sound ended but unfortunately did not still pick the last phrase.

[Hunanbean]

Hmm. Perhaps it just cannot recognize the last phrase, or more silence needs to be added?

Edit: My mistake. You are correct. It is still truncating the end. It was just the audio now finishing at the correct spot

[Hunanbean]

yes, verified that any file I try, regardless of added silence at the end, does truncate the last phrase.

[steveway]

I think I found the cause.
There is of course still the possibility that allosaurus can't recognize all the phonemes.
But my code did accidentally skip the last few phonemes in some cases.
The reason was the logic I used to chunk phonemes into possible "words", I use some peak detection of the time between the phonemes to split between possible words.
If that result was uneven then the loop over that would skip the last one.
I changed this a bit now: steveway@fb94377

[Hunanbean]

With the files I added one second of silence to, this now picks up that last phrase. However, the problem still remains on the same files without the added silence.

Thank you

[steveway]

Allosaurus is likely not picking up that part at all.
Can you send a file which has this problem?
I can test to see if it is really Allosaurus or something we do then.

[Hunanbean]

Here is an example. The jenna file says the same words, but is recognized without silence added. The salli file says the same words, but the last portion is only recognized with silence added. For the salli voice, both with and without silence are included.
ZippityDoDa.zip

Thank you

[steveway]

I now add half a second of silence at the end for Allosaurus, if you then increase the emission to about 1.4 it recognizes your file to the end.
It's a bit strange, without upping the emission it doesn't work, even if I add 10 seconds of silence.
You can test this with a re-download and install of this release: https://github.com/steveway/papagayo-ng/releases/tag/1.6.1

[Hunanbean]

That is working well. Thank you

[aziagiles]

@steveway The 'Convert Phonemes' function not working properly in the current master allosaurus branch. When converting from CMU39 to Preston Blair, Rhubarb or Fleming Dobbs, alot of missing phonemes are notified.

[steveway]

@aziagiles I see, I fixed this just now in Commit: steveway@fc971f1
I also saw that there are some more accesses to the voices via the LipSyncDoc Objects voices list.
All those accesses should be replaced by using the new Nodes.
For now it should still work.

[steveway]

Sorry, I didn't want to close this, must have miss-clicked.
Anyway, this is now pretty solid and I believe more stable than the old version.

[aziagiles]

@steveway The convert phonemes function now works great. Thanks for the fix. But please, how do we use the rhurbarb for automatic speech/phonetics generation/alignment in this version of Papagayo-NG especially as allosaurus conversion comes already automatic at the start when our audio is loaded.

[steveway]

Hello @aziagiles, before this was determined by some simply logic, first it will try Allosaurus and if that fails it will try Rhubarb.
But I changed this now.
With my new changes you have the option to manually select between Rhubarb and Allosaurus.
For that there is a Combobox in the settings and the button for VoiceRecognition now has a little arrow which will pop out a menu where you can change that same selection.
5c88ef9

[aziagiles]

@steveway Hello Steve. I can't resize the audio text length in waveform from the end when using preston blair, fleming dobbs or rhubarb breakdowns.

[aziagiles]

@steveway Also, the Convert Phonemes function doesn't work from Rhubarb or Fleming dobbs breakdowns to Preston Blair.

[aziagiles]

@steveway Please, how does the Add, Remove and change Voice functions work?

[steveway]

Hi, I see. I believe I found the cause and fixed it.
The problem was that for the right side constraint we fell back on "0" when we couldn't get the information from the parent objects.
This is now fixed, it should now always get the last frame in this case.
I also fixed the Conversions a bit.
The problem is that the old Project files don't save the Phoneme Set used, and the new .pg2 also didn't.
I updated the new ones to save it, the old ones can't be changed because this would create possible problems with other projects which try to use that.
There is also now a new menu, one button tries to convert as before and the other one sets the currently selected set as the used one.
So if you have an old project you can first select the phoneme-set and set it and then convert it to the destination.
Adding and removing Voices can be done with the little + and - Buttons.
You will see that a new Tab will appear for the newly created voice.
You can switch by clicking on the tabs and remove one by clicking on - .

[aziagiles]

@steveway Hello Steve. I really appreciate the work you keep putting on this version of Papagayo-NG. I tested the recent version and noticed a slight error on the Convert Phonemes function. Conversion from Rhubarb to other phoneme breakdowns doesn't work.

[steveway]

I see why that is.
The phoneme conversion for Rhubarb has never been adjusted:
https://github.com/morevnaproject-org/papagayo-ng/blob/master/phonemes/rhubarb.json
It seems it was based on the Preston Blair one, but nobody adjusted the phoneme_conversion dictionary yet.

[steveway]

I started a little tool which can help to create these conversions.
For now it only displays the mouths for the CMU set and the Rhubarb one in a grid with some Checkboxes.
There is no functionality behind the Checkboxes.
But for visualizing this might help.
Maybe instead of a bunch of Checkboxes we can use some Dropdowns.
In that case we will show the Pictures for the CMU set as is now and right besides there will be a dropdown with the rhubarb pictures inside besides each one.
Then you could go through the CMU phonemes one by one and select the best fitting Rhubarb one right next to it.

[steveway]

And another update.
I changed the Helper Tool as I mentioned.
It can also now save the changes.
Currently some things are hard coded in that script, like the input and output paths.
This is how this tool looks:

So, just start it, go through the list from the top to the bottom and select the fitting phonemes and at the bottom click on save.
It will save a "test_set.json" file in the phonemes directory.
If everything works out then switching the rhubarb.json with that should fix the problem with the conversions of that.
I will likely improve the tool so it can show and modify any phoneme set.
There are likely improvements we can make for the other sets.

[Hunanbean]

It is amazing what you have turned this software into. Thank you!
I look forward to testing this later today.

p.s. If you note any improvements that should be made to the CMU mouth shapes, please let me know

[aziagiles]

You're right @Hunanbean . He @steveway has really turned this software into something very huge and awesome. But somehow, It will surprise you guys to know I haven't yet updated to Papagayo NG 1.6.4.1 but I still use Papagayo NG 1.6.3. I believe somehow, allosaurus and allosaurus conversion to other phoneme breakdown is better in version 1.6.3 than 1.6.4.1. In version 1.6.3 after conversion, the mouths change on a more normal and natural speed, while in version 1.6.4.1 after conversions, the mouths change extremely fast and not very natural.

[steveway]

Thanks everyone, I hope Papagayo-NG will be used more thanks to these improvements.

@aziagiles That is likely because of the Emission Multiplier.
Try lowering the value slightly below 1 maybe.

I gave the Phoneme Conversion Helper Tool a big overhaul already.
You can select which set you want to modify and then it will allow you to modify conversions for the other sets.

Here I selected the Rhubarb Set and it allows me to modify the conversions to CMU 39, Preston Blair and Fleming Dobbs as you can see.
At the moment it will save it to a file with a slightly different name in the phonemes directory.
And Papagayo-NG needs to be adjusted to use these new conversions if available.
So, if anyone wants to try improving the conversions, go ahead. 👍

[aziagiles]

@steveway Ok I get it. Thanks for the detail explanation and improvements.

[steveway]

No Problem.
I've done a few more changes now on my current work branch: https://github.com/steveway/papagayo-ng/tree/cli_test
Papagayo-NG is now changed to use these new conversions.
At the moment only the conversions from CMU_39 will do anything useful because only these have been created.
With the help of the new helper tool we should be able to create the other conversions, these have to be created for every combination.
But the end result should be able to be more correct than the current hack we have now.

[steveway]

Alright, I've improved the Helper Tool very slightly.
And I've created some first conversions for all the combinations.
You can test this out with this branch:
https://github.com/steveway/papagayo-ng/tree/cli_test

2021-08-19.11-55-21.mp4

I added a new Option in the General Settings for the Playback, you can set it to hold the phonemes.
It seems that the output from Allosaurus expects that we hold the last phoneme during playback instead of inserting "rest" phonemes like before.
As you can see in that video this makes the playback look much more correct.
This fixes the problem @aziagiles mentioned:

In version 1.6.3 after conversion, the mouths change on a more normal and natural speed, while in version 1.6.4.1 after conversions, the mouths change extremely fast and not very natural.

(There is still some problems with the end not always being correctly recognized, but that seems to be a problem from Allosaurus itself.)

[steveway]

Ok, I apparently didn't yet push the mentioned Playback change.
It is now available in my CLI Test version.
I also fixed a bug in my phoneme helper tool which saved the phonemes incorrectly before and I fixed the phoneme conversions accordingly.
There might still be some improvements to be made but it's quite good now already.
Note that conversion is a lossy process, not all sets can be mapped perfectly and you will lose information when converting.

[Hunanbean]

It appears to be working as it should, but i have limited testing ability at the moment. Very cool, thank you!

[steveway]

Hello, I merged a few changes now since they seem to work pretty well.
The newest change also fixes a bug some of you experienced with Allosaurus.
Sometimes it seemed it did not recognize everything to the end.
The chunking to words I had produced a problem in some cases.
This should now be fixed and all the phonemes Allosaurus recognized should appear.

[aziagiles]

@morevnaproject Hello Konstantin. I really believe we have a pretty good lip sync software here (Papagayo-NG). The problem now is, for us Blender 2.93.x users, the LipSync Importer & Blinker addon doesn't work in the Blender 2.93.x version. Even this version upgraded by iCEE for Blender 2.8 (https://github.com/iCEE-HAM/io_import_lipSync-blender2.8?fbclid=IwAR32EEQyncGUDDbvpBzEu1Hr5V6J1SPNz7aVXRIeABZEQKSM32CVqspvy3A) doesn't work in version 2.93.x. Please, I really pray the addon be upgraded as it would be very helpful for us using it in major projects.

[Hunanbean]

I put in a request over on the MakeHuman forums to see if someone would update that plugin for us, but no responses. I took a look at the code, but i am no programmer.

I think is has to do with
`
Function Arguments

bpy.types.UILayout.column (align, heading, heading_ctxt, translate), was (align)

bpy.types.UILayout.row (align, heading, heading_ctxt, translate), was (align)

bpy.types.UILayout.template_shaderfx (), was (data)

`
But again, not sure. I will take another look today, more out of curiosity than of an expectation that i can fix it.

[Hunanbean]

Nope. Nothing i can do at this point. Please disregard what i thought it was.
EDIT: Nevermind, i fixed it.
Lines, 491, 496, 503 and 509, just change the = before EnumProperty to a :

Here is the fixed version:
Papagayo-NGLipsyncImporterForBlender Updated for Blender 2.93

[Hunanbean]

Sorry for this spam post. With all the hassle i had editing the previous message, i did not want @aziagiles to miss the fact it is fixed due to seeing my previous 'failure to do so' message.

[steveway]

That is pretty nice.
Yeah we need more up to date plugins.
I have an Issue here for that: #61
We should modify this script too to use the .json export data.
The .json files can save some more information, like the tags and it even includes a list of the used phonemes.
So before you walk through all the phonemes you can use that information to show the phonemes which will occur.
That way you don't necessarily need to create shapes/mouths for all phonemes.
I should also try to get the greasepencil importer there into a more usable state.

[aziagiles]

@Hunanbean Thank you. it works great.

[Hunanbean]

@aziagiles Glad i could help! Thanks for letting me know it is working for you. Be well!

[aziagiles]

@steveway I just downloaded the current master branch of Papagayo-NG Allosaurus github version but it failed to load. I guess there is a bug.

[steveway]

Mhh, that should work.
I'm using it right now.
I just merged a few little improvements from my branch.
Can you update and try again?
And if it fails to load, can you post the output? Best to start from a command line for that.

[aziagiles]

@steveway Ok. It works now.

[steveway]

Alright, since we now have a few plugins which can import Papagayo-NG Lipsync files to some programs, Blender and Krita for now, I made a little helper tool.
While testing I had to re-draw some mouths to see if the results work.
Especially at the beginning I had to look up what the phonemes were supposed to look like.
For that purpose I made a little tool based on the Pronunciation Dialog that shows you how the mouths look.
Just set the mouth image set, the phoneme set and click on the button for the phoneme you wish to know:

You can find the code here: https://github.com/steveway/papagayo-ng/blob/master/PhonemeVisualizer.py
I guess we will then include this with the installer as it's own .exe file for Windows and for other operating systems it has to be it's own separate program, Appimage for Linux and whatever it is macOS uses.

[steveway]

Apparently PyInstaller has something called "Multipackage Bundles".
https://pyinstaller.readthedocs.io/en/latest/spec-files.html#multipackage-bundles
With some modification to the .spec file we should be able to create multiple exe files for the main Papagayo-NG program, this Visualizer tool and maybe the Phoneme Conversion Helper Tool I made too.

[aziagiles]

@steveway Hello. I really like the progress you have been putting on the Papagayo-NG program. But this morning, when I did an automatic breakdown of my audio using Allosaurus breakdown, the last phoneme between words was held. I know it will be helpful to lots of people, but I prefer when they go back to rest position in the spaces between words just as in Papagayo-NG version 1.6.3. I thought if I uncheck the 'Hold Phonemes during playback' option in the Preferences/Settings window it will solve the problem, but it didn't.

[steveway]

@aziagiles Yes, I think it makes more sense during playback to only insert rest phonemes if we are not inside a word currently.
I've changed the logic a bit so it will behave like that during playback.
To note is that this only affects playback in Papagayo-NG, it does not change the data.
In my quick test it looks quite good, even with the automatic recognition by Allosaurus.
My Greasepencil and Krita Plugins have a checkbox for the rest frames after words and sentences.

[aziagiles]

@steveway You are right, it makes alot of sense the way it is already. My problem is to maybe fix the 'Hold Phonemes during playback' option, so users can choose. To be honest with you, both work well in different cases. As of now, whether one tick the 'Hold Phonemes during playback' option or not, it still holds the phonemes during playback.

[steveway]

I see, the setting was sometimes not loading correctly because QSettings does not handle bools correctly when loading from .ini files.
I fixed that.
I also changed this one setting and split it into two.
One will display the rest frame between every phoneme and the other will only display rest frames between words.

The descriptions of the setting is also a bit clearer now this way.

[aziagiles]

@steveway Thank you very much for the prompt fix. Hope you're having a nice day. I have a last recommendation for the Papagayo-NG software. It's still related to Holding back phonemes. I was wondering if the code could be modified in such a way that, a user can choose whether the phonemes should be held or not when exporting the file in .dat format or other formats. In the animation I'm currently working on, there are many cases I'll need them not to be held back. I know in your Grease Pencil importer addon, there is a place to check that, but for us using the Lip Sync Importer addon, we can't modify at that stage.

[aziagiles]

@steveway Just downloaded the recent master branch and thanks so much for considering the above recommendation. I believe the ''Show Rest Frames after Words'' function together with the ''Apply Rest Frame settings on Export'' function were actually it. But I think the ''Show Rest Frames after Phonemes'' function can be taken off as I don't believe anyone will ever use it.

[aziagiles]

@steveway Just tested the software again, and it seems like the ''Apply Rest Frame settings on Export'' function didn't work. I think it needs fixing.

[steveway]

Yes, it's not yet doing anything. 3dcc3b7
It's a little bit more difficult to add this to the export methods.
But I already have an idea how to add it without changing the code too much.

[steveway]

Alright, I've added some functionality.
For now it only adds rest frames between words on export to MOHO.
And it will only add the rest frame if there is space free in between.
The whole thing was a bit more confusing than it needed to be because it seems that MOHO starts at 1 while most other software starts at 0.
I'm not sure if we want to add this functionality to the other export options.
It's better to add some logic during import and keep the data unchanged.
Of course for MOHO that change makes sense since we can't update the importer into their software.
Well, we could ask them if they want to support the new JSON format for input and while they are at it they can add the logic to insert rest frames during import depending on user choice like my Blender and Krita Plugins do.
Maybe they will be responsive to this since Mike Clifton is back at the wheel of MOHO.

[aziagiles]

@steveway Hello Steve. I'm very happy with your last comment because it goes alongside my line of thought. I believe the 'Show Rest Frames after phonemes' should be taken off, and replace with something like 'Show Rest Frames after Sentences' whereby, when the words within a sentence are being said, phonemes are held back, but when it encounters a silence within the dialogue of say greater or equal to 8 frames (1/3 of a second), the Rest frame appears. And 'Show Rest Frames after Words' should just be the same type of phoneme breakdown as in Papagayo-NG version 1.6.3 and lower where phonemes are not held back. Below, is a test video of a lip sync exercise I did illustrating the concern.

0001-0891.mp4

[steveway]

@aziagiles That sounds like a useful feature.
There is silence detection in Pydub, I've tried before to split the sounds into words based on that.
But it's a bit fiddly, so I didn't get a good result yet.
I'll have to do some testing to see how we can combine the information we have from Allosaurus and Pydub to get something usable, that might take some time.

[aziagiles]

@steveway OK

[aziagiles]

@steveway I just downloaded the current master branch of Papagayo-NG Allosaurus github version but it failed to load. I guess there is a bug.

[aziagiles]

@steveway Ok. Let me try once again.

[aziagiles]

@steveway Adding it in Papagayo-NG alongside Allosaurus and Rhubarb will be a great idea. That will be awesome.

[aziagiles]

@Hunanbean Hello bro. I just downloaded the "Papagayo-NG Lipsync Importer For Blender" addon from your github page and notice, it doesn't work in Blender 3.5.x. Please can it be updated?

[aziagiles]

@Hunanbean I believe the problem comes as a result of modifications made on the Pose Library function in recent versions of Blender, and the addon script will thus needs some retouching in other to function properly taking this into account.

[Hunanbean]

@aziagiles Howdy! Shoot. Ok, i just tested it and am experiencing the same thing. I will talk to CGPT4 about it, because i am still no programmer :) I'll see what we can do

[aziagiles]

@Hunanbean Ok. will do just that and give you feedback.

[aziagiles]

@Hunanbean What should I fill in the space of pose library? or I should leave it empty.?The default may be to put "Current File" as that's where my mouth shapes are, but when I do, it doesn't work.

[Hunanbean]

@aziagiles Ok, the biggest issues is, i use Shape Keys which got fixed by the change in that line. I am not set up to even try Pose Libraries.. Is there a way you can send me a generic file with a pose library i can use to test with?

[aziagiles]

@Hunanbean receiving error messages. I'm on a Windows 64 bit computer.

[Hunanbean]

@aziagiles If you have some generic .Blend with a configured Pose library and a test model, i will keep trying to fix the code, but i do not even know how to make pose libraries yet. I am not sure if uploads can be done here, so you could send it to [email protected] if you have such a file

[aziagiles]

[email protected]

I've just replied you on your email box, with the blend file attached.

[aziagiles]

I've just replied you on your email box, with the blend file attached.

Ok, i just got it, thanks. I will see what i can do. This is going to take some time though, so probably will not have much until tomorrow or the day after as they are working on the power here, and it is scheduled to be down for several hours :/

Ok. Best of luck.

[aziagiles]

@Hunanbean Hello Hunanbean. I just sent you 2 emails. The second one is attached with another blend file containing mouth shapes I did in Blender 3.51 for testing of the lipsync addon. Please check.