aria-valuspa/avp

This is the ARIA-VALUSPA Platform, or AVP for short. Use this platform to build your own Virtual Humans with audio-visual input and output, language models for English, French, and German, emotional understanding, and many more. This work was funded by European Union Horizon 2020 research and innovation programme, grant agreement No 645378.

avp's Issues

DNS error

We've seen a DNS error from the Agent Core.
I need to investigate it.

• clone the project
  • git clone [email protected]:ARIA-VALUSPA/AVP.git && cd AVP && git checkout f39073e871b7b84cdaca8658c4735cd3b202df2b
  • run setup cd RunMeFirst\ && get_models.bat
• launch AMQ (start-1-...)
• launch the Agent-Core (start-3-...)
• observe the error
  • [Thread-0] ERROR e.a.d.behaviours.AlignmentGeneration - Timeout or unreachable or failed DNS lookup

Error could not parse pipeline on start-4-agent.bat

While starting start-4-Agent, got the following.
( Maybe i miss some parameters to be set before)

  [factory___] create instance of 'Camera'
  [options___] # WARNING # could not open file 'options/camera.option'
  location: ..\..\source\ioput\option\OptionList.cpp (202)
  [factory___] store instance of 'Camera' as 'noname006'
  [camera____] # WARNING # No Capture Device found

 location: ..\..\source\Camera.cpp (183)
 [framexml__] # WARNING # sensor: could not set output 'video'
 location: ..\..\source\XMLPipeline.cpp (793)
 [framexml__] # WARNING # could not load pipeline from file D:\ARIA\ARIA-System\Agent-
 Input\ssi\pipes\all-in-one-final\all-in-one-final.pipeline (r:37,c:5)
 location: ..\..\source\XMLPipeline.cpp (303)
 [__________] # !ERROR! # could not parse pipeline from 'D:\ARIA\ARIA-System\Agent-
 Input\ssi\pipes\all-in-one-final\all-in-one-final'
 location: ..\..\..\tools\xmlpipe\Main.cpp (296)

Anything to change in config ?
thanks

PS: This is not directly related, but i try also to launch

ARIA-System\Agent-Input\ssi> update_bin.bat,

but script is sticked to a E: drive

 SET SSI_BIN_DIR=E:\openssi\bin\x64\vc140

Encoding FML with ActiveMQ

Hello,

I have an issue with the connection with Greta.
I've written FML in french so there are special characters like "é"
If I use Greta FML File reader it works perfectly but if I build a scenario then the FML is sent through eu.aria.util.translator.api.ActiveMQConnector I think and I guess the encoding is wrong because "é" becomes something like "A(c)".

StandAlone ASR client with VAD ON， But the time mismatch

Hello, I have complie the asr part successfully. while I test my kaldi chain model on a long audiot with the standalone asr client with VAD turned on. I found that the start and end time of each audio segment does not match with their real time in the original audio. Is there any bug in the client?

Unity3D port

Would be good to have the Unity3D port in. For a release 3.0.1, shall we say?

Problems with Linux

When running Linux on the virtual machine I stuck with the network settings. According to https://github.com/ARIA-VALUSPA/AVP/tree/master/ASR there should be a rule defined which i cannot find. Can you tell me the parameters which i have to add there?

Another more general one: Is it possible to run the ASR only on windows, but also with cublas to speed up the whole process or is linux mandatory to provide a smooth running?

age, gender, and emotion not working

hi, i downloaded your repo and got it mostly working, after copying the launcher files like suggested in the other issue thread, but the age and emotion detection isn't working

it may be due to the following errors i've noticed in the window start-4-agent.bat was running in.

note, i have this running on my windows 7 laptop, with the ASR being done on a AWS Ubuntu box.

several of warnings like:

[trainer___] # WARNING # failed loading trainer from file
'../../models/speech/chunks.audio.compare.age.gold.linsvm.trainer' (r:0,c:0)
location: source\Trainer.cpp (1240)
[classifier] # WARNING # could not load trainer '../../models/speech/chunks.audio.compare.age.gold.linsvm'
location: ....\source\Classifier.cpp (192)
...
aEs0315 14:58:03.750545 9312 common.cpp:122] Cannot create Cublas handle. Cublas won't be available.
sEi0315 14:58:03.760546 9312 common.cpp:129] Cannot create Curand generator. Curand won't be available.
...
[trainer___] # WARNING # failed loading trainer from file '../../models/speech/close.mfccdd[-f 0.01 -d 0.015 -w 0.04].filler.gold[-c
5].lin[-s 0 -e 0.01 -B 0.1].final.trainer' (r:0,c:0)a
location: source\Trainer.cpp (1240)
l[classifier] # WARNING # could not load trainer '../../models/speech/close.mfccdd[-f 0.01 -d 0.015 -w 0.04].filler.gold[-c 5].lin[-s
0 -e 0.01 -B 0.1].final'
location: ....\source\Classifier.cpp (117)

note, i dont think these are related to the age, gender, and emotion not working, but there were a bunch of these warnings too.

WARNING: alice/body.mesh is an older format ([MeshSerializer_v1.40]); you should upgrade it as soon as possible using the OgreMeshUpgra
de tool.
WARNING: bone.mesh is an older format ([MeshSerializer_v1.40]); you should upgrade it as soon as possible using the OgreMeshUpgrade too
l.
Can't assign material bone to SubEntity of Reference-bone-0.9298522953614133 because this Material does not exist. Have you forgotten t
o define it in a .material script?

Missing folder DocumentationMaterial in repo

Wiki has several broken links like this one :
https://github.com/ARIA-VALUSPA/ARIA-System/raw/master/DocumentationMaterial/FMLTemplatesTags.png

these documents can help. Thanks to restore.

StandAlone ASR client fails if webrtcvad is installed (python 3)

Client run well as long as webrtcvad was not installed (with only the warning message about it.)
Once webrtcvad installed ( caution pip install webrtcvad needs a valid compiler on windows) , program fails line 116 : buffer += msg -> cannot convert 'bytes' object to str implicitly .
I saw that buffer is initialised as a string buffer = "" , while msg comes from the queue.

Note: Config is python3.5.2 as indicated in doc.

Incorrect/inconsistent documentation for ASR

Will there be any documentation for ASR on the Wiki?

That would be a valid question from a user ;)

Launcher agent won't start

start-3-core.bat starts well and show two windows for activeMqsender & receiver,
but start-4-agent.bat goes wrong with the following :

D:\ARIA\ARIA-System>start-4-agent.bat
Traceback (most recent call last):
 File "launcher.py", line 3, in <module>
 File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
 File "site-packages\wx\__init__.py", line 17, in <module>
 File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
 File "site-packages\wx\core.py", line 12, in <module>
ImportError: No module named 'wx._core'
Failed to execute script launcher     

I try adding wxpython through : pip install wxPython ( as administrator) and this goes well.
But still the same error with launcher.
(by the way, launcher only exists in exe, not in source code to debug)

multi_thread server

hello I was wondering if the sever support multi_thread, so it could handle more than one request at the same time

get_models missing some models?

Hi guys,
It looks like some models are missing from the get_models.
Specifically the files missing are: models\speech*.norm~
Cheers!

Documentation question

I have two issues when creating my own scenario:

1. I cannot find the files where the turn management is specified (e.g. when does the avatar wait for an answer and when not)
2. I cannot find the files where performative behaviour is specified (e.g. greeting)
   Is there a list for all possible "fml" and "bml"?
   thanks in advance

Remove update_bin.bat from public AVP

@frankenjoe said in #8 :
---8<---
regarding the 'update_bin.bat', this is only needed if you have a newer ssi version installed and want to update dlls. probably we should remove this from the public branch.
---8<---

I think I agree so let's do it.

ASR model not available

Hello,

Currently our team at INRIA (supervised by Prof. Justine Cassell) is trying to run this Platform. The site where the ARIA ASR model is stored seems to be unavailable - https://noxi.aria-agent.eu/models/2017-10-17/ARIA-ASR-English.tar.gz

It would be really helpful if we could get some information regarding the model.

Regards
Biswesh

Provide Unity3D documentation

As it says on the tin. Should go here: https://github.com/ARIA-VALUSPA/AVP/wiki/Documentation

There's a stub at the bottom of the documentation page.

DM threads not closing properly

There's a problem with closing the DM from the start-3-core.bat, if you close it by clicking the 'X', where not all Java processes are neatly ended. If you close the process by pressing 'CTRL + C', that's okay, but we should work on a better solution.

If you don't close it this way, a bunch of Java processes build up and you need to go to your Task Manager and end the Java processes manually.

Maybe @g-pechorin and I can look at how we can fix this?

Lipsync not working

In the public AVP, Greta appears to have no lipsync to speech. Any ideas what to do to solve it? :)
Slightly related / the greta LogWindow does not show any logging...

kaldi command

hi, it looks like your kaldi ASR commands ( online2-audio-nnet2-latgen-faster and online2-audio-nnet2-latgen-faster-gpu) are custom kaldi commands.

i wanted to do something similar, but with nnet3 chain models instead of nnet2 models. are you planning on adding support for nnet3 models or if not can you share the code you used to make your commands, so i can try to modify it for nnet3 models.

thanks