bug: Cortex exited with code null immediately after loading a model

[mshpp]

bug: Cortex exited with code null immediately after loading a model

[vansangpfiev]

Thank you for reporting the issue.
Could you please provide more information, it would help us to debug:

• output for cat /proc/cpuinfo
• attach the app.log, it is in ~/jan/logs/ directory

[mshpp]

Sure, both files are attached
cpuinfo.txt
app.log

[vansangpfiev]

From the log and cpuinfo, I think Cortex crashed because we don't have avx build with fma flag off.
Could you please download a nightly build for cortex.llamacpp and replace the .so lib in the path: /home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-cpu/cortex-cpp/engines/cortex.llamacpp/
Download link:

https://github.com/janhq/cortex.llamacpp/releases/download/v0.1.18-25.06.24/cortex.llamacpp-0.1.18-25.06.24-linux-amd64-noavx.tar.gz

[mshpp]

This works, now I can get a response from the model. However, it seems that only the first round of inference works -- that is, I can only reliably get a single answer. The next one gets stuck on loading for a long time, and then it either completes or it doesn't and keeps loading. This is with the same TinyLLaMA model, which runs quite fast even on my hardware.

[vansangpfiev]

Thanks for trying the nightly build. Could you please share the app.log again?