Comments (11)
Seems to do the trick with Aphrodite-engine ...
Oobabooga is still freaking out. It seems like a default template problem, but I'm honestly not sure. It doesn't give any particularly useful feedback when it pukes. On the plus side, it isn't crashing, now ... but it also isn't giving me much to work with. :-( Here's what I'm seeing, now: https://youtu.be/ZLUoX4YEjqk
I wish I had time to give you more of a hand with this. I have been in love with Twinny from Day One, and it has allowed me to start comparing responses from various coding models. The response time from Aphrodite, as you can see in the video above, is nearly instantaneous.
After a few days of troubleshooting, there is one thing I wish it had: A "reset to default settings" button, somewhere. ha ha ha
from twinny.
It indeed was. Super-weird. I wonder how that even got into my cut-and-paste buffer ...
Still not sure what needs to be done, here, but this is at least the right video. 😅
from twinny.
Hi thanks for the report. I don't personally use oobabooga API so having trouble to test it. Please submit a pull request with a fix for it if you can.
Many thanks,
from twinny.
I'm not sure I have time to learn where this lives, but I think I found (one of) the main issues. I was able to get working last night on the first shot of a new chat. After that, it fails (usually 400, bad request). A little more Wireshark later, I was able to find this in one of the payloads:
{
"role": "assistant",
"content": "<|im_start|>assistant\nHello! How can I assist you with your coding needs today?\n",
"type": "",
"language": {}
},
I think "language" needs to be a string, not an object. (i.e. "" not {} )
The logs say it isn't getting a "string" for that parameter. If I make this change in Postman, it works. Not sure where it would be outputting an object instead of a string, here, but maybe that helps you narrow it down?
from twinny.
Side note: It's technically a mimic of the OpenAI API, at this point. If you get it working, it should also work with VLLM, Aphrodite-engine, ChatGPT, and others. You could maybe get away with "OpenAI Compatible" for that dropdown ...
(In truth, I'm getting the log message from Aphrodite because Ooba just implodes when it gets that payload ...)
from twinny.
Hmm ok, thanks for that. I think we already have a PR to fix a similar issue. #159 which has requested changes, if it stays stale for much longer I will take care of it.
Many thanks for the report and detailed response/likely fix.
from twinny.
@zaqhack I just released version 3.8.9 which should address the issue of the non-compliant fields in the payload for the openai api spec, please could let me know if it helps.
Many thanks
from twinny.
Will add it just for you. By the way, the video link seems to go to the wrong link...
from twinny.
Ok, so I just got this working on my local instance. I've to pushed a new version so that ooba worked with /v1/completions by default now and updated the code to stream the data from the correct property path. Also, in ooba CMD_FLAGS.txt
I had to add --api
and --listen
flags for the API to work. After these things we're done it started to work. FYI ooba seems to be streaming junk completions to me but I'm not sure if it's the model/template im using. I don't use obba at all really so I am not sure, Ollama is just way better in my opinion.
from twinny.
Thanks!
If I can use a model with Aphrodite, I don't look elsewhere. Unfortunately, the acceleration it uses limits my choices a bit by not spanning video cards to allow greater VRAM total (it uses it for acceleration, not more model space). For Twinny, that's a mixed bag. It works well for Deepseek 7b, but I can't fit the bigger models onto one card with it ... for that, I need Kobold or Ooba or whatever. Is what it is. :-)
I should check out Ollama, I guess.
from twinny.
It should work now, closing.
from twinny.
Related Issues (20)
- feat: open new chat window in new editor tab HOT 1
- Edit and re-submit in chat mode HOT 4
- Robot icon keeps spinning, no inference HOT 1
- Code completion not working HOT 14
- Code completion works, but chat just spins the progress circle indefinitely HOT 2
- Configured providers but twinny not sending any requests to provider. HOT 3
- Multiline completion is confusing HOT 1
- Code snippets in the chat window loose syntax highlighting occasionally HOT 6
- FIM doesn't work with Keep Alive = -1 HOT 1
- Cannot read long model names when configuring provider HOT 1
- Option to save provider configuration to disk
- invalid option provided option="" HOT 4
- Codeqwen uses same FIM template as stable-code HOT 3
- Context Length Option With File Context Enabled Doesn't Limit Length HOT 1
- Chat workspace on right of visual code HOT 1
- Add OpenAI provider HOT 3
- FIM completion flexible context HOT 1
- Feature Request: Addition of a Visual Studio Plugin for Twinny HOT 1
- Sidebar shortcut/binding doesn t focus/unfocus properly
- newline shortcut in the prompt?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twinny.