Comments (9)
You’re right about this. We are pushing a number of fixes for HF server and clients soon. We’re just starting to play with them — expect many rough edges for a couple of weeks, but then should become something stable
from dspy.
Update: I just figured out the reason it is not working was somehow related to my usage of ngrok, if I have the the server and client running on the same machine it works fine. I will try to get it running through ngrok as well now.
from dspy.
Thanks for looking into this! This HF side of DSP is not very mature right now, so let us know if you face any issues. The FLAN models may not be ideal here based on our initial testing
from dspy.
Why are the FLAN models not ideal? Just because they lack the "emergent abilities" to really make use of DSP programs or some technical reason? I wasn't planning on using FLAN for my experiments but Llama and Open Assistant type models. So I am wondering if there is anything i should watch out for.
from dspy.
Yeah, you may see better results with decoder-only models, like the ones you mentioned, especially if you use closed LLMs as the demonstrator. We’ll share more soon
from dspy.
Very nice, I am excited to see you further progress.
(I hope "compiling" LoRa adapters on consumer GPUs is somewhere on your roadmap, since that would be a really exciting feature for me)
In my fork of DSP I have now adapted the hf_client.py a little bit so that you can pass an ngrok url to the object and it will just work with the hf_server.py.
This is very useful for me since I am serving the models in a HPC cluster an then ngrok them out to the public internet so I can access my models from any device so I think this may be useful for others as well :)
from dspy.
Yup we already have these LoRAs — @paridhimaheshwari2708 maybe we should merge something soon?
from dspy.
I noticed that when using hf_client that my kwargs are not sent to the hf_server. This is a problem when trying the self-consistency use cases. Thus in hf_client in the generate function I changed:
From:
payload = {"prompt": prompt, **kwargs}
To:
payload = {"prompt": prompt, "kwargs": kwargs}
This seemed to fix the problem.
Can you tell me the reason why the ** operator was there initially?
from dspy.
By the way things are much more stable now with TGI client in DSPy. Feel free to let us know if you face problems in the new version of the framework.
from dspy.
Related Issues (20)
- Chat support in Ollama impacting normal generation, something with `"content": "".join(text)` HOT 1
- Claude does not work with the new experimental setting. HOT 1
- ModuleNotFoundError: No module named 'dspy.datasets'; 'dspy' is not a package
- Question: BootstrapFewShot - Intermediate Label Bootstrapping HOT 1
- Signatures rarely work out of the box HOT 2
- OpenAI + ReAct : TypeError: unhashable type: 'list' HOT 1
- How to get multiple output from the dspy program? (e.g., two answers for a question)
- Deprecation warnings point to non-existing `dspy.set_log_level` HOT 1
- BuilderConfig doesn't have "trust_remote_code" key when loading HotPotQA HOT 1
- 'ReAct' object has no attribute 'best_programs'
- dspy.Retrieve cannot read csv file HOT 1
- dsp.settings to modify templates. Where can I find them? HOT 1
- List index error using MIPROv2 HOT 1
- Miprov2 doesn't work with TypedPredictors HOT 1
- import error: AttributeError: module 'threading' has no attribute '_Condition'. Did you mean: 'Condition'? HOT 1
- semver dependency error when setting up dev container HOT 2
- dspyComponent module only allows for a dspy.Predict module as input HOT 2
- `named_parameters()` and `named_submodules()` for dictionary attributes with non-string keys
- MIPROv2 - Error generating fewshot examples: name 'metric_threshold' is not defined HOT 1
- Debug in DSPy HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dspy.