Comments (3)
One of the issues here is that since we don't treat any node as a "master" or "worker" (all nodes are equal), the "head" and "tail" nodes are dynamic. However, the prompt needs to be sent to the "head" and the tokens are received by the "tail".
We could hack this by breaking the p2p equality assumption, but that would need to be fixed down the line.
The better thing is to:
- forward the prompt until you hit the "head" (this should be fairly simple).
- forward API requests to the "tail" that has all the generated tokens (this sounds hard and hacky). The other thing we can do is only serve the API from the "tail"
from exo.
Fixed the former: 1d5c28a
The latter is harder and for now, we'll only support API requests to the "tail". You can always force a node to be the "tail" by specifying its node-id to be sorted last e.g. with python3 main.py --node-id "xxx-node"
from exo.
Fixed in f2895cb
from exo.
Related Issues (20)
- the exo reasoning results are messy HOT 6
- An error module occurs when running llama3_distributed.py HOT 1
- How to use the downloaded local model HOT 7
- Wrong model referenced for Lllama-3.1 70B for tinygrad inference engine? HOT 1
- Exo not detecting Nvidia GPUs HOT 1
- Using LlamaIndex Openai like API to request exo via the ChatGPT API endpoint HOT 3
- [BOUNTY - $200] Bluetooth Networking Module HOT 1
- More robust networking
- Error occurred when running llama
- Question about Exo HOT 3
- [BOUNTY - $200] Support MLX community models in tinygrad inference engine
- The size of the network HOT 1
- [BOUNTY - $100] Parallelise Model Loading HOT 1
- tinygrad inference engine fails with BEAM=1 due to not running on main thread HOT 1
- Support for Reflection 70B model HOT 1
- The python version requirement has become a major problem for many devices that don't work. HOT 3
- Broken Links in README.md HOT 1
- iPod support
- [BOUNTY - $100] Pixtral Support
- HF_ENDPOINT support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from exo.