Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
Home Page: https://inference.readthedocs.io
License: Apache License 2.0
Several dependencies cannot be found when installing dependencies using pip install -e ., such as numpy, versioneer, and llama_cpp.
If there is no model that needs to be launched locally, the model needs to be downloaded first.
While downloading the model, the worker stopped executing.
For the API design, we can make it OpenAI compatible.
https://platform.openai.com/docs/api-reference/completions/create
orca 3b sometimes generates somethin like: [answer]###[unexpected tokens].
To avoid it, we may consider add builtin stop words for builtin models.
Whisper is an open-source model created by OpenAI.
The author of ggml provides a high-performance inference impl using ggml called whisper.cpp.
It is very cool to support serving whisper and combine it with LLMs. Here's a demo:
https://twitter.com/ggerganov/status/1642115206544343040?s=20
brew install portaudio
pip install sounddevice
pip install soundfileIn [1]: import sounddevice as sd
In [2]: myrecording = sd.rec(int(10 * 48000), samplerate=48000, channels=1)
In [3]: sd.play(myrecording)https://github.com/spatialaudio/python-sounddevice/blob/0.4.6/examples/rec_unlimited.py
whisper: https://github.com/openai/whisper
whisper.cpp: https://github.com/ggerganov/whisper.cpp
whisper.cpp python bindings: https://github.com/aarnphm/whispercpp
In [1]: import sounddevice as sd
In [2]: myrecording = sd.rec(int(10 * 48000), samplerate=48000, channels=1)
In [3]: sd.play(myrecording)
In [5]: from whispercpp import Whisper
...:
In [6]: w = Whisper.from_pretrained("tiny")
whisper_init_from_file_no_state: loading model from '/Users/jon/.local/share/whispercpp/ggml-tiny.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51865
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 384
whisper_model_load: n_text_head = 6
whisper_model_load: n_text_layer = 4
whisper_model_load: n_mels = 80
whisper_model_load: f16 = 1
whisper_model_load: type = 1
whisper_model_load: mem required = 127.00 MB (+ 3.00 MB per decoder)
whisper_model_load: adding 1608 extra tokens
whisper_model_load: model ctx = 73.58 MB
whisper_model_load: model size = 73.54 MB
whisper_init_state: kv self size = 2.62 MB
whisper_init_state: kv cross size = 8.79 MB
In [7]: w.transcribe(myrecording)
Out[7]: ' [sad music] [sad music] [sad music] [sad music]'A dashboard will provide the necessary monitoring and performance metrics, enabling efficient management and optimization of our system.
Below are the key features I envision for a dashboard:
Implement a short cut for users to launch a model locally with a single command, like:
plexar model launch -n <built-in model name>or
plexar model launch -p <path to custom model>It is hard to install llama-cpp-python with optimized cmake args currently. A better choice could be letting users install it.
To do that, we should not import llamacpp in global namespace, but inside the LlamaCppModel. ImportError should be captured and raise again with an installation guide.
We should try to run models on the fastest device available. Take Apple M1 as an example, we should leverage the power of metal as much as possible.
Traceback (most recent call last):
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict
output = await app.get_blocks().process_api(
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/gradio/blocks.py", line 1352, in process_api
result = await self.call_function(
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/gradio/blocks.py", line 1077, in call_function
prediction = await anyio.to_thread.run_sync(
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/Users/jon/Documents/miniconda3/envs/ml/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/Users/jon/Documents/repo/plexar/plexar/actor/gradio.py", line 141, in _refresh_models
return gr.Dropdown.update(value=launched[0], choices=launched)
IndexError: list index out of range
We can leverage https://github.com/li-plus/chatglm.cpp to serve chatglm-6b.
chatglm.cpp/convert.pyCurrently, plexar.model.llm.core.LlamaCppModel.generate takes a prompt as the input and returns a completion.
We can optimize it by leveraging the argument stream provided by llama_cpp.llama.Llama.__call__.
Implement a multi-model serving framework. The following roles are needed.
Controller manages workers, allocate resources and launches models on workers, and provides interfaces to users for interacting with models. It should contain the following components:
Execute operations on models according to controller's commands.
File "/Users/jon/Documents/repo/plexar/plexar/actor/model.py", line 56, in chat
raise AttributeError("chat")
AttributeError: [address=127.0.0.1:9999, pid=22149] chatNote that the issue tracker is NOT the place for general support.
When running the model_ref.generate() function in iPython, there seems to be a client created for every word generation, eventually leading to the following error:
gaierror: [Errno 8] nodename nor servname provided, or not known
python -m plexar.deploy.cmdline supervisor -a localhost:9999 --log-level debug
python -m plexar.deploy.cmdline worker --supervisor-address localhost:9999 -a localhost:10000 --log-level debug
import sys
from plexar.client import Client
client = Client("localhost:9999")
model_uid = client.launch_model("wizardlm-v1.0",7,"ggmlv3","q4_0")
model_ref = client.get_model(model_uid)
async for c in await model_ref.generate("Once upon a time, there was a very old computer.", {'max_tokens': 512}): sys.stdout.write(c['choices'][0]['text'])
First the warnings are printed: Actor caller has created too many clients ([some number] >= 100), the global router may not be set.
Then we have the gaierror after the [some number] exceeds 240.
Users may want to serve multiple replicas of a model.
Let users specify a model and the desired number of replicas, xinference will launch and manage them as a group. A load balancer needs to be created to distribute the incoming inference requests evenly across these model replicas.
TheBloke/orca_mini_13B-GGML
A worker should maintain a heartbeat connection with the controller. Each heartbeat should include information about the running models.
If the heartbeat is interrupted, the controller should cease scheduling models for that worker and label the models running on that worker as unavailable.
Once the heartbeat is restored, the controller should perform a health check using the information provided by the worker.
OpenLLM: https://github.com/bentoml/OpenLLM
FastChat pytorch impl: https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/model_worker.py
FastChat vllm impl: https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/vllm_worker.py
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.