danielgross/localpilot

localpilot's Issues

503 Service Unavailable

Here is my User/setting.json in VSCode

{
  "editor.formatOnSave": true,
  "editor.formatOnType": true,
  "[typescriptreact]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "[typescript]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "[javascript]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "search.exclude": {
    "**/.git": true
  },
  "[json]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "editor.formatOnPaste": true,
  "[less]": {
    "editor.defaultFormatter": "esbenp.prettier-vscode"
  },
  "cmake.configureOnOpen": true,
  "github.copilot.advanced": {
    "debug.testOverrideProxyUrl": "http://localhost:5001",
    "debug.overrideProxyUrl": "http://localhost:5001"
  }
}

But when I select Mistral Model，it posts to /v1/engines/copilot-codex/completions and get 503 Service Unavailable

Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO:     Started server process [25920]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:root:Running: python3 -m llama_cpp.server --model /Users/song/models/mistral-7b-instruct-v0.1.Q5_K_M.gguf --n_gpu_layers 1 --n_ctx 4096
INFO:     127.0.0.1:61688 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: Mistral-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q5_K_M.gguf', 'type': 'local', 'filename': 'mistral-7b-instruct-v0.1.Q5_K_M.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/song/Codes/venv/localpilot/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/song/Codes/venv/localpilot/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/song/Codes/venv/localpilot/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/song/Codes/venv/localpilot/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='127.0.0.1' port=8888 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x103ea65f0>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 503, b'Service Unavailable', [(b'Connection', b'close'), (b'Proxy-Connection', b'close'), (b'Content-Length', b'8481'), (b'Content-Type', b'text/html;charset=utf-8')])
INFO:httpx:HTTP Request: POST http://localhost:8000/v1/engines/copilot-codex/completions "HTTP/1.1 503 Service Unavailable"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
INFO:     127.0.0.1:61693 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 503 Service Unavailable

Icon "disappears" in light/bright mode

Because the icon is white, when using light mode, it basically blends with the background. I changed the color so I can see it in any mode, bright or dark:

not working in vscode

I have added to settings.json(user), and restarted vscode

    "github.copilot.advanced": {
        "debug.testOverrideProxyUrl": "http://localhost:5001",
        "debug.overrideProxyUrl": "http://localhost:5001"
    }

but the plugin still reports needs an official subscription

New Feature Request: Support for Ubuntu OS

Hello Maintainers,

I recently came across your incredible package, "Use GitHub Copilot locally on your Macbook with one-click!".

I would love to be able to use this package in Ubuntu as well.

The current version of the package supports MacOS only. This feature request is to extend the support to Ubuntu. This would enable a wider range of developers to benefit from this package.

Windows compatibility

How this local copilot can be windows compatible?

TCP exception failure

I have a similar issue as #7, with the current requirements.txt.
I have an M1 Macbook Pro, ProductName: macOS, ProductVersion: 12.3.

DEBUG:root:Running: python3 -m llama_cpp.server --model /Users/daniel/models/mistral-7b-instruct-v0.1.Q5_K_M.gguf --n_ctx 4096
INFO:     127.0.0.1:64413 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: Mistral-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q5_K_M.gguf', 'type': 'local', 'filename': 'mistral-7b-instruct-v0.1.Q5_K_M.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/daniel/utils/localpilot/venv/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=8000 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))
INFO:     127.0.0.1:64424 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
anyio._backends._asyncio.ExceptionGroup: 2 exceptions were raised in the task group:
ERROR:    Exception in ASGI application
anyio._backends._asyncio.ExceptionGroup: 2 exceptions were raised in the task group:
----------------------------
Traceback (most recent call last):
  File "/Users/daniel/utils/localpilot/venv/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 167, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/Users/daniel/utils/localpilot/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1627, in connect_tcp
    await get_running_loop().create_connection(
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 61] Connect call failed ('::1', 8000, 0, 0)
----------------------------
Traceback (most recent call last):
  File "/Users/daniel/utils/localpilot/venv/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 167, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/Users/daniel/utils/localpilot/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1627, in connect_tcp
    await get_running_loop().create_connection(
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/opt/homebrew/Cellar/[email protected]/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 8000)

127.0.0.1:8000/docs seems to be working, though: I can query the Mistral model just fine there.

If the default model is local, the llama.cpp server doesn't run

Seems to just be because the run_server() call is called when using set_target(), but that's not called on startup. Slightly related to #5.

Error building wheel

This occurs after running these commands:

python3 -m venv venv

source venv/bin/activate

pip3 install -r requirements.txt

ERROR CODE:

ninja: build stopped: subcommand failed.

  *** CMake build failed
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for llama_cpp_python
Failed to build llama_cpp_python
ERROR: Could not build wheels for llama_cpp_python, which is required to install pyproject.toml-based projects

不知道怎么用

Model CodeLlama-7b found in /Users/zhangyixin/models.
Model Mistral-7b found in /Users/zhangyixin/models.
Model CodeLlama-34b not found in /Users/zhangyixin/models. Would you like to download it? (y/n) y
Downloading CodeLlama-34b from https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF/resolve/main/codellama-34b-instruct.Q4_K_M.gguf...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1165 100 1165 0 0 1437 0 --:--:-- --:--:-- --:--:-- 1441
5 18.8G 5 1081M 0 0 8771k 0 0:37:31 0:02:06 0:35:25 9.9M

5 18.8G 5 1089M 0 0 8769k 0 0:37:31 0:02:07 0:35:24 9573k
11 18.8G 11 2122M 0 0 6523k 0 0:50:26 0:05:33 0:44:53 3358k
11 18.8G 11 2126M 0 0 6511k 0 0:50:32 0:05:34 0:44:58 3393k

13 18.8G 13 2636M 0 0 5025k 0 1:05:28 0:08:57 0:56:31 2298k

100 18.8G 100 18.8G 0 0 2536k 0 2:09:43 2:09:43 --:--:-- 2221k
Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO: Started server process [76092]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)

^C% (base) zhangyixin@zhangyixins-MacBook-Pro localpilot % INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [76092]

(base) zhangyixin@zhangyixins-MacBook-Pro localpilot %
(base) zhangyixin@zhangyixins-MacBook-Pro localpilot % python app.py --setup

Model CodeLlama-7b found in /Users/zhangyixin/models.
Model Mistral-7b found in /Users/zhangyixin/models.
Model CodeLlama-34b found in /Users/zhangyixin/models.
Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO: Started server process [57594]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)

没有界面，

TCP exception failure

When GitHub Copilot extension is trying to establish a connection to the server, the following error occurs:

INFO:     Started server process [170892]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:root:Running: python3 -m llama_cpp.server --model /home/ubuntu/models/codellama-7b.Q5_K_S.gguf --n_gpu_layers 1 --n_ctx 4096
INFO:     127.0.0.1:46582 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: CodeLlama-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q5_K_S.gguf', 'type': 'local', 'filename': 'codellama-7b.Q5_K_S.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=8000 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))
INFO:     127.0.0.1:50468 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    await get_running_loop().create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 8000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/anyio/_core/_sockets.py", line 230, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/home/ubuntu/Desktop/localpilot/proxy.py", line 66, in proxy
    r = await client.post(url, data=data, headers=headers, timeout=30)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 365, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/Desktop/localpilot/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

New Feature Request: Support Windows

Are there any plans to support Windows?

only macos?

Support for Windows?

Please add support for WIndows!

How to use in JetBrains

1. Download https://plugins.jetbrains.com/plugin/17718-github-copilot/versions/stable and unzip
2. Search for overrideProxyUrl in agent.js
3. Change it and test (checking this now, if anyone has success please let me know...)

Do you need a subscription to Github Copilot for this to work?

Not obviously stated in the README and I don't see any error output in the CLI.

[INFO] [auth] [2023-10-04T14:22:44.353Z] Invalid copilot token: missing token: 403 
[ERROR] [default] [2023-10-04T14:22:44.354Z] Extension activation failed: "No access to GitHub Copilot found. You are currently logged in as [username]."

Needs to sign in in Github Copilote

In order to use the local models you need to have a Github copilote account, if you have any solutions please share it

vscode remote

I'm trying to host this on my Mac Studio I vscode remote into from all my laptops via ssh.

Could you add a blurb in the README how to set it up for remote vscode? I'm assuming there are additional steps, or it just works?

running a model other than CoPilot crashes

(fun) ➜  localpilot git:(main) ✗ python app.py
Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO:     Started server process [8436]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:root:Running: python3 -m llama_cpp.server --model /Users/deven367/models/codellama-7b.Q5_K_S.gguf --n_gpu_layers 1 --n_ctx 4096
INFO:     127.0.0.1:49968 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: CodeLlama-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q5_K_S.gguf', 'type': 'local', 'filename': 'codellama-7b.Q5_K_S.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=8000 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))
INFO:     127.0.0.1:49970 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
  | ExceptionGroup: multiple connection attempts failed (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('::1', 8000, 0, 0)
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 8000)
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 230, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/projects/public/localpilot/proxy.py", line 65, in proxy
    r = await client.post(url, data=data, headers=headers, timeout=30)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

Vim support

Any plans to support Vim?

Docker localpilot

As a localpilot user
I want a Docker container
So that I can run localpilot with a simple docker command rather than installing and running Python et al. from my local env

stay the prompts on the machine?

Hello,

I understand that you need a github account to run the thing and for authentication, but for me the question is if all the prompts stay on the machine and therefore also my own code?

Thanks for the project

Question about model choices

I'm still learning about running models locally. Could I ask how you decide which version of each model you will run?
I see different versions like Q5_K_S and Q4_K_M. I understand the main driver is memory when choosing between 7B, 13B, 34B, etc but how do you decide which quantization is right?

I'm on a 32GB M2 Max MacBook Pro.

Integration with Ollama

It would be cool if the script could access already downloaded ollama models, so you wouldn't have to install the same model twice.

requirements.txt needs macOS to build.

pip install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Collecting anyio==4.0.0 (from -r requirements.txt (line 1))
Obtaining dependency information for anyio==4.0.0 from https://files.pythonhosted.org/packages/36/55/ad4de788d84a630656ece71059665e01ca793c04294c463fd84132f40fe6/anyio-4.0.0-py3-none-any.whl.metadata
Using cached anyio-4.0.0-py3-none-any.whl.metadata (4.5 kB)
Collecting certifi==2023.7.22 (from -r requirements.txt (line 2))
Obtaining dependency information for certifi==2023.7.22 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.3.0 (from -r requirements.txt (line 3))
Obtaining dependency information for charset-normalizer==3.3.0 from https://files.pythonhosted.org/packages/5d/42/9d11cb5f3326501d7e913d2581a4e3b237ef7ee3e121b2faa89c1676125b/charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Using cached charset_normalizer-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (32 kB)
Collecting click==8.1.7 (from -r requirements.txt (line 4))
Obtaining dependency information for click==8.1.7 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: exceptiongroup==1.1.3 in /home/chris/.local/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.1.3)
Requirement already satisfied: h11==0.14.0 in /home/chris/.local/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (0.14.0)
Collecting httpcore==0.18.0 (from -r requirements.txt (line 7))
Obtaining dependency information for httpcore==0.18.0 from https://files.pythonhosted.org/packages/ac/97/724afbb7925339f6214bf1fdb5714d1a462690466832bf8fb3fd497649f1/httpcore-0.18.0-py3-none-any.whl.metadata
Using cached httpcore-0.18.0-py3-none-any.whl.metadata (18 kB)
Collecting httpx==0.25.0 (from -r requirements.txt (line 8))
Obtaining dependency information for httpx==0.25.0 from https://files.pythonhosted.org/packages/33/0d/d9ce469af019741c8999711d36b270ff992ceb1a0293f73f9f34fdf131e9/httpx-0.25.0-py3-none-any.whl.metadata
Using cached httpx-0.25.0-py3-none-any.whl.metadata (7.6 kB)
Requirement already satisfied: idna==3.4 in /home/chris/.local/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (3.4)
Collecting pyobjc-core==10.0 (from -r requirements.txt (line 10))
Using cached pyobjc-core-10.0.tar.gz (921 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
running egg_info
error: PyObjC requires macOS to build
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(localpilot) chris@FORGE:~/ai/localpilot$ pip install pyobjc-core
Defaulting to user installation because normal site-packages is not writeable
Collecting pyobjc-core
Using cached pyobjc-core-10.0.tar.gz (921 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
running egg_info
error: PyObjC requires macOS to build
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Adding compatibility with LLM Studio model structure

I've noticed on reddit, twitter, etc that a lot of people have started using local LLMs running on LLMStudio and others. I have a PR ready to change the file structure for how we save and upload locally saved models so one can directly use the models that are downloaded from LLMStudio rather than having to re-install it for localpilot.

Would appreciate if someone could grant me write access so I can make a PR in this repo.

Appreciate your work on localpilot!