edwios/freechat

Llama 2 macOS app

FreeChat

Chat with Meta’s Llama 2 on your MacBook without installing any other software. Every conversation is saved locally, all conversations happen offline.

• Customize persona and expertise by changing the system prompt
• Try any llama.cpp compatible model
• No internet connection required, all local
• Conversations sync natively via iCloud

h264.Screen.Recording.2023-09-13.at.1.54.49.PM.mov

Install

Download the TestFlight here: https://6032904148827.gumroad.com/l/freechat-beta

Mac App Store soon!

Or build from source via "Archive" in Xcode after completing dev setup below.

Goals

FreeChat is a native LLM appliance for macOS that runs completely locally. Download it and ask your LLM a question without doing any configuration. A local/llama version of OpenAI's chat without login or tracking. You should be able to install from the Mac App Store and use it immediatly.

• No config. Usable by people who haven't heard of models, prompts, or LLMs.
• Performance and simplicity over dev experience or features. Notes not Word, Swift not Elektron.
• Local first. Core functionality should not require an internet connection. There are lots of great clients for GPT-4, claude, etc. This is not one of them.
• No conversation tracking. Talk about whatever you want with FreeChat, just like Notes.

Upgrade your models

Once you're up and running, it's fun to try different models in FreeChat. The AI training community is releasing new models basically every day. FreeChat is compatible with any gguf formatted model that llama.cpp works with. Models can be found on HuggingFace. Most models have a "model card" by the author that discusses its training and abilities.

Models are usually named with their parameter count (e.g. 7B) and are formatted with different levels of lossy compression applied (quantization). The general rule of thumb is that models with more parameters tend to be slower and wiser and more quantization makes it dumber.

Here are a few we've tried and recommend:

Spicyboros

This is a fun Llama 2 based model for uncensored chat.

• Spicyboros-7B-2.2-GGUF
• Spicyboros-13B-2.2-GGUF
• Spicyboros-70B-2.2-GGUF

Code Llama

This is the latest set of Llama models from Meta. It is a version of their last foundation model, Llama 2, with additional training on coding tasks. It's good at programming but also has wider knowledge. Read the release note here.

• CodeLlama-7B-GGUF
• CodeLlama-13B-GGUF
• CodeLlama-34B-GGUF
• Llama-2-70B-chat-GGUF

Samantha

This is an interesting model that's inspired by the AI in the 2013 movie "Her", also named Samantha. It's based on Meta's foundation Llama models but has been trained on philosophy, psychology, and personal relationships. She's an assistant but also want to be a friend.

• Samantha-1.11-13B-GGUF
• Samantha-1.11-CodeLlama-34B-GGUF

Dev Setup

1. Download spicyboros-7b-2.2.Q3_K_S.gguf to FreeChat/Models/NPC
2. Open FreeChat.xcodeproj
3. Run and fix errors?

Roadmap / TODO (roughly in order):

• Chat with Llama 2 7B without installing anything else
• Try any llama.cpp compatible model
• Change system prompts to modify personas or expertise
• Download models from within the app (shrink app from 3GB to 10mb, way better for updates)
• Advanced settings (prompt format, temperature, repeat penalty)
• Personas - save system prompt / model settings for later and change personas when you create a new conversation
• Search conversations

Contributing

Contributions are very welcome. Let's make FreeChat simple and powerful.

Credits

This project would not be possible without the hard work of:

• Georgi Gerganov for llama.cpp
• Meta for training Llama 2
• Jon Durbin for training Spicyboros, the default model
• TheBloke (Tom Jobbins) for model quantization
• Monica Kogler for the FreeChat logo and uncountable UX consults

Also many thanks to Billy Rennekamp, Elliot Hursh, Tomás Savigliano, Judd Schoenholtz, Alex Farrill for invaluable spitballing sessions.