anishfish2/kianix

Artificial Vtuber designed to stream on Twitch

Kianix - AI-Based Vtuber on Twitch

Welcome to the Kianix project repository! Kianix is an AI-based Virtual YouTuber (Vtuber) designed to stream on Twitch. The project's primary goal is to create a versatile Vtuber capable of playing different games, singing, and incorporating various functionalities over time during Twitch streams. Additionally, Kianix utilizes PineconeDB to implement external memory for enhanced interactivity.

Project Structure

Here's an overview of the project's structure:

• /games: This folder contains Reinforcement Learning (RL) implementations for games, including Mario and Dark Souls. The Dark Souls implementation was sourced from amacati on GitHub, and proper credit has been given.

• /backstory: This file contains the backstory and character details of the Vtuber model.

• /bark_out.wav: A temporary file containing a test from running the "bark" model on Hugging Face.

• /config.json: This configuration file stores important parameters, such as the memory value of the Vtuber model.

• /create_index.py: This script creates a PineconeDB index for memory storage.

• /currentAction.txt: Describes what the current model is doing during the stream.

• /delete_indexes.py: Clears the current PineconeDB databases.

• /full_query.py: Allows querying of the language model (LLM) being used.

• /functions.txt: Used to affect the Unity model running as the UI for the Vtuber.

• /initialize_kianix.py: Starts the PineconeDB for the Vtuber's memory.

• /insert_memory.py: Inserts a memory into Kianix's PineconeDB memory database.

• /keynotes.txt: Contains information you want to be stressed by the Vtuber during streams.

• /kianix_functions.py: The most important file, containing all the functionalities of the Vtuber, including generating jokes, responding to Twitch chat, playing games, and generating questions.

• /live2D.py: Interacts with the concurrently running Unity model.

• /plans.txt: Lists the Vtuber's daily plans and activities.

• /requirements.txt: Lists project dependencies.

• /reset_index.py: Resets the Kianix memory database.

• /shared.txt: Used for concurrent thread/script transfers, with plans to update to MongoDB.

• /texttospeech.py: Takes input and reads it out in the voice of the Vtuber.

• /twitchchatscrape.py: Continuously retrieves the Twitch chat for interaction during streams.

Future Plans

The project's future plans include:

1. Fixing file concurrency issues using MongoDB.

2. Polishing the current features and improving the Vtuber's voice.

3. Continuing to add more features and interactivity to enhance the streaming experience.

4. Testing out cutting edge research questions relating to dialogue, integrated memory, and AGI

Thank you for exploring the Kianix project repository! If you have any questions, suggestions, or would like to contribute, feel free to reach out.

Anish Karthik