huggingface/unity-api

Hugging Face API for Unity 🤗

This Unity package provides an easy-to-use integration for the Hugging Face Inference API, allowing developers to access and use Hugging Face AI models within their Unity projects.

Table of Contents

1. Installation
2. Usage
   • Configuration
   • Example Scene
   • API Usage in Scripts
3. Tasks
4. Support

Installation

Via Git URL

1. Open the Unity project you want to add the package to.
2. Go to "Window" > "Package Manager" to open the Package Manager.
3. Click the "+" in the upper left hand corner and select "Add package from git URL".
4. Enter the URL of this repository and click "Add": https://github.com/huggingface/unity-api.git

Usage

Configuration

In the Hugging Face Website:

1. Generate an API key in https://huggingface.co/settings/tokens, we **advise you to create a Fine-Grained Token.

2. When the API key is created click on Set Permissions

3. Authorize Inference with this API key

After installation, the Hugging Face API wizard should open. If not, open it by clicking "Window" > "Hugging Face API Wizard".

4. Test the API key.

5. Optionally, update the endpoints to use different models.

Try our tutorial

To help you getting started, we wrote a tutorial where you create a robot agent that understands text orders and executes them.

The tutorial 👉 https://thomassimonini.substack.com/p/building-a-smart-robot-ai-using-hugging

The demo 👉 https://huggingface.co/spaces/ThomasSimonini/SmartRobot

Example Scene

To try the included example scene, follow these steps:

1. Click "Install Examples" in the Hugging Face API Wizard to copy the example files into your project.
2. Navigate to the "Hugging Face API" > "Examples" > "Scenes" folder in your project.
3. Open the "ConversationExample" scene.
4. If prompted by the TMP Importer, click "Import TMP Essentials".
5. Press "Play" to run the example. You should be able to use the UI to interact with the model.

API Usage in Scripts

The package includes a HuggingFaceAPI class that you can use from your scripts.

1. Import the HuggingFace.API namespace in your script.
2. Call the API method for the task you want.

using HuggingFace.API;

HuggingFaceAPI.TextToImage("a cat in a hat", result => {
    // Do something with the result, which in this case is a Texture2D
}, error => {
    // Handle errors
    Debug.LogError(error);
});

For a more advanced scripting example, refer to the included example scripts.

Tasks

Task	Status
Conversation	✅
Text Generation	✅
Text to Image	✅
Text Classification	✅
Zero Shot Text Classification	✅
Question Answering	✅
Translation	✅
Summarization	✅
Sentence Similarity	✅
Speech Recognition	✅

Support

If you encounter issues or have questions about the package, open an issue on the repository.

unity-api's Issues

Multiple models

Really good job!
Is it possible to use more than one model for each task simultaneously?
Thank you.

[FEATURE REQUEST] Run Models Locally

Hi,
Thanks for making the great tutorial (on Substack)!
Might it be possible to allow users to run models locally (instead of using the API), and bundle the models into their apps? This would be great for offline usage.
Thank you!

[FR] Please add release tags

Please create tagged releases for this package.

[BUG] Conversation API deprecation

Conversation API is now deprecated on the hub, since it's the same as Text Generation.

The Conversation task should be revised to be a wrapper around text-generation, to avoid breaking existing demos.

[Documentation] Getting started instructions and Access Tokens

The Usage > Configuration section in the README says that to generate an API key, people need to go to https://huggingface.co/settings/profile

However, there's no option to generate API keys there, the correct link to generate the access tokens is: https://huggingface.co/settings/tokens

On that note, I suggest you rename the API key (in the README and the Hugging Face API Wizard) to Access Tokens. There are also the SSH and GPG keys which might yield confusion.

Missing Image to Image and Text/Image to Image

For example... Controlnet

[BUG] Getting internal server errors when trying to connect to hugging face models

Describe the bug
Getting the following errors:
Attempted request to https://api-inference.huggingface.co/models/distilbert-base-cased-distilled-squad failed: HTTP/1.1 503 Service Unavailable - {"error":"Model distilbert-base-cased-distilled-squad is currently loading","estimated_time":20.0}
UnityEngine.Debug:LogWarning (object)
HuggingFace.API.APIClient/d__1:MoveNext () (at ./Library/PackageCache/com.huggingface.api@cb0acecbe2/Runtime/Implementations/APIClient.cs:73)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)

To Reproduce
Steps to reproduce the behavior:

1. Follow tutorial to install HuggingFace to Unity
2. Try to load the example code to use "HuggingFaceAPI.QuestionAnswering(inputText, OnSuccess, OnError, content);" function.
3. Set the timeout in the Huggingface setting window to be more than 3 seconds (it will timeout if you don't).
4. Click run in Unity editor
5. See error

Desktop (please complete the following information):

• OS: Windows 11
• Unity 2022.3.6f1

Additional context
Add any other context about the problem here.

I need to specify text or text_target in text classification

I try calling the api by huggingfaceapi.textclassification("some string", response =>...) but got the error"you need to specify text or text_target". Where can I specify that in my unity C# code?

will not run on unity 2022 lts or 2022.3.0f1

Describe the bug
Whenever I try to run the example scenes the following error occurs:

"NullReferenceException: Object reference not set to an instance of an object
HuggingFace.API.Editor.HuggingFaceAPIWizard.OnGUI () (at ./Library/PackageCache/com.huggingface.api@cb0acecbe2/Editor/HuggingFaceAPIWizard.cs:60)
UnityEditor.HostView.InvokeOnGUI (UnityEngine.Rect onGUIPosition) (at /Users/bokken/build/output/unity/unity/Editor/Mono/HostView.cs:512)
UnityEditor.DockArea.DrawView (UnityEngine.Rect dockAreaRect) (at /Users/bokken/build/output/unity/unity/Editor/Mono/GUI/DockArea.cs:386)
UnityEditor.DockArea.OldOnGUI () (at /Users/bokken/build/output/unity/unity/Editor/Mono/GUI/DockArea.cs:377)
UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at /Users/bokken/build/output/unity/unity/ModuleOverrides/com.unity.ui/Core/IMGUIContainer.cs:355)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/GUIUtility.cs:190)"

Looks like the hugging face api for unity only works on Unity 2020.3.48f1? But I have to work on 2022 LTS release...

iMac Intel with Ventura 13.6

Thanks for any help!

How to download the model to the local call API

Because my internet connection is not very good, I would like to download the model to my local machine and use the Hugging Face API for calling. How can I achieve this?

TextToImage Queries

1.) How to pass various parameters like a seed, output resolution, etc?
2.) Caching of responses, is this being worked upon?
3.) Changing the endpoint to https://api-inference.huggingface.co/models/stabilityai/stable-diffusion-2-1 and hitting generate produces the same result for the same string always.

Android support

Great repo! My question is - does it work on Android?

I did some research but couldn't find much - except for some comments on YouTube that speech recognition doesn't really work on Android ("when i export to an a Android Device the text always is "you", no matter what did i say. I don't know if needs another configuration because in the unity editor works fine").

Could you please clarify?

Thank you!

[FEATURE REQUEST] Implement AudioClassification Task for CLAP Models

Is your feature request related to a problem? Please describe.
Currently, the Unity API for Hugging Face does not support the AudioClassification task, particularly for CLAP models. This limits the potential for developing interactive applications and games that can utilize audio recognition capabilities in Unity.

Describe the solution you'd like
I would like to see the implementation of the AudioClassification task in the Hugging Face Unity API, with support for CLAP models, especially the 'laion/clap-htsat-unfused' model (https://huggingface.co/laion/clap-htsat-unfused). This would enable Unity developers to integrate advanced audio classification features into their applications, enhancing interactivity and user engagement.

Additional context
The CLAP model (https://huggingface.co/docs/transformers/model_doc/clap) has significant potential for applications in game development and interactive experiences where audio input can trigger specific actions or responses. Implementing this feature in the Unity API would greatly benefit developers looking to explore innovative audio-based interactions in their projects.

Specify language in Speech Recognition

[Bug] Package should not throw error for missing config

When loading the package for the first time it should not throw an error about missing config file.

Instead, either pop up a dialog or change the log from error to warning.

In either case, giving the user time to create/configure should be done before presenting with error message.

[BUG] The referenced script (Unknown) on this Behaviour is missing!

Hi,

I am using Unity3D 2022.3.14.f1 and I get following warning which leads to an error:

The referenced script (Unknown) on this Behaviour is missing!

I am getting this error in each of the 3 examples. The examples are not working.

Thanks

Martin

Increase default max timeout value

TextToImageExample is failing with the current setting increasing the timeout fixes the generation.

[FEATURE REQUEST] Update default endpoints

The default endpoints are now very old, and should be updated to something more modern

[Bug] Missing assembly definitions for examples

Script 'Packages/com.huggingface.api/Examples/Scripts/ConversationExample.cs' will not be compiled because it exists outside the Assets folder and does not to belong to any assembly definition file.
Script 'Packages/com.huggingface.api/Examples/Scripts/TextToImageExample.cs' will not be compiled because it exists outside the Assets folder and does not to belong to any assembly definition file.

[BUG] Task AutomaticSpeechRecognition not found.

Describe the bug
I have successfully integrated the API in to the unity editor. When I get a release build on IOS, I am receiving the error "Task AutomaticSpeechRecognition not found."

To Reproduce
Steps to reproduce the behavior:
1- Run on iOS device.

• OS: iPhone 13 Pro running on iOS 16

HuggingFaceAPI.AutomaticSpeechRecognition(_bytesRecorded, response => {
               Debug.Log($"Message is = {response}");
               voiceChatPrompt.Show(response);
               IsWaitingApiResult = false;
           }, error => {
              // Task AutomaticSpeechRecognition not found
               Debug.LogError(error);
               IsWaitingApiResult = false;
           });

Future improvements

This issue lists the future improvements we can make based on user's feedback:
✅ #4

• Set parameters and expose them via config file.
• Specifying language for compatible models.