microsoft/onnxruntime-unrealengine

Apply a Style Transfer Neural Network in real time with Unreal Engine 5 leveraging ONNX Runtime.

This code sample is to show you how to use the new Neural Network Inference (NNI) Plugin in Unreal Engine 5 which implements ONNX Runtime to allow you to add Machine Learning (ML) Models in your projects. ONNX Runtime is a library to optimize and accelerate machine learning inferencing.

We are using open source models from the ONNX model zoo to apply a style transform to the scene during game play. Check out the before and after pictures below to see how one of the models is able to stylize the scene.

Before:

After:

Statement of Purpose

This repository aims to grow the understanding of using ML models through the NNI Plugin in Unreal Engine 5 by providing an example of implementation and references to support the Microsoft Build conference in 2022. It is not intended to be a released product. Therefore, this repository is not for discussing ONNX Runtime, the NNI plugin or requesting new features.

Prerequisites

• Unreal 5

  • Check to make sure you have the minimum hardware and software requirements to run UE.
  • Installing UE5 can take hours depending on your network and hardware. Please see the UE docs for information on how to install UE5

• Visual Studio

  • Visual Studio 2019 v16.11.5 or later (recommended)
  • Visual Studio 2022
    • Include .Net Core 3.1

• Windows

• The initial version of the NNI plugin supports CPU inference on PC (Windows/Linux/Mac) and Consoles (PS5/Xbox Series X). At the moment, GPU evaluation is only supported for Windows DirectX 12.

• This project will only work up to version 5.2 of Unreal. Check out these docs for how to use the updated NNE plugin

How to run this project

Options 1: Run with Epic Games Launcher

• Open the Epic Game Launcher and launch Unreal 5

• Browse to the location that you cloned the project.
• Select the FPStyleTransfer.uproject file and click open. This will load the same project.

NOTE: If you receieve the below message click Fix Now and it will recompile the project.

• Once the project is loaded click play to see the project work!

Option 2: Open and run in Visual Studio.

• Browse to the location that you cloned the project.
• Open the FPStyleTransfer.sln file.

• Hit F5 to run the project.
• Click play to see the project work!

How to select a different model

There are 5 differnet models included from the ONNX Model Zoo. You can find them under the Content > Models folder in the UE Content Browser. To add your own models, drag and drop them into the Models folder in the Content Browser.

To select one of the other preloaded models or one that you added yourself.

• Select the StyleTransferConfig blueprint from the Outliner
• In the Details under the Default section find the Neural Network property and select the dropdown. You will see a list of models available.

• Select the model you would like to apply from the dropdown, and then click play to see the new style applied.

Resources

Grab open source models to add to your project:

• ONNX Model Zoo
• Huggingface Models

Connect with ONNX Runtime (ORT) on Social!

• Twitter
• YouTube
• Linkedin