mmehdimousavi/aiplayground

A virtual environment that allows changing isolated features in the image

AIPlayground: An Unreal Engine Based Data Ablation tool for Deep Learning

Image, GroundTruth, Prediction

Citation:

Paper

What Is Data Ablation?

Data Ablation is the study of data's effects on the performance of your network. Does your network get confused when it sees reflection in images? the only way to be sure is to test it. AIP enables you to test the effects of various aspects of data on a ML model, Image processing Algorithm or an AI Agent. Testing things like this is not feasible for natural scenes or crowd-sourced data. AIP will allow you to swap in different aspects of data without changing the other aspects, ie. isolating that exact feature that you are looking for. For example, roughness of a metal surface, reflectiveness of a tile, or the resolution of textures. With various settings for you to customize like lighting, fidelity (reflections, antialiasing, shadows, render scaling, etc), you will be able to generate vast datasets of otherwise identical images but with subtle differences to isolate and study the effects of different settings on your Models.

Watch demo video:

AIP:

A Virtual Environment for Various Machine Learning and/or a Dataset Generation tool for Image processing.

AIP Can be used to Generate Entire Datasets for various Machine Learning purposes. It supports Depth Estimation, Surface Normal Estimation and Pixelwise Segmentation for up to 256 Classes out of the box. AIP Features Keyboard macros and an interaction Module named "Probe" that can remember and reproduce the same Image over different circumstances. AIP's code can be used to make new environments while keeping the features for other purposes.

AIP uses customizable Post-processing for generating ground-truth annotations, making it flexible for use in characters and cameras, as well as global unbound post processing volumes.

How to use:

- Using the App:

Step 1: Download Probe
Step 2: Set up Probe in Python
Step 3: Download and Open AIPlayground.exe 
Step 4: Run Probe.

Probe Will now start gathering Images based on your setup specifications. Memory module will be saved in a folder called "Probe". If you need to reproduce the images, you need to make sure to save that file.

- Using the Source Code:

Step 1: Download AIP Core
Step 2: Import AIP Core Folders into any Unreal Engine Project
Step 3: Set up your Game Mode and Default Pawn to AIP
Step 4: Import the Input settings file provided in the AIP Core
Step 5: Run the Project
Step 6: Export your project or Run it in an independent window. Probe will recognize your Window handle and start sending commands to it. 

Probe Will now start gathering Images based on your setup specifications. Memory module will be saved in a folder called "Probe". If you need to reproduce the images, you need to make sure to save that file.

AIP Character Hotkeys:

- Segmentations Glass:

Press G for a Global Class Based Pixelwise labels. Press 0-9 and Ctrl+1 to Ctrl+4 to Access Binary Class-based Pixelwise Labels.

- Surface Normals Glass:

Press R to view the Pixelwise Surface normal in 6 Axis represented by 6 Unique Colors.

- Depth Glass:

Press T to view the prespective Depth (Distance of each pixel to the camera)

- Other Glasses:

E for Greyscale Colors H, J, K, L for Very High, High, Medium, Low Graphics B for Unlit Pass Y, U, I, O for switching between maps

Map:

aiplayground's Issues

ERROR No such file directory: 'ProbeLog\\look_rotate.txt'

When I run the Probe.py file for my Unreal Engine 5.2 project, I get the following error. How can I access the 'look_rotate.txt' file?
Thank you.

Executing source code

Hi, thanks for sharing this amazing work!

I'm trying to open your code on UE, but couldn't figure out how to access AIP's assets (such as rooms).
Could you please elaborate on how to run the code in UE? Thanks!

change camera height and camera parameter?

Hi, well done job!
I wonder it is possible to change the camera height and modify camera len paramters?