kevinhughes27/tensorkart

self-driving MarioKart with TensorFlow

TensorKart

self-driving MarioKart with TensorFlow

Driving a new (untrained) section of the Royal Raceway:

Driving Luigi Raceway:

The model was trained with:

• 4 races on Luigi Raceway
• 2 races on Kalimari Desert
• 2 races on Mario Raceway

With even a small training set the model is sometimes able to generalize to a new track (Royal Raceway seen above).

Dependencies

• python and pip then run pip install -r requirements.txt
• mupen64plus (install via apt-get)

Recording Samples

1. Start your emulator program (mupen64plus) and run Mario Kart 64
2. Make sure you have a joystick connected and that mupen64plus is using the sdl input plugin
3. Run record.py
4. Make sure the graph responds to joystick input.
5. Position the emulator window so that the image is captured by the program (top left corner)
6. Press record and play through a level. You can trim some images off the front and back of the data you collect afterwards (by removing lines in data.csv).

Notes

• the GUI will stop updating while recording to avoid any slow downs.
• double check the samples, sometimes the screenshot is the desktop instead. Remove the appropriate lines from the data.csv file

Viewing Samples

Run python utils.py viewer samples/luigi_raceway to view the samples

Preparing Training Data

Run python utils.py prepare samples/* with an array of sample directories to build an X and y matrix for training. (zsh will expand samples/* to all the directories. Passing a glob directly also works)

X is a 3-Dimensional array of images

y is the expected joystick ouput as an array:

  [0] joystick x axis
  [1] joystick y axis
  [2] button a
  [3] button b
  [4] button rb

Training

The train.py program will train a model using Google's TensorFlow framework and cuDNN for GPU acceleration. Training can take a while (~1 hour) depending on how much data you are training with and your system specs. The program will save the best model from all epochs of training to disk when it is done.

Play

The play.py program will use the gym-mupen64plus environment to execute the trained agent against the MarioKart environment. The environment will provide the screenshots of the emulator. These images will be sent to the model to acquire the joystick command to send. The AI joystick commands can be overridden by holding the 'LB' button on the controller.

Future Work / Ideas:

• Add a reinforcement layer based on lap time or other metrics so that the AI can start to teach itself now that it has a baseline. The environment currently provides a reward signal of -1 per time-step, which gives the AI agent a metric to calculate its performance during each race (episode), the goal being to maximize reward and therefore, minimize overall race duration.
• Could also have a shadow mode where the AI just draws out what it would do rather than sending actions. A real self driving car would have this and use it a lot before letting it take the wheel.
• Deep learning is all about data; perhaps a community could form around collecting a large amount of data and pushing the performance of this AI.

Related Projects:

Xbox Game AI - Uses PYXInput for direct control of any Xbox/PC game.

SerpentAI - Game Agent Framework to create AIs for any game.

Donkey Gym - OpenAI Gym Environments for self-driving "Donkey Car".

AirSim - An Unreal Engine simulator for autonoumous vehicles.

Special Thanks To

• https://github.com/SullyChen/Autopilot-TensorFlow

Contributing

Open a PR! I promise I am friendly :)

tensorkart's Issues

load_imgs function missing

Hello,

The load_imgs function, used in 6ab288d, is missing from the codebase :)

Thanks for this awesome project!

Add start button to manual override

Its annoying because I can't quite in the middle of a level when using the bot driver

Change model input to be single channel

The color information is likely useless. I am not sure why they bother to use it in the nvidia paper.

First screenshot png repeated

I installed all the dependencies and everything works fine, except the recording: I get always the same initial screenshot png repeated over and over. Joystick inputs are displayed in the graph and recorded correctly.
Any suggestions?

OS: Ubuntu 16.04

Update record.py to use gym

Now that the main play.py program uses the gym environment, record.py should be updated to use this framework as well. This will simplify the setup/recording process and should allow for more consistency between the two aspects of execution.

ctrl-c doesn't stop play.py

I always have to use ctrl z to put it in the background and use kill -9 _pid_. I am wondering if this is related to the code or my system (I was messing with a bunch of things when I wrote this to try and dpo fake joysticks).

@bzier @snawara does this happen to either of you?

Update README with new dependencies

Now that we've integrated with gym-mupen64plus, the README should be updated to reflect this dependency.

play.py: Joystick not responding since installing gym-mupen64plus

Hello, I think I am in the final step :)
I installed mupen64plus, my joystick, everything went smoothly. I run record.py, joystick inputs responding. I run play.py, I have the two screens opened but when the race starts, nothing, mario do not move. And I realized that my joystick in no more responding since I followed the installation of gym-mupen64plus.
If I start from scratch mupen64plus's install, it works. Then, when I install gym-mupen64plus, joystick doesn't work. The joystick is still recognized and get the right signals in record.py, when I test it through jstest, it is ok etc ... I think there is something modifying mupen64plus.
Can you please help me ? I am in the final step :))

Port to Windows/Mac if possible (not really an issue, but it limits usability of this bot) (also merely trying to run it on Windows)

I'm wanting to try to run this bot, but the only problem is I'm on Windows, & not Linux like what the bot seems to run. Will the bot run anyway, even though I'm not on Linux? If not, then I'll have to whip something up to make it work.

how can I download the game for mac?

Errors with new gym-based play.py

When I try to use play.py to see how my trained model performs, I get errors. Due to that final OSError, I thought it might need to be ran as root, but that didn't help either. Does this fit more as an issue with gym-mupen64plus @bzier ?

[2017-04-24 17:47:35,617] Making new env: Mario-Kart-Luigi-Raceway-v0
('ControllerHTTPServer started on port ', 8082)
initial_disp: :0
Starting xvfb with command: ['Xvfb', ':1', '-screen', '0', '640x480x24', '-fbdir', '/dev/shm']
Changed DISPLAY to: :1
Starting emulator with comand: ['vglrun', 'mupen64plus', '--resolution', '640x480', '--input', '/usr/local/lib/mupen64plus/mupen64plus-input-bot.so', '/home/marty/Development/gym-mupen64plus/gym_mupen64plus/ROMs/marioKart.n64']
Traceback (most recent call last):
  File "play.py", line 67, in <module>
    env = gym.make('Mario-Kart-Luigi-Raceway-v0')
  File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 161, in make
    return registry.make(id)
  File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 119, in make
    env = spec.make()
  File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 86, in make
    env = cls(**self._kwargs)
  File "/home/marty/Development/gym-mupen64plus/gym_mupen64plus/envs/MarioKart64/mario_kart_env.py", line 32, in __init__
    super(MarioKartEnv, self).__init__(mk_config['ROM_NAME'])
  File "/home/marty/Development/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 64, in __init__
    self.xvfb_process, self.emulator_process = self._start_emulator(rom_name=rom_name)
  File "/home/marty/Development/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 226, in _start_emulator
    stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Close called!

Agent Output

I guess this is not a technical issue, but maybe it is, who knows 😄 So after training the agent on 10 races, and then running through play.py, my agent outputs only accelerate command ([0 0 1 0 0]). Manual override works well. Data CSVs have all inputs recorded properly, and all images are from the game. I train for 200 epochs ( around 8k data samples) and loss goes down to around 0.3, while eval loss hovers around 0.5. Btw, you should track eval accuracy, not loss, it would be more informative.

Does it look like a technical error, or just bad agent? During the data recording, accelerate command is always constant (1), so maybe it just learned to output that all the time...

Use PIL to take screenshots instead of wx

Skip to bottom for final suggestion:

While looking through your project and trying it out myself, I had problems running it on Windows.

The error was as follows
Traceback (most recent call last): File "A:/github/mBeierl/TensorKart/record.py", line 207, in <module> app.frame = MainWindow() File "A:/github/mBeierl/TensorKart/record.py", line 30, in __init__ self.create_main_panel() File "A:/github/mBeierl/TensorKart/record.py", line 49, in create_main_panel img = wx.Image(320,240) File "A:\Programme\Anaconda2\lib\site-packages\wx-3.0-msw\wx\_core.py", line 2882, in __init__ _core_.Image_swiginit(self,_core_.new_Image(*args, **kwargs)) TypeError: String or Unicode type required

I guess it may be because I don't have wx version 3.0.3. the only version available for windows seems to be 3.0.
First: Is that correct or is there a 3.0.3 version out there that I missed?

Second: I then started to fix those small issues
e.g. in record.py line 49

img = wx.Image(320,240) 
 self.image_widget = wx.StaticBitmap(self.img_panel, wx.ID_ANY, wx.Bitmap(img))

to

img = wx.EmptyBitmap(320,240) 
self.image_widget = wx.StaticBitmap(self.img_panel, wx.ID_ANY, img)

Then it worked without errors, but only black screenshots were created.

Finally, I tried PIL (http://effbot.org/imagingbook/) and was able to create a working screenshot with

import psyscreenshot as ImageGrab
im = ImageGrab.grab(bbox=(2,52,320,290) # worked best with emulator window x1
im.save("test.png")

Do you think it would be worth the effort to change the screenshot system to use this library instead? I have no idea how good the performance is with this library. Will test this out and update this issue with my progress.

Cheers and thanks for this great project!

Can't play emulator and record.py together

Hi, I've been trying to configure a Switch Pro Controller to work with this project, so I'm using pygame_sdl2.controller instead of inputs. I got the Python UI working, but when I run record.py after running the emulator, the emulator can't detect the controller anymore. I'm wondering if you encountered this problem at all, or if it has to do with the change in packages.

Robust Screenshots

The current screenshot engine is not very robust. It works on a recent install of Ubuntu or Linux Mint but numerous issues have been reported on other systems:

• On MacOS people have reported the screenshots are always black: #15
• Someone reported that the screenshots are repeated on Ubuntu 16.04 and an nvidia card #27
• I opened an issue about exploring using GTK for screenshots #33 in hopes that it might be more robust and remove the wx dependency
• Numerous requests for windows and cross platform support which does not work with the current setup as far as I know

SerpentAI could be a good reference for the way to do this. They have a nice more fully fleshed out architecture for screen capture and claim to be cross platform:

• GameFrame
• FrameGrabber

For the actual screenshots they use mss

Black screen capture on mac

Crudely changed take screenshot in utils.py as per http://wxpython-users.1045709.n5.nabble.com/ScreenDC-wxMac-td2358205.html:

import os
def take_screenshot():
os.system('screencapture scr.png')
screen = wx.Bitmap('scr.png')
return screen.GetSubBitmap(wx.Rect(10,60,615,480))

edit: removed unused lines. Also not quite sure how yet, but need to remove alpha from screenshot. e.g. to remove in prepare samples: imread(image_file)[:,:,:3] however, this leaves the play.py with an alpha layer since it uses take_screenshot() as well

Screenshots Black Screen on Ubuntu 14

Hello,
What would be the reasons to have black screenshots when using record.py ?
Thanks,
Xavier

Aspect ratio of processed images not equal to raw images

I noticed that the Sample class has shape (200, 66, 3) while the raw images have an aspect of (640, 480, 3). Why is there a difference in aspect ratio?

Basic question: play a demo

Greetings,

I wanted to see some demo running and how the system runs and tried to run python test.py without any inference from a model (fixing joystick = [0, 0, 1, 0, 0]).
At first, it seemed well, but finally I got the problem below without seeing the agent moving around.
It must be a trivial issue, but I'd like you to help me solve the problem.

Thanks.

can't drive, no direction event

Input Warning: Couldn't open rumble support for joystick #1
Input Warning: Couldn't open rumble support for joystick #2
Input Warning: Couldn't open rumble support for joystick #3
Input Warning: Couldn't open rumble support for joystick #4

Can we share some pre-trained weights?

Amazing project I can't appreciate more of the fun it brought me, it'd be great if you can share some pre-trained weights

Data class holds all samples in memory

This won't scale much longer since if I add any more training data it won't all fit in memory. In this project they have a list of all the training images and only load what is required for each training batch.

My current way might be a bit quicker to train since the IO is front loaded but eventually something like this might be required.

Screenshots with gtk?

Someone posted this link on HN https://www.pastery.net/xafjmn/ (from this article https://news.ycombinator.com/item?id=14145551)

Its a GTK screen cap that might work better.

module wx not found.

Not able to record data

Hi
I am trying to generate training data on another game and when I run record.py I get the following error.

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\Tushar\miniconda3\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "record.py", line 162, in on_btn_record
    self.start_recording()
  File "record.py", line 207, in start_recording
    os.mkdir(self.outputDir)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:/Users/Tushar/Desktop/github/TensorKart/samples/2020-05-16_13:53:22'

Model Trained: Straight Line

Hello,

I was trying run the trained model and I thought that maybe the player was going in straight line due to lack of training nevertheless after generating the model with samples for around 6 races the behavior is still the same, any ideas about what the issue might be? For example the easiest track is Luigis racetrack and as soon as the player gets to the first curve the model is not reacting as expeted I think. I am not seeing the values between -80 and 80. Is worth to mention that I did double check the Joystick samples and those are ok going from -1 to 1 according to the record.py file.

Best Regards!

Unable to access the X Display, is $DISPLAY set properly?

Hi all- very, very new to the world of programming languages; I've just started using Ubuntu for my work as I need it to run a python program (FSLeyes, for neuroimaging) on my Windows laptop (Dell Latitude E6330). After much trial and tribulation, it actually worked - running 'fsleyes' in the command line ran the program. However, on subsequent attempts, it gives me the 'Unable to access the X Display, is $DISPLAY set properly?' message when I attempt the same command. Would anyone know what might be going on, and how to fix it? Thanks so much!

No such file or directory: 'samples/luigi_raceway/data.csv'

The folder sample is empty ...

Unable to access the X Display, is $DISPLAY set properly?

Hello,

I have recorded my game (luigi_raceway) and trained the data.csv to generate the model_weights.h5 file.

When I run "python play.py", it shows the followings:

Using TensorFlow backend.
[2017-10-09 07:48:53,027] Making new env: Mario-Kart-Royal-Raceway-v0
('ControllerHTTPServer started on port ', 8082)
initial_disp: :1
Starting xvfb with command: ['Xvfb', ':1', '-screen', '0', '640x480x24', '-fbdir', '/dev/shm']
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
(EE) 
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) 
Changed DISPLAY to: :1
Starting emulator with comand: ['vglrun', 'mupen64plus', '--resolution', '640x480', '--input', '/usr/local/lib/mupen64plus/mupen64plus-input-bot.so', '/home/exe/gym-mupen64plus/gym_mupen64plus/ROMs/marioKart.n64']
Calling wx.App() with DISPLAY: :1
Unable to access the X Display, is $DISPLAY set properly?
Close called!

I have searched the Google, but can not find correct answers.

How can I get access to X Display ?
Thanks a lot.

Use python-inputs instead of pygame

https://pypi.python.org/pypi/inputs.

Why? - Its a much lighter dependency than pygame.

Please add a license to the repo

Could you please add an explicit LICENSE file to the repo so that it's clear under what terms the content is provided, and under what terms user contributions are licensed?

Per GitHub docs on licensing:

Generally speaking, the absence of a license means that the default copyright laws apply. This means that you retain all rights to your source code and that nobody else may reproduce, distribute, or create derivative works from your work. This might not be what you intend.

Thanks!

record.py dont recive joystick signal.

i install all dependencies, test the joystick and work fine but when i test record.py, it dont get the joystick signals.

pd: dont create screenshots in record mode.
SO: ubuntu 16.04

Image size question

@kevinhughes27 Just curious about the sizes used in utils.py. On my machine, the default mupen64plus window is 640x480, yet you're using a rectangle of 615x480. Also, in prepare_image() you are resizing the image to 200x66. Can you help me understand why these sizes are being used in this way? Also, why the call to reshape the image array prior to resizing it? Thanks.

Add the Wii version of Mario Kart

• Dolphin is an excellent emulator so this should be doable
• The Wii version has the steering wheel so I could actually do the steering angle as my output.

Problems running TensorKart (./record.py) with MuPen64Plus!

Hi!

After installing all the requirements and dependencies, I ran the following command along with an instance of MuPen64Plus running Mario Kart 64.

Input:

./record.py

This is what happened...

Output:

iMac:TensorKart shyamalsuhanachandra$ ./record.py
2017-01-19 19:37:50.129 Python[14589:68221] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to /var/folders/m1/b_t9_2151y30ryvtr_2gznch0000gp/T/org.python.python.savedState
Traceback (most recent call last):
  File "./record.py", line 207, in <module>
    app.frame = MainWindow()
  File "./record.py", line 30, in __init__
    self.create_main_panel()
  File "./record.py", line 49, in create_main_panel
    img = wx.Image(320,240)
  File "/usr/local/lib/python2.7/site-packages/wx-3.0-osx_cocoa/wx/_core.py", line 2882, in __init__
    _core_.Image_swiginit(self,_core_.new_Image(*args, **kwargs))
TypeError: String or Unicode type required

How can I fix this problem?

Note: I didn't make any changes to the code besides install the dependencies with brew instead of pip in certain scenarios. Also, the screen in MuPen64Plus flickers quite a bit, what should I do to prevent this from happening?

Assertion error running play.py

Hello Kevin, I ran into your project when I was digging about Tensorflow so I decided to give it a try and see how it works. After some changes (bold text) suggested in previous comments. I was able to record, prepare samples and finally try to play the trained agent but now I am getting an error so maybe you might know whats happening. Here the changes that I have added to my environment some screenshots and a video.

If i dont use EmptyImage and EmptyBitmap wx does not work.
record.py

Images

    img = wx.**EmptyImage**(320,240)
    self.image_widget = wx.StaticBitmap(self.img_panel, wx.ID_ANY, wx.**EmptyBitmap**(320,240))
    bmp = wx.**EmptyBitmap**(Screenshot.SRC_W, Screenshot.SRC_H)

I had to disable XVFB from the config file otherwise I am not seeing anything on my desktop, for some reason it is not working for me.
config.yml (gym_mupen64plus)

XVFB config:

USE_XVFB: false
XVFB_CMD: Xvfb
TMP_DIR: /dev/shm
VGLRUN_CMD: vglrun

Finally when I run the play.py I am getting an assertion error.

I am not sure the record is working as expected because I am seeing some strange values. Is this the expected output of record.py?

Here a video when the issue happens. Just before the race starts.

AssertionError nvec.mp4.zip

Hope you could help and give some advise.

Best Regards!

Oscar

Multi level

Does this approach extrapolate to different levels? If I train my net with level A, would the trained model work with level B?

How could I record game from Keyboard control

@bzier @kevinhughes27

For the moment, TensorKart records from Joystick input control when calling record.py

As I do not have a joystick, I am trying to change the code so that it can record from Keyboard input control.

I read http://mupen64plus.org/wiki/index.php?title=KeyboardSetup and when I play the MarioKart, I use Left Shift to do "Go" and Up/Left/Down/Right to control the direction.

So should I change the code in https://github.com/kevinhughes27/mupen64plus-input-bot/blob/master/src/controller.c ?
Need I also change code in https://github.com/kevinhughes27/TensorKart/blob/master/record.py ?

Is the file SDL_keysym.h http://www.libsdl.org/release/SDL-1.2.15/include/SDL_keysym.h useful ?

Thanks a lot!

Choppy sound issue

I have to mute the audio while watching the network play because it becomes very choppy/laggy. The audio is fine when I am recording though. Does anyone else have this problem?

Btw, I think a nice improvement would be to include audio snapshots as input to the network.

Problem with installing wxPython==4.0.0a1 on Ubuntu 14.04

I'm using Ubuntu 14.04 and while installing the requirements, it stopped because wxPython version 4.0.0a1 couldn't be installed. But without this recent version, record.py won't work. I guess it doesn't know img = wx.Image(320,240). It says: TypeError: String or Unicode type required. Is there any quick solution or workaround?

TensorKart and Universe openAI

Hi,
I just found your project on nvidia blog.
Why do not integrate it in https://github.com/openai/universe?