fs-driverless/formula-student-driverless-simulator

A virtual world where Autonomous Systems from different Formula Student teams can compete in time-trial challenges

formula-student-driverless-simulator's Issues

Track shows black, flickering areas

Those black spot's shouldn't be there.

sinewave example autonomous system

To show the world how their autonomous system will connect to the simulator we will develop a sinewave autonomous system. This autnomous system wants to receive all the sensors but ignores it's data. Instead it just publishes car setpoints (speed and steering) based on a hard-coded sinewave.

This setup has to be tested just like the final system will work. So run this sinewave AS on a separate gpc instance and connect through the gcp to the simulator.

Extensive documentation is very important for this thing!

Add a section to the GCP setup doc for developers who have access to the FSOnline GCP project

This would consist of a lot less steps and just show how to create a VM instance and what snapshot to use. At this point this is only aimed at developers in the sim-dev team (from FSTD), but at some point it might be relevant for other teams who want to get a remote simulation environment setup quickly for testing

Define mission controll messages

Since the teams will have no access to their autonomous systems during competition times, and since we will run multiple events (trackdrive + autocross), the simulator must inform the autonomous system's what is going on. Also, when the autonomous system needs to do a re-run for whatever reason the AS must receive a signal telling it to reset.

Create message definitions + documentation + reserve topic names for these ros messages.

Spectator view for livestreaming

Is it possible to stream camera footage from inside unreal engine to some external streaming platform?

Add a new camera inside the unreal engine world that is not coupled to the car. This camera can just be statically located with a top-down view of the track. What this camera sees must be streamed to some third party. Go online, find some online streaming platform (youtube? twitch?) and get this camera view live in their.

The goal of this task is to figure out how streaming to some external location works. We will need this for the livestream and getting an understanding of how this works is essential.

GPS: Heading

Add heading to the gps information

Hide 'record' button

In the right bottom corner is a 'record' button. Let's hide it.

Google cloud snapshot for quick development

Provide a google cloud snapshot of a windows vm pre-installed with the simulator and ros bridge to get started quickly.

Add noise to interoceptive sensors (IMU, GPS)

Simple gaussian noise will be enough for now. This will make sure teams have to use some kind of filtering algorithm, even if the sensors do not have drift.

Integration handbook

Write a guide on how to integrate an autonomous system into the simulator.

• High level
• Go/finish signals
• Sensor suite boundries
• Sensor descriptions
• ROS topics publish / subscribe description

Visualize backend errors to the user

When something goes wrong in the backend, let's show the user.

Stream ground truth cone location data from Unreal Engine to ROS

This would be really useful for early autonomous software pipeline integration tests, where we wouldn't have to rely on retraining and adapting the perception stack before being able to test path planning and controls algorithms / delays, etc

Map building tutorial

Create a tutorial document describing how to build a new map. This can be be based on documentation provided in the AirSim project. It should include:

• How to set the track (cones)
• How to set the weather and day cycle conditions. We don't want automatic environment changes so we must document how to set these to some static value.
• How to change the car-starting position

F12 Dynamically update car sensor suite

The airsim rpc api must be extended with a new function that sets the sensor suite. For this first version this includes:

• Camera mounting on the car (x, y, z, roll, yaw, pitch)
• Lidar mounting on the car (x,y,z, roll, pitch, yaw)
• GPS position on the car (x, y, z)

Car looks customization

Figure out how to change the car 3d model and skin without reloading the simulation.

VNC dying when opening VS Code

Steps to replicate:

• Start the VM instance from GCP
• ssh into the instance and restart vncdisplay.service
• On your laptop run

ssh -NL 5900:localhost:5900 <user>@<ip>
vncviewer DotWhenNoCursor=1 localhost:5900

• Log in through the remote display
• Open VS Code
• Wait for the VNC tunnel to interrupt and close
• Inspecting the logs should show something like the screenshot below.

or (when running sudo journalctl -ru vncdisplay.service):

Setup Github Actions

Automatically test the simulator workspace by compiling the code.

GPS: Timestamp

The gps messages published to ros are not timestamped. They should be! It would be great if the timestamp is the actual timestamp that the gps datapoint was captured.

F23 Cone DOO counter

The formula student 2020 rulebook writes:

D 9.1.2: A cone is Down or Out (DOO) if the cone has been knocked over or the entire base of the cone lies outside the box marked around the cone in its undisturbed position.

Make something in unreal engine that counts the number of DOO cones during a simulation and show this value in the airsim heads on display.

Store a list of timestamps when every DOO cone when DOO.

Make github (readme) pages pretty!

Bring your artistic spirit to life and add some nice art, pictures, etc to the github readme's!

Import ZalaZone map into project

We want to use the zalazone maps. Let's get those imported in this project, setup an example track and make the car drive in here.

F28 Reset world

When the airsim reset rpc call is triggered the following must happen:

• The car must be placed on the start location, velocity, torque and steering must be set to 0.
• All cones must be placed upright to their original location
• The referee status must be reset. Number of DOO cones must be set to 0, laptimes list must be cleared and the driving state must be set to NOT_STARTED

After this reset has finished the whole system must be ready to do another run.

Error: IOLoop is already running

Get this error while running the operator and clicking 'send GO signal`

Exception in thread Thread-41:
Traceback (most recent call last):
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\threading.py", line 1254, in run
    self.function(*self.args, **self.kwargs)
  File "webserver.py", line 265, in referee_state_listener
    ref = self.client.getRefereeState()
  File "..\AirSim\PythonClient\airsim\client.py", line 1130, in getRefereeState
    referee_state_raw = self.client.call('getRefereeState')
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\site-packages\msgpackrpc\session.py", line 41, in call
    return self.send_request(method, args).get()
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\site-packages\msgpackrpc\future.py", line 25, in get
    self.join()
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\site-packages\msgpackrpc\future.py", line 22, in join
    self._loop.start()
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\site-packages\msgpackrpc\loop.py", line 22, in start
    self._ioloop.start()
  File "C:\Users\sijme\AppData\Local\Programs\Python\Python38-32\lib\site-packages\tornado\ioloop.py", line 755, in start
    raise RuntimeError("IOLoop is already running")

F25 Stop the car after crossing the finishline after required number of laps

When the car crosses the finish line after the 10th lap (this value will be configurable whenever more events then trackdrive were added), the car's breaks must be engaged fully and the car must come to a stop. This whole thing must happen inside Unreal Engine 4.

Notify users of invalid sensor configuration.

Print warnings to console (or send them as logs to the webGUI) if the sensor suite requirements are not compliant with the rules set out in the integration handbook. It is basically a unit test of the teams settings.json file. We might not even have to do it at runtime.

Simulator offer http endpoint

Be able to send start/stop signals to the simulator over an http endpoint.

Make referee status available remotely

A client of the airsim rpc api wants to request how the car is doing. Inside UE4 this information is available. The 'Referee status' exists for now of the following fields:

• Number of DOO cones together with the timestamp of when every individual cone went DOO. Required: #3

Implement an rpc endpoint that provides these values on request.

Keep in mind that in a later stage the following fields will be added to the referee status:

• List of elapsed time for every finished lap. Required: #5
• State. One of the following: Required: #4 and #5
  • NOT_STARTED: when the car has not yet crossed the starting line
  • DRIVING: The car is driving on the track, not yet reached the required number of laps
  • FINISHED: The car has crossed the finish line for the number of required number of laps.
  • OFFCOURSE: the car went off course as described in the rulebook and was stopped.

Lap timer is giving wrong numbers

When vieweing the lap times in the operator the values are wrong.

Remove support for multi-vehicles from ros bridge.

This project is built for formula student driverless competitions where always 1 car is on the track at the same time. We don't expect this will change within the next 5 years.

AirSim supports multiple vehicles and this makes AirSim pretty complex. Like there is this whole layer of abstraction everywhere which we just won't use in this project. This results in slower development and a higher learning curve for new engineers.

If the time comes that multiple vehicles have to be supported a new architecture has to be discussed because a single unreal engine instance will never be able to simulate sensors for multiple cars.

Over the last few weeks we have already started to remove support for multiple vehicles. The car name and topic names have been hard-coded and code for drones has been partly removed. We will continue to remove support for multiple vehicles as we go.

This ticket is here as future reference to understand why we are removing multi-vehicle support.

Monitoring ROS-Bridge

We have to know how the ros-bridge is performing. Because if the simulation breaks, we have to be able to prove that the autonomous system was receiving sensordata correctly. This will determinate if a team gets a retry. We are going to do this by measuring all the things:

• RPC Call delay: Track the time that it takes for rpc calls to complete. We can measure this by adding a timing block in the ros-bridge. Note the timestamp before requesting and calculating the difference to the current timestamp when it completes.
• Actual publishing frequency: Track the frequency at which we are publishing to the sensor topics.
• Control-commands frequency: Track the frequency at which we are receiving control commands.

All these statistics should be printed to the console once a second.

GPS: publish noise levels

Use the covariance fields to publish the noise level of the gps captures.

Remove all drone things from the repo

Many drone-related code (multirotor and mavlink) was added to the repo when we forked airsim. Since we are not using it, and it is only confusing, we want to remove it.

Logbook

What happens inside the simulation is stored in a single logbook. This includes all referee updates, which ASs were selected and which tracks were used. If something unexpected went wrong like a system's crash or error, a short description of what happened is included in this logbook. It gives a timeline of everything that happened to always go back afterwards and check what happened. The logbook is stored on disk so that in the event of a whole system crash we will still have the logbook. It is also shown within the referee's web interface.

Validate that the dimensions of the cone are in line with rules

The rules describe a specific type and size of cone. We have to make sure simulated cones are the right size and colors.

Cone requirements as descibed in the fsg competition handbook 2020 v1.0

Changing weather/daycicle programmatically

Like adding rain, fog, snow and setting the day/night cycle.

Define sensor suite

Create a document describing the sensor suite for teams. What sensors are available? How many of each? Which properties are teams able to configure?

F29 Detect Off-course

The formula student 2020 rulebook writes:

D 9.1.5 Off-course (OC) An OC occurs when the vehicle has all four wheels outside the track boundary as indicated by edge marking.

Make something in Unreal Engine that continuously checks if the car is still within the track. If the car goes OC, the car must be stopped until the simulation is reset. The OC status must be visible on the heads on display.

F07 Are cone physics realistic?

Cones must move according to real physics! When a cone is hit, it must move or fall over. This task is to investigate the default cone behavior. Is it good enough? If not, make a follow-up task describing what needs to be improved.

Do not allow negative brake or throttle commands

In the control command callback of the ROS Bridge, a throttle/brake command >0 will be set to 0.

F27 Web interface for official

Build the following web interface. This can be just static html with a bit of js (or any fancy framework of your choice).

It should have have all components as shown in above picture but can be re-ordered to your liking. The following alerts/interactions should be built in:

When selecting a team other than the one currently selected, before sending any requests, show a big red alert stating: "When selecting another team the whole simulator will be reset. Any ongoing laps will be stopped and all timers will be reset. Any items currently in the logbook will stay there". One red button "I understand, change to team X" to continue and one green button "Do not change the team, go back".

A similar popup should be shown when the 'reset simulator' button is pressed.

When the 'go' and 'stop' buttons are pressed a less intrusive confirmation box should be shown.

All interactions with the backend are out-of-scope for this task.

Setup the Unreal Engine project

Add the UE4 project to this repository. We have to research best practices regarding UE4 projects in git/github.

• Create a new project, give it a nice name and add it to git
• Populate it with an (empty) airsim project
• Check the license for all added content
• Add any required attributions in the readme
• Document how to get this project running on a new machine, including airsim documentation references.

Improve camera performance

At this moment the cameras are laggy. With 3 camera framerate can drop to 3 fps and with 1 camera it drops to 20 fps. We want: steady 30 fps for the onboard camera and the livestream angle.

This is the epic adventure to make the simulator performant.

F23, F24 Lap timer and counter

Implement a lap timer that tracks how long the car did for every lap.

The formula student 2020 rulebook writes:

D 8.2.5 Starting - A go-signal from RES is used to indicate the approval to begin. Timing starts after the vehicle crosses the starting line.

Let's use the center point of the car as the measurement point for timing. So start and stop the timer when the car's center point crosses the start/finish line.

A time value for every lap must be recorded and shown on the heads up display.

A lap counter can be derived from the number of timed laps. This value must also be shown on the heads up display.

Transmitting the lap times and lapcounter to some place outside unreal engine is out of scope for this issue.

Simulator v1

The simulator program will be the 'head quarters' of this system. The competition officials will be able control the events using a gui provided by the simulator, the autonomous systems will connect to the simulator to retrieve sensor data and send trajectory setpoints.

The challenge in this task is to create the first version of this program.

It must be able to connects to airsim and retreives lidar + camera stream from the car. It must wait for a start signal (an http post requset to some endpoint will suffice). When the start signal is received the camera and lidar stream must be forwarded to an external autonomous system through ros. Publishing on a ros topic is good enough for now. The simulator must listen on a ros topic for car setpoints and forward those to the car inside airsim.

This program must be built in python.

F21 Car Dynamic Model

Airsim uses this code to move the car according to inputs. It is pretty good but we require an individual four-wheel motor model. The input must be 4x motor torques and 1x steering angle. We have to create our own by copying this model and changing it.

This also involves changing the CarControls message so that the four-wheel car can actually be controlled remotely.

System architecture description

Describe the systems architecture in the docs. Make sure it is understandable for everyone that is not in the core development team.

Quick button click prevention.

There are quite a few buttons that you shouldn't press quickly after each other. For example, if you press the 'LAUNCH simulator' button quickly, you get many simulators. More cases exist.

Lets try to:

1. in the webserver build checks that stop requests when a the simulator and ros-bridge are launching/stopping.
2. Visualize in the web gui what the operator is currently doing. For example a status-text showing "launching simulator..."
3. Disable buttons during requests.

Unreal process crashes

I wasn't paying attention to the machine, but after leaving the simulation running for about 7 mins I got a crash report with the following logs:

LoginId:860ba023ccb1a0522953411da87a6924-000003ea
EpicAccountId:

Caught signal

libUE4Editor-Core.so!FMallocBinned2::Malloc(unsigned long, unsigned int) [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Public/HAL/MallocBinned2.h:447]
libUE4Editor-Core.so!FMemory::Malloc(unsigned long, unsigned int) [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Public/HAL/FMemory.inl:21]
libUE4Editor-Renderer.so!FMeshDrawShaderBindings::CopyFrom(FMeshDrawShaderBindings const&) [/home/developer/UnrealEngine/Engine/Source/Runtime/Renderer/Private/MeshPassProcessor.cpp:656]
libUE4Editor-Renderer.so!FMeshDrawCommand::FMeshDrawCommand(FMeshDrawCommand const&) [/home/developer/UnrealEngine/Engine/Source/Runtime/Renderer/Public/MeshPassProcessor.h:450]
libUE4Editor-Renderer.so!FDynamicPassMeshDrawListContext::AddCommand(FMeshDrawCommand const&) [/home/developer/UnrealEngine/Engine/Source/Runtime/Renderer/Public/MeshPassProcessor.h:767]
libUE4Editor-Renderer.so!void FMeshPassProcessor::BuildMeshDrawCommands<TMeshProcessorShaders<FHitProxyVS, FHitProxyHS, FHitProxyDS, FHitProxyPS, FMeshMaterialShader, FMeshMaterialShader, FMeshMaterialSh$
libUE4Editor-Renderer.so!FHitProxyMeshProcessor::AddMeshBatch(FMeshBatch const&, unsigned long long, FPrimitiveSceneProxy const*, int) [/home/developer/UnrealEngine/Engine/Source/Runtime/Renderer/Private$
libUE4Editor-Renderer.so!GenerateDynamicMeshDrawCommands(FViewInfo const&, EShadingPath, EMeshPass::Type, FMeshPassProcessor*, TArray<FMeshBatchAndRelevance, TMemStackAllocator<0u> > const&, TArray<FMesh$
libUE4Editor-Renderer.so!FMeshDrawCommandPassSetupTask::AnyThreadTask() [/home/developer/UnrealEngine/Engine/Source/Runtime/Renderer/Private/MeshDrawCommands.cpp:785]
libUE4Editor-Renderer.so!TGraphTask<FMeshDrawCommandPassSetupTask>::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type) [/home/developer/UnrealEngine/Engine/Source/Run$
libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasks() [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:1029]
libUE4Editor-Core.so!FTaskThreadAnyThread::ProcessTasksUntilQuit(int) [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:854]
libUE4Editor-Core.so!FTaskThreadAnyThread::Run() [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:931]
libUE4Editor-Core.so!FRunnableThreadPThread::Run() [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
libUE4Editor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/developer/UnrealEngine/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:177]
libpthread.so.0!UnknownFunction(0x76da)
libc.so.6!clone(+0x3e)

Provide rosbags with example data

Record all topics published by the default settings.json into a rosbag while driving with a joystick. The goal is to kickstart teams' development.

Image topics simplification

currently there are way too many unused image topics. This is because image_transport is being used to send the images. Thats best practice and all but we won't be using any of these compressions. We are just going to use raw transport because low latency is most important and we have bandwidth to spare. Let's remove image_transport and instead use a single topic for each camera.

Q: Do we need the ImageInfo?

Also need to change the camera topic names to /fsds/camera/CAMERANAME