far-lab/crossculturaldriving

Basic simulator to run cross cultural driving style studies

crossculturaldriving's Issues

Prompt questions in scenario 8 and 9 automatically

Question, add answer

Why did you make eye contact with the other driver?

Add does not apply

hazard light fix

every time the question includes the "other driver", have a does not apply option

update the technical protocol

delete a part of the first half

d

Unconfimed issues (Legacy)

• Crashes on QN if people drive the wrong way.
• Show STOP only once obstacle was left behind

Hand articulation and expression of pedestrian

For v0.9.3

Open Issues:

• Speed limits in GPS: CP 50 kmh/30mph, NS/NIS 80kmh/50mph @HaukeCornell
• QUestionnaire trigger stopes working after a while.

Issues until 2022/31/5:

• Rerun no light in cabin (was fixed once by placing additional light source in vehicle)
• Questionnaire sometimes (CP1-CP3) is tiny and too small to select anything.
• Intersection GPS trigger are too close behind the TL @HaukeCornell
• SA Q trigger are too far at the end of the intersection road @HaukeCornell
• TL instructions for NI are missing @HaukeCornell

Needed improvements

• Location photos in the questionnaire @HaukeCornell

Bugs that mysteriously vanished:

• Rerun shows wrong or duplicate visual scenes (CP2, CP6, NI1, NI2, NSI1, NSI2)

New Fices (5/12/2022):

• Hebrew inconsistent reversing and line breaks (Please I need help fixing this!)
• In the client the display of the "current question/ total questions" shows a total of zero in all scenarios.
• On the server-side, the same display shows the current question count which I don't understand where this number came from. Like 72/14 for NSI2-5 (Related to the previous question and should be solved)
• The scenario timer on the server-side should stop the moment the participants switched to the waiting room
• Is it possible to have some sort of indication (like a change of color) for scenarios already visited?

New Fixes(4/26/2022):

• Add Timer to the Server screen
• Change Steering Wheel calibration to the "10 and 2 position"
• Add Question Number to the Server QN info
• PairName TextBox not correctly working. ( cannot enter any name)
• Question out of the total question on the server-side display
• Add a screenshot ( well ok its implemented, but not tested at all; well I guess I want feedback)
• Partially conditional questionnaire (follow up question suppression)
• Move the questionnaire in front of the steering wheel
• Move the mirror up a little
• Fix timestamps

Fixes(4/12/2022):

• Engine sound hearable without any distance attenuation ( i need help with testing. trouble replicating it )
• Reverse Option R for Driving (We will not implement this)
• (ReRun) Recording Vehicle sound (Only wind sounds are recorded no engine sounds)
• New Server Interface
• Update QN trigger to instruct the participant to stop.
• FixTrafficLights (Red=>green)
• Break lights are not shared across the network.
• (ReRun) Horn Recording
• PleaseWait Screen at the end of a participant
• ResearchersInfo on the Questionnaire state
• GoBack Button for the questionnaire ( We are only retaining the "FINAL" answer from the participant)
• increase center mirror height
• Show scenario ID.
• TotalQuestionCount For the Participant.
• Faulty initial camera orientations for Camera FOUR @HaukeCornell You mentioned this is fixed for now?
• Add stopping line for all scenarios
• Check if the Horn works
• Brake the car when entering the SA trigger box
• Make the participants stop at the end of the scenario @HaukeCornell
• Skin color yellow or blue

Fixes:

• Hebrew Support (untested)
• Increased the height off the Questionnaire up:(1.5m => 2m up) forward:(4.5m=>4.25m)
• Increase FoV for ReRun
• NS1 fix trigger box @HaukeCornell
• Invisible border around the world
• NI1 (change GPS potentially)
• Add scenario name to the server screen

Ground Truth:

• Used signaling(bool)
• Used horn (bool)
• Signaling light (array of directions)
• Speed range
• Speed variance
• Used breaks (bool)
• Lane location (array of locations for each car, in case there is a change)
• Lane change (bool)
• Total scenario time
• Used Headlight Lights (bool) We dont have headlights
• Horn type (momentary / continues – array in case of more than one) Is in the quantitative data

Needed Features:

• Log files with all the quantity data
• Extra data (ground truth in the SA-Q log file)
• Traffic light implementation
• (ReRun) Addative loading of the Visual Scene [Implemented and initial tests.(needs more testing)]
• Counter balanced versions for all "NI" and "NSI" scenarios @HaukeCornell
• NSI needs a split road(different visuals @HaukeCornell
• NI1 there is no need for any other GPS instruction except for go straight
• NI1 We can place the SA-Q trigger sooner (fixed)
• NSI1 missing trigger box for the SA-Q and this is way one of the participant went over the edge.
• NSI2 There was no sync between the drivers. Car B pass before A had a chance to merge
• NSI2 There is a GPS instruction to go to the signal light, but I’m not sure there is one
• SA-Q CP – add to all question the answer “There was no other driver”

Hard to fix problems(will Not Fix for now):

• (ReRun) Recording and Playaback TrafficLight Integration

Nice To Have:

• On the server show the current IP for easy connection.
• Auto-Connect Quest headset (10sec. timer) @HaukeCornell could you take a look at this.
• IP-input field not working on first startup (VR-UI). @HaukeCornell
• Auto-Recenter on launch (Implications for user manual)

Adjustments (no coding required):

• Street height matches terrain height (new intersection)
• Rear view mirror position (too low)
• reduce scene complexity >70fps is a must @HaukeCornell

Testing and Improvments:

• Full Performance assessment (2+participants) (Quest)
• Full Performance assessment (2+participants) (Server)
• disable VR for windows builds

Issues that appeared in user testing:

Fixed Issues:

• necessary to see the users, and their perspective (using the rerun camera rig during server view)
• (ReRun) No Hand Gesture recording
• (ReRun) No signaling light recording
• Spawning and hidding the QN on sceneload (makes the scenario-end less jaring)
• QuestApp Shutdown on disconnect (when the Server exits or crashes)
• Speedometer Integration
• TrafficLight Integration
• Basic Performance assessment (1 participant)(Quest)

Fixes:

• World Designs and two-player scenarios
• (ReRun) Recording and Playaback Speedometer Integration
• Questionnaire Data recording and Serverside Storage
• Questionnaire Button input (Implement onButtonUp)
• Properly name and build the Quest app
• Physical steering wheel rotation matches virtually

ReRun

• Implement Automatic scene recording
• Automatic naming
• PlayBack Automatic Scene Loading
• PlayBack control
• Verify: Body Pose and Care pose recording
• Verify playback (Especially in server builds

Next Steps:

• Synchronized Actions (Software interface?) (e.g. traffic lights)
• CSV exporter?
• 4 Player setup?

---

Fixed

• StartUp GUI connect to the actual program.
• Automatically instantiate RenderTextures for each vehicle Camera Manager (MonoBehavior) on the client and server-side.
• Calibrate participant position
• Implement the Questionnaire.

Finished features

• Associate Steering wheel to participants (Setup) (Untested)
• Associate Steering wheel to participants (Runtime)
• Hand Gesture sharing?
• ReRun Recording controls (automatic recording start)
• Speedometer
• Add a speedometer to the car

Add navigation screen (wrist watch) to pedestrian

Little Things To Change

This is a hub for minor functionalities that can be implemented to enhance the development, configuration, and testing process.

• In Editor_EnumToGameObject, draw the inspector such that the list structure becomes many lines of "text(enum) dragfield(gameobject)"
• Organize the Asset folder more so less folders are directly exposed on the first layer.
• Create a file to store all the needed enums

--------------------------questions to ask------------------------

• How do the display windows work? If I'm creating canvas UI element, where should I put them?
• Can I just simply add Traffic Light Supervisor in runtime? (it's enabled both in StartServer and StartRerun)

move question location so the mirror isn't in the way

Motion Sickness prevention

• Use 1 simple shader for all materials
• Add a virtual nose
• Lower the car drop height
• Add motion blur
• Increase “caging”: emphasize the car interior and reduce the outlook to the moving world
• Increase visual fidelity

Performance improvements

• Use 1 simple shader for all materials
• Move far clipping plane closer (currently at 100)
• [ ]

The performance is low even on a gaming laptop without any rendered geometry.

https://skarredghost.com/2016/11/28/how-a-developer-can-reduce-motion-sickness/
https://virtualspeech.com/blog/motion-sickness-vr
https://www.researchgate.net/publication/342661085_Virtual_Reality_Sickness_A_Review_of_Causes_and_Measurements
Factors Associated With Virtual Reality Sickness in Head-Mounted Displays: A Systematic Review and Meta-Analysis
https://www.researchgate.net/publication/321661932_Cyber_Sickness_in_Virtual_Reality_-_Literature_Review
Getting Rid of Cybersickness Book
Meta analysis impact of fidelity on simulator sickbess

Scenario Design Guidlines

• Make a new document where scenarios guidelines are discussed wki
• Lower car drop height when loading scenario (can make people nauseous)
• Be a lot more proactive in telling people how to behave if they do mistakes (miss the TL, miss the navigation)

Add Objects for the participant to walk Towards

take out the words "in particular" in the questions

FALL2023 - Issues and Improvements

Open Design Questions

• What do we want to change about the QN?
• Limit top speed (More signage? GPS warnings?)
• What scenarios do we replicate?
• Do we want to use the new environment?

General ToDos:

• Update SOP See: https://www.overleaf.com/read/nckntwdpwwdx
• Start A wiki!
  • Wrote Scenario Design Guidelines see here!
• Scene Update
  • Less depressing houses => Integrate @lees846 (https://github.com/FAR-Lab/XC_AV_NPC)
  • Fix Z-fighting in lees846 new Unity model
  • More variety in trees and objects
  • More variety in curvy road leading to the scenario
• ReRun
  • Recording and Playback Traffic Light Integration
  • (ReRun) Recording Vehicle sound ( Missing engine sounds)
  • Additions from from @leo-c1 found in Rerun Annotation Tool

Performance

• Use 1 shader for all materials.The performance is low even on a gaming laptop without any rendered geometry see [1]. And [2] for more references on this subject.
• Full Performance assessment (2+participants) (Quest)
• Full Performance assessment (2+participants) (Server)
• disable VR for windows builds

Attempts/ideas at fixing Simulator/motion Sickness

• Add motion blur
• Increase “caging”: emphasize the car interior and reduce the outlook to the moving world
• Increase visual fidelity

New Features

• Add turn indicator sound (blinker)
• Reverse Option R for Driving (We will not implement this)

NPC vehicle

• Naive approache (Playback one specific file)
  • Implementation
  • Testing
• Scenario Hyperspace (Finding the closest matching scenario (time and position) to play the other perspective
  • Implementation
  • Testing
• Modelling vehicle control inputs, base on vehicle speed, distance etc.
  • Implementation
  • Testing
• Waypoint (Deisigend behavors)
  • Implementation
  • Testing

Pedestraian implementatonion

• Details to come from @MarioYangNYU

Future Work

• Interwoven streets that allow for longer drivers in which interactions

ROS implementation

• Have planning meeting with @FrankBu0616 and @jankolkmeier

Testing

• Come up with test cases/scenes that can be used to ensure
  • FrameRate
  • Network Latency
  • Visual Fildeity/consitency
  • Mirrors (position)
  • Mirrors (FrameRate/rendering)
  • Questionnaire

Old Bugs

Participant Experience

• Engine sound hear-able without any distance attenuation ( i need help with testing. trouble replicating it ) (*)
• VR/real button positioning should be aligned after scenario (does not rotate back)

Technical Issues

• Align virtual and physical steering wheel rotation (during drive and QN)

Scenario Details

These issues can be fixed inside the unity editor and do not requiree code changes.)

• Option straight is missing from QN CP8
• NSI3 (Q4) add a picture
• NSI3 (Q5) picture wrong for participant A (approaching merging road from wrong side)
• NSI3 A said I was not on the main road, it does not give me the proper option
• Refine QN: CP5, Q5 the car is always pointing straight at an intersection (whats the purpose of question)
• Alternative lane markers, in US usually a yellow line shows opposing traffic
• Alternative markers for scenario start and end than normal traffic signs
• Lower car drop height when loading scenario (can make people nauseous)
• In NI1 the auto-stop can confuse people, let them finish the scenario (pass by the car)
• NSI1 check if conditional questions for A and B are correct
• Speed limit signs
• More interaction and repair provoking scenarios i.e. Scenarios that are leading to repairing interactions (gestures)

Future Work

• 4 Player setup?

Student Projects:

• Different HardWare for turn indicator
• Software interface/api/examle for other nterfaces like bikes

---

References:

[1] low performance even with a gaming Laptop:

[2] https://skarredghost.com/2016/11/28/how-a-developer-can-reduce-motion-sickness/
https://virtualspeech.com/blog/motion-sickness-vr
https://www.researchgate.net/publication/342661085_Virtual_Reality_Sickness_A_Review_of_Causes_and_Measurements
Factors Associated With Virtual Reality Sickness in Head-Mounted Displays: A Systematic Review and Meta-Analysis
https://www.researchgate.net/publication/321661932_Cyber_Sickness_in_Virtual_Reality_-_Literature_Review
Getting Rid of Cybersickness Book
Meta analysis impact of fidelity on simulator sickbess

Record all Skeletons in ReRun

Hands and heads pedestrian still showing

Next steps

• The first line of the CSV is not the header.

Single-participant issues #19

Original from Ryan: https://github.com/FAR-Lab/CrossCulturalDriving2021/issues/19#issue-1157635513

Single-participant version

Calibration was off - likely because participant needs to be looking straight forward when Unity records the hand positions
Calibration step - Participants consistently read 9 and 3 and put hands at 10 and 2
Scenario length compared to questionnaire length feels very short
Scenarios do not inspire usage of hand signals/movements off the wheel
Steering wheel more stiff than expected
Somewhat hard to see instructions/gps screen and drive at same time

ZED Skeleton starts to get strange after a while(Slouching).

Can't see hands from other driver

eye contact - feature request

Better avatar

English is shown during the Questionaire eventhough Hebrew was selected

Have been trying to replicate the issue without success. I need more data to understand when where and how it happens.

Technical Issues and Documentation ToDos:

Shaders, and visual fidleity => conemplating hiring an external Android export to improve photorealism
Implemented but needs testing:

Priority:

• #34
• rerun in general?
• rerun camera setup not working ast the moment @MarioYangNYU
• Naieve NPC implementation (run study in HongKong and NYC and for #GeorgiaTech)
• 4view expeirmenter view
• SOP
• HandTracking
• Automatic Client shutdown on disconnect or server shutdown
• @MarioYangNYU @albertosarus23 How did you resolve tracking discrepancy between Zed hand tracking and Quest Hand tracking?

Scenarios:

• Implement scenarios into strangeTown
• (spawn points) @deydavedey can you post the link to the doc here so I can easily find it when implementing the scenarios. (https://www.dropbox.com/scl/fo/ekrh0stf5i2gp70764oye/h?rlkey=7xgjs9k4jzwrmkwb3da0mpk8j&dl=0)
• (navigation Screen) (?)
• Validate Scenarios(@iamsrijan724)
• Fix lighting

Things That broke That need to be fixed again:

• Fix the Navigation Screen Networked over the GameObject rather then the VR_player
• Seat Calibration ( XR-Hands implies reconfiguring how that script works)
• Add strange Town
• De-spawing of any content associate to a client once they disconnect
• Need to fix QN DataLogger!!
• Need to go through the Various Client and server objects making sure we disable everything we don't need! e.g. ReRun elements on the client!

Technical Issues:

• Fix and test farlab_logger.cs
• DataLogging needs performance improvments
• Overhaul ServerUI (remove all onGUI and use canvas) @MarioYangNYU
• The Questionnaire code should also be updated
• Investiagte Audio spatializer @albertosarus23 (Maybe @lees846 can make a demo scene so we can test it )
• Abstract class for intractable (Used to be known as NetworkedVehcielController) @DavidGoedicke
• Abstract Class for Client (used to be known as participantInputCaputre) @DavidGoedicke
• StartUpScript (Spawn VRUI/serverUI) @MarioYangNYU
• Fix ParticipantSpawen Type during client joining server @MarioYangNYU
• HandTracking Networked
• StateChangeManagment (subscribe able delegate)
• Add scripts on GameManager instead of toggling. (need to talk about traffic light) @MarioYangNYU

Documentation

• Add XRHands Dependency

Pedestrian Version

Thursday:

• ReRun (For Auto-confrontation)
• Navigation screen for pedestrian
• Update Spawning Points
• Questionnaire screen for pedestrian

---

• Full skeleton tracking for the logger
  • Eye contact (eye tracking)
• body language
• SA _1 questions.
• enable Auto-confrontation
• Make Spawn type (embodiment type) join independent

general questions:

• What are the standard signals people use to distinguish the motion of a pedestrian to show intent?

Data Logging

Problem:
Header is the second row in the logging file.
Proposed solution
Suppress logging if head is not yet written in farlab_logger.cs.

before "hurry up", show straight arrow

Long Arms... remove hand pose update from XR HANDS

Addition of self driving vehicles

Visual Fidelity Improvements (Georgia Tech)

• - Port the project to URP
• - Create a branch of the project that uses a Deferred Rendering pipeline with URP
• - Fix the aspect resolution when rendering with three cameras (to have the 3 monitor support)
• - Profile game on both versions
• - Create a new vehicle interior in Blender
• - Use a high fidelity vehicle interior that we got from the asset store

Issues/improvements observed in studies

Bugs

• Option straight is missing from QN CP8
• Other driver is not inside the car, but besides it (perspective from B)
• Crashes on QN if people drive the wrong way. (fixed if the debugger is not running)

Improvements

• Lower car drop height when loading scenario (can make people nauseous)
• VR/real button positioning should be aligned after scenario (does not rotate back)
• Adjustable mirror position (too low)
• Align virtual and physical steering wheel rotation (during drive and QN)
• Add turn indicator sound (blinker)
• Limit top speed (sign, GPS or other)
• Show STOP only once obstacle was left behind

QN bugs improvements

• NSI3 (Q4) add a picture
• NSI3 (Q5) picture wrong for participant A (approaching merging road from wrong side)
• NSI3 A said I was not on the main road, it does not give me the proper option

Look into

• Refine QN: CP5, Q5 the car is always pointing straight at an intersection (whats the purpose of question)
• Alternative lane markers, in US usually a yellow line shows opposing traffic
• Alternative markers for scenario start and end than normal traffic signs
• Different HW for turn indicator
• In NI1 the auto-stop can confuse people, let them finish the scenario (pass by the car)
• NSI1 check if conditional questions for A and B are correct

How to run the study

• Be a lot more proactive in telling people how to behave if they do mistakes (miss the TL, miss the navigation)

Rerun Improvements

• Adjustable 3rd camera
• Speedometer recording
• Traffic light recording
• show field of view

Better Scenarios

More diversity in scenarios

• Less depressing houses
• More variety in trees and objects
• More variety in curvy road leading to the scenario

More driving with more interaction

• Interwoven streets to allow for longer sections of driving

Increase likelihood of interaction encounters that require repair

• Speed limit signs

More interaction and repair provoking scenarios

• Scenarios that are leading to repairing interactions (gestures)

when one driver finishes answering questions and is waiting for other driver to answer, have a please wait pop up?

ROS integration

Step1

• Experiment with unity ROS package
• Build simple cone bot model in unity
• Build fake lidar/camera in Unity
• Make the robot follow a player (using ROS code)

Step 2 (Integration into StrangeLand

• Client_Object
• Interactable_Object

Fail safe visualizations in case of a simulator crash.

Should the simulator crash the participants get pulled into the void (Steam VR constructor).

remove green cylinder + replace it specific object/place for the pedestrian to head to

Remove the Traffic Light in the waiting room...