robofit/arcor

Augmented reality-based human-robot interaction.

ARCOR (formerly known as ARTable) - the main repository

ARCOR - vision of a near future workspace, where human and robot may safely and effectively collaborate. Our main focus is on human-robot interaction and especially on robot programming - to make it feasible for any ordinary skilled worker. The interaction is based mainly on interactive spatial augmented reality - combination of projection and touch sensitive surface. However, more modalities are integrated or currently under development.

Repositories / packages

This repository holds the main components of the system, which are not specific to any particular setup (combination and type of components) or robot:

• art_brain - central node which communicates with robot, manages program execution, holds current system state etc.
• art_bringup - to launch the system.
• art_calibration - AR marker-based mutual calibration of cameras.
• art_collision_env - manages detected as well as artificial objects within the workspace.
• art_db - permanent storage for object types, programs, etc.
• art_instructions - for each supported instruction there is its definition in yaml and respective classes for art_brain and art_projected_gui. Those classes are loaded on startup based on /art/instructions parameter.
• art_led - RGB LED strip interface.
• art_projected_gui - shows system state, allows to set program parameters, etc.
• art_projector - calibrates projector wrt. Kinect and displays scene generated by art_projected_gui.
• art_simple_tracker - not a real tracker, it "tracks" objects based on already assigned IDs and performs position/orientation filtering from multiple detectors.
• art_sound - a sound interface: plays sound for selected system events (error).
• art_table_pointing - uses Kinect skeleton tracking to compute where user points on the table.
• art_touch_driver - reads data from touch foil (which is HID device) a publishes it as ROS messages.

Additional repositories:

• arcor-msgs - ROS messages, services, actions.
• arcor-utils - Python helper classes.
• arcor-detectors - detectors (currently, only AR code detector wrapper).

For each integrated robot, there are two repositories: one with custom packages providing high-level functions compatible with arcor ROS API and one with implementation of art_brain plugin (-interface one):

• PR2
  • https://github.com/robofit/arcor-pr2
  • https://github.com/robofit/arcor-pr2-interface
• DOBOT Magician
  • https://github.com/robofit/arcor-dobot
  • https://github.com/robofit/arcor-dobot-interface
• Empty (dummy) arm
  • https://github.com/robofit/arcor-empty-arm

Currently supported setups (see links for further information):

• arcor setup 1
• arcor setup 2
• arcor setup 3

Any supported setup may be used with any supported robot (or even without one).

Functionality

The system has two main modes: setting program parameters and program execution.

The video below briefly introduces the system and shows how we did its first UX testing:

Currently, the robot program has to be created beforehand (e.g. using script like this. Then, program parameters could be easily set using the projected interface. In order to make setting parameters as simple as possible, the system is based on complex instructions, with high-level of abstraction (for supported instructions see instructions.yaml).

API

All topics, parameters and services can be found in /art namespace.

TBD

Installation

TBD

Contributing

• Follow PyStyleGuide or CppStyleGuide
  • for Python, you may use pre-commit hook to automatically format your code according to PEP8 (just copy the file into .git/hooks).
• Use catkin_lint to check for common problems (catkin_lint -W2 your_package_name)
• Use roslint to run static analysis of your code.
• Ideally, create and use unit tests.
• Feel free to open pull requests!

Publications

• MATERNA Zdeněk, KAPINUS Michal, BERAN Vítězslav, SMRŽ Pavel a ZEMČÍK Pavel. Interactive Spatial Augmented Reality in Collaborative Robot Programming: User Experience Evaluation. In: Robot and Human Interactive Communication (RO-MAN). NanJing: Institute of Electrical and Electronics Engineers, 2018 (to be published).
• MATERNA Zdeněk, KAPINUS Michal, BERAN Vítězslav a SMRŽ Pavel. Using Persona, Scenario, and Use Case to Develop a Human-Robot Augmented Reality Collaborative Workspace. In: HRI 2017. Vídeň: Association for Computing Machinery, 2017, s. 1-2. ISBN 978-1-4503-4885-0.
• MATERNA Zdeněk, KAPINUS Michal, ŠPANĚL Michal, BERAN Vítězslav a SMRŽ Pavel. Simplified Industrial Robot Programming: Effects of Errors on Multimodal Interaction in WoZ experiment. In: Robot and Human Interactive Communication (RO-MAN). New York City: Institute of Electrical and Electronics Engineers, 2016, s. 1-6. ISBN 978-1-5090-3929-6.

arcor's Issues

Integration of more sensors

Is it possible to use "libed_kinect_plugin.so" more than once (https://github.com/robofit/ar-table-itable/blob/master/art_ed/config/fitlab.yaml)?

User tracking (with Kinect2 or unorganized pointcloud)

https://github.com/PointCloudLibrary/pcl/tree/master/gpu/people
https://github.com/strands-project/strands_perception_people/tree/indigo-devel/rwth_upper_body_skeleton_random_walk
https://github.com/mcgi5sr2/kinect2_tracker
http://szymonkaliski.com/blog/2017-01-18-kinect-2-osx/
https://github.com/criverc/K2Skeltrack/tree/criverc_WIP (https://github.com/jmerdich/ros-skeltrack)
https://github.com/spencer-project/spencer_people_tracking (pose only)

System-wide parameters

Some commonly used parameters should be defined for all nodes - at one place.

• table/workspace width, height (/art/table/width, /art/table/height).
• what else?

art_projected_gui: More versatile projector calibration

Calibration now works only if whole projected checkerboard is visible - use some AR marker (ChArUco?) robust to occlusion etc.?

https://github.com/rizasif/ros-indigo-aruco-sample
http://docs.opencv.org/3.1.0/db/da9/tutorial_aruco_board_detection.html

Continuous integration (testing)

Curently, travis can't check much as there is not much code to compile. It would be good to:

• write unit tests (at least for the main parts of the system)
• write ros tests
• run static analysis -> fail on error
• launch file testing

http://wiki.ros.org/unittest
https://github.com/hcrlab/wiki/blob/master/software_engineering/unit_testing.md
https://github.com/hcrlab/wiki/blob/master/software_engineering/continuous_integration.md
http://wiki.ros.org/roslint
http://wiki.ros.org/roslaunch (roslaunch_add_file_check)

Undistort projector image

program_helper: place pose bez frame_id

U place_pose instrukce se zřejmě neukládá frame_id a program helper pak vrací PoseStamped s prázdným frame_id.. Možná by bylo dobré při ukládání kontrolovat jestli je frame_id vyplněné a když ne, tak nedovolit uložit

art_simple_tracker - weighted averaging

• object detectors should publish detections in camera frame_id
• tracker should do weighted averaging based on distance from camera to object (closer = better detection)

art_touch_driver: attempt to assign to self.slot which is None

Traceback (most recent call last):                                                                                 
  File "/home/dev/artable_ws/src/artable-repos/artable/art_touch_driver/src/node.py", line 275, in <module>                                                                                                                            
    node.process()                                                                                                                                                                                                                     
  File "/home/dev/artable_ws/src/artable-repos/artable/art_touch_driver/src/node.py", line 179, in process                                                                                                                             
    self.slot.x = event.value                                                                                                                                                                                                          
AttributeError: 'NoneType' object has no attribute 'x'

AR Tripod (Projector + Kinect2 + NUC)

• prepare HW
• separate repository (ar-table-tripod)?
• package with projector node (extract it from art_projected_gui)
• node to transform + filter pointcloud??

Time synchronization of all PCs

cron / ntpdate + chrony?

art_projected_gui: Make it RPM-independent

There should be no hard-coded font sizes etc. so one can easily change resolution per meter (rpm).

Pick&place: set orientation

• Support in UI: #55.
• Support in art_brain
• Test

Picking from feeder

• art_projected_gui
  • selection of object type
  • button for activation of arm interactive mode -> move arm automatically in front of user and then switch the interactive mode on)
  • visualize feeder / direction of feeder somehow?
• art_brain
  • open gripper in learning phase
  • check/test state_pick_from_feeder
• object detection + tracking
  • detectors should publish detections in camera frame_id with RPY 0/90/180/270 degrees
  • tracker should do weighted averaging (or better - kalman filtering?) based on distance from camera to object (closer = better detection)
    • https://github.com/pykalman/pykalman
    • http://campar.in.tum.de/Chair/KalmanFilter
  • use AR code bundle on both ends of al. profiles?
  • object detector using PR2 forearm cameras
• art_pr2_grasping:
  • prepare to pregrasp pose (pose in ProgramItem with -0.2m in x-axis of end eff)
  • wait for detection, get object ID
  • add object to collision map
  • open gripper
  • move to pregrasp pose based on detected pose
  • cartesian movement to grasp pose
  • close gripper, check its state, attach collision object
  • move upward
  • move out of feeder

art_brain, art_projected_gui: Use ProgramArray instead of Program

Using ProgramArray.msg instead of Program.msg (Program.msg will serve as container for logically related instructions (program block) - e.g. assembly of one side of the trolley)

• Rewrite DB #42
• Write helper class to deal with Program/ProgramBlock/ProgramItem
• Integrate it in art_brain
• update interface_state_manager
• Integrate it in art_projected_gui

Display projected GUI in RViz

https://github.com/lucasw/rviz_textured_quads -> need to generate Image + CameraInfo

MoveIt - Octomap - self-filtering problem

With point clouds it works,with depth there is problem with self-filtering of the robot (so arms get into collision often).

https://groups.google.com/forum/#!topic/moveit-users/RsmXVih0SIE

Check if place pose for the object is reachable

UI should ask brain - brain asks art_pr2_grasping for left/right arm - brain replyes

What if robot takes object from somewhere using left arm but right arm is required to place the object?

UI should indicate invalid (unreachable) place pose.

Program stop, pause, stepping

• stop (there is already service, red button in art_projected_gui)
• pause (there is already service)
• stepping - need to think about it

Object tracking

Generate IDs (uuid?) and use VISP (http://wiki.ros.org/visp_tracker) for tracking - now all IDs are stored in art_db.

art_projected_gui: Polygon can be moved during program execution

art_projected_gui: Newly appearing objects of selected type are not selected

art_projected_gui: SceneLabel

Display label stored in ProgramItem message. Allow resizing / repositioning -> labels will be hand coded to templates

Service for recalibration of projectors

art_projected_gui - trigger calibrate_projectors state from a new service.

Integrate touch foil

art_touch_driver

• read singletouch data (python-evdev?) -> publish as PoseStamped (/art/interface/touchtable/single)
• calibration (homography from arbitrary number of points) - ask art_projected_gui to show those points (service)
• latched bool topics: /art/interface/touchtable/calibrated, /art/interface/touchtable/calibrating
• normal (not latched) topic: /art/interface/touchtable/touch_detected (std_msgs/Empty)
• parameter /art/interface/touchtable/active_area -> array of x,y coordinates defining touchable area on the table
• empty service /art/interface/touchtable/calibrate (called by art_brain)
• save calibration on the parameter server

art_projected_gui

• service to show calibration points: /art/interface/projected_gui/touch_calibration (PointStamped[])
• read /art/interface/touchtable/touch_detected and display calibrations points (one by one)
• new cursor type
• make pointing/touch cursors work together
• show active area

art_brain

• wait until art_projected_gui is ready, check if touchtable requires calibration (/art/interface/touchtable/calibrated) and if so, ask it to calibrate by calling service /art/interface/touchtable/calibrate

Other stuff

• test calibration with a new table
• make a cool video ;-)

art_projected_gui bug

[INFO] [WallTime: 1504615938.726930] Notification: This program item seems to be done
Traceback (most recent call last):
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/touch_table_item.py", line 128, in ps_cb_evt
    self.cb(msg)
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/touch_table_item.py", line 202, in touch_cb
    self.delete_id(msg.id)
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/touch_table_item.py", line 179, in delete_id
    self.touch_points[id].end_of_touch()
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/touch_table_item.py", line 35, in end_of_touch
    self.pointed_item.cursor_release()
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/item.py", line 160, in cursor_release
    self.cursor_click()
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/button_item.py", line 53, in cursor_click
    self.clicked(self)
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/list_item.py", line 76, in item_clicked_cb
    self.item_selected_cb()
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/program_item.py", line 433, in item_selected_cb
    self.item_switched_cb(self.block_id, self.item_id)
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/gui/ui_core_ros.py", line 822, in active_item_switched
    self.learning_vis(block_id, item_id, read_only)
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/gui/ui_core_ros.py", line 756, in learning_vis
    self.select_object_type(self.ph.get_object(block_id, item_id)[0][0])
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/gui/ui_core.py", line 266, in select_object_type
    if it.object_type.name == obj_type_name:
AttributeError: 'NoneType' object has no attribute 'name'
Traceback (most recent call last):
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/gui/ui_core_ros.py", line 1105, in object_cb_evt
    obj.set_orientation(conversions.q2a(inst.pose.orientation))
  File "/home/dev/artable_ws/src/artable/art_projected_gui/src/art_projected_gui/items/object_item.py", line 125, in set_orientation
    self.lx = self.m2pix(self.inflate + self.object_type.bbox.dimensions[2])
AttributeError: 'NoneType' object has no a

Calibration data should be in /art/ namespace

Object classification

art_projected_gui: Use Object's BB

• To show detected objects
• When setting place pose (instead of circle)
• Make sure it does not interfere with object detection (as the current circle do)

art_brain, art_projected_gui: Execute instructions as they are learned

During learning, the instructions will be executed and trolley will be assembled - otherwise it might be difficult to learn some tasks (as #46 and #47)

• Button in GUI to execute learnt instruction
• Execution of the current ProgramItem
• Solve: what to do when switching instructions

Run projector node / kinect2_bridge on NUC

• Projector node
• Kinect2
• Prepare some easy way how to launch that stuff

art_projected_gui: Cursor

• Fix mouse control in debug view
• Test and fix pointing cursor
• Add checking of scene boundaries

art_projected_gui crash

  File "/home/ikapinus/catkin_ws/src/ar-table-itable/art_projected_gui/src/gui/ui_core_ros.py", line 258, in interface_state_evt
    self.add_place(translate("UICoreRos", "OBJECT PLACE POSE"),  it.place_pose, obj.object_type, obj_id,  fixed=True)
AttributeError: 'NoneType' object has no attribute 'object_type'

Initialization - start of the system

One launch file for:

• table
• PR2
• each NUC
• update documentation / README

art_calibration: Fix Z axis so it goes upwards

Glueing

cartesian movement between two points
teach start / end point for each profile using robot arm (art_projected_gui)

• Extend https://github.com/robofit/ar-table-common/blob/master/art_msgs/msg/ProgramItem.msg
• Implement functionality (new package?)
• Support in art_projected_gui
• Support in art_brain

Drilling

tool pick up, move to drilling pose, cartesian movement + gripper rotation, cartesian movement, place tool back, teach using robot arm

• Extend https://github.com/robofit/ar-table-common/blob/master/art_msgs/msg/ProgramItem.msg
• New package for kinesthetic teaching functionality (drilling/gluing)?
• Support in art_projected_gui
• Support in art_brain

art_db: Rewrite

• Rewrite art_db to use http://wiki.ros.org/mongodb_store as a permanent storage - we use arrays in our messages a lot and it's not possible to save them using current approach.
• Store only object type (name, bounding box / 3D model, additional info)
• Update program storage (to enable #44)

art_projected_gui: after main node restart, projector node should reconnect

API documentation

Generate documentation (markdown?) from API file: https://github.com/robofit/ar-table-itable/blob/master/art_bringup/config/api.yaml

Robot interface vs robot without 'look_at' capability

ZdenekM@f8b814a

Touch table - search for device name

...as device file (/dev/event...) changes from time to time.

art_projected_gui: update to use PyQt5 and Python3

Look at https://github.com/mmisono/pyqt5-example

Hover for fixed (detected) objects

At least for touch cursor it does not work.