zebradevs/fetch_pbd

Programming by Demonstration for Fetch

Fetch Programming by Demonstration

TODO: Travis build status and coveralls testing?

This repository is based on PR2 Programming by Demonstration. This version is for the Fetch Robot.

The original PR2 Programming by Demonstration was done by Maya Cakmak and the Human-Centered Robotics Lab at the University of Washington.

System Requirements

This PbD is designed for Ubuntu 14.04 and ROS Indigo.

Installing from Source

Clone this repository and build on the robot:

cd ~/catkin_ws/src
git clone https://github.com/fetchrobotics/fetch_pbd.git
cd ~/catkin_ws
catkin_make

To make sure all dependencies are installed:

cd ~/catkin_ws
source ~/devel/setup.bash
rosdep update
rosdep install --from-paths src --ignore-src --rosdistro=indigo -y

Making Changes to the Web Interface

To make changes to the web interface, you will want to install all of the web dependencies:

roscd fetch_pbd_interaction
cd web_interface/fetch-pbd-gui
./install_web_dependencies.sh

Note that the above adds the following paths to the end of your ~/.bashrc:

export NVM_DIR="/home/USERNAME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"  # This loads nvm

You will want to source your ~/.bashrc file again before running the software. Now all of the dependencies are installed. After you make changes to the web interface, to build you can run:

roscd fetch_pbd_interaction
cd web_interface/fetch-pbd-gui
./build_frontend.sh

Now when you roslaunch this software it will run using your newly built changes!

Running

Commands on Fetch Robot

Terminal #1

source ~/catkin_ws/devel/setup.bash
roslaunch fetch_pbd_interaction pbd.launch

You can run the backend without the "social gaze" head movements or without the sounds by passing arguments to the launch file:

source ~/catkin_ws/devel/setup.bash
roslaunch fetch_pbd_interaction pbd.launch social_gaze:=false play_sound:=false

You can also pass arguments to the launch file to save your actions to a json file or load them from a json file. This behaviour is a bit complicated. It is recommended that you specify the full path to files or else it will look in your .ros folder. If you specify a from_file then actions will be loaded from that file. They will replace the ones in your session database. Whatever was in your session database will get stored in a timestamped file in your .ros folder (not overwritten). If you specify a to_file then the session you are starting will be saved to that file.

source ~/catkin_ws/devel/setup.bash
roslaunch fetch_pbd_interaction pbd.launch from_file:=/full/path/from.json to_file:=/full/path/to.json

Using the GUI

In your browser go to ROBOT_HOSTNAME:8080 in your browser to use the GUI. There is a "mobile" interface, which is the same but without the visualization. This can be useful for just saving/deleting primitives.

The main page lists all the available actions. You can directly run/copy/delete actions from the main page. Or hit the "Edit" button to see more information on that action.

On the "Current Action" screen, most of the buttons are pretty self-explanatory. You can execute the entire action using the "Run" button at the bottom of the screen. This will execute all of the primitives in the order they appear in the Primitive List. You can click on a specific primitive (either the marker or the list item), to highlight the primitive.

You can show/hide the markers for each primitive by clicking the marker icon for the primitive in the Primitive List.

You can change the order of the primitives by dragging them to a new position in the list.

You can edit the position and orientation of certain primitives by clicking the edit icon or by moving the interactive marker.

You can change the frame that certain primitives are relative to by right-clicking the marker.

You can also change the name of the action.

You can record objects in the environment using the "Record Objects" button. The objects will appear in the viewer. Poses can be relative to these objects in the environment.

Grasping in Fetch PbD

You can now (optionally) run Fetch PbD with a grasp suggestion service. This is a service that takes in a point cloud for an object and returns a PoseArray of possible grasps for that object. Fetch Pbd provides feedback by publishing which grasp was chosen by the user. Any service that adheres to the .srv interface defined by SuggestGrasps.srv and optionally the feedback .msg interface, GraspFeedback.msg, can be used. If you want to start Fetch PbD with a grasp suggestion service:

source ~/catkin_ws/devel/setup.bash
roslaunch fetch_pbd_interaction pbd.launch grasp_suggestion_service:=grasp_service_name grasp_feedback_topic:=grasp_feedback

The grasp_feedback_topic is optional and can be excluded. Then in the interface you can right-click on objects in the scene and add a grasp for that object. Initially a grasp primitive is generated that is just a placeholder and does not specify any poses. To use the grasp suggestion service to generate grasps, you then right click the blue placeholder box and select "Generate grasps". The service will return a list of grasp poses. Fetch PbD sets a pre-grasp that is 15 cm away from the grasp. The first grasp generated is shown and you can switch to other grasp options by right-clicking the grasp marker and selecting one from the list.

Grasps are executed like any other primitve and can be added to actions in combination with other primitives.

Code Interface

You can also access the actions you've programmed through code. You still need to run pbd_backend.launch.

Commands on Fetch

source ~/catkin_ws/devel/setup.bash
rosrun fetch_pbd_interaction demo.py

System Overview

Interaction Node

The pbd_interaction_node.py handles the interaction between speech/GUI and the rest of the system. Changes happen through the update loop in interaction.py and also through the callbacks from speech/GUI commands. interaction.py also subscribes to updates from the pbd_world_node.py, which notifies it of changes in objects in the world. Through callbacks and the update loop, interaction.py hooks in to session.py. session.py handles creating actions and primitives and saving them to the database.

Arm Control Node

The pbd_arm_control_node.py is how the robot's arm is controlled to execute actions/primitives. It provides a lower level service interface to move the arm. The interaction node interacts with this through the interface in robot.py.

World Node

The pbd_world_node.py handles the robot's perception of the world. Other nodes ask the world node about the state of the world and can both send and subscribe to updates to the world. Its main function is to provide a list of objects currently in the scene.

Social Gaze Node

The social_gaze_server.py handles the movements of the robot's head. This is also controlled through the robot.py interface. The sounds are also provided through this interface.

fetch_pbd's Issues

Tuck arm button on the interface

Enables easy driving

indigo debbuilds are failing of fetch_pdb_interaction

It looks like some missing build dependencies.

For example:

http://build.ros.org/view/Ibin_uT64/job/Ibin_uT64__fetch_pbd_interaction__ubuntu_trusty_amd64__binary/81/console

00:53:13 [100%] Built target fetch_pbd_interaction_generate_messages
00:53:13 make -f CMakeFiles/install_web_dependencies.dir/build.make CMakeFiles/install_web_dependencies.dir/depend
00:53:13 make[4]: Entering directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:13 cd /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6 /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6 /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu/CMakeFiles/install_web_dependencies.dir/DependInfo.cmake --color=
00:53:13 make[4]: Leaving directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:13 make -f CMakeFiles/install_web_dependencies.dir/build.make CMakeFiles/install_web_dependencies.dir/build
00:53:13 make[4]: Entering directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:13 cd /tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/web_interface/fetch-pbd-gui && ./install_web_dependencies.sh
00:53:13   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
00:53:13                                  Dload  Upload   Total   Spent    Left  Speed
00:53:13 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  9321  100  9321    0     0   295k      0 --:--:-- --:--:-- --:--:--  303k
00:53:13 => nvm is already installed in /home/buildfarm/.nvm, trying to update using git
00:53:13 
=> 
00:53:14 => Appending source string to /home/buildfarm/.bashrc
00:53:15 nvm is not compatible with the npm config "prefix" option: currently set to "/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/debian/ros-indigo-fetch-pbd-interaction/home/buildfarm/.nvm/versions/node/v7.2.0"
00:53:15 Run `nvm use --delete-prefix v7.2.0 --silent` to unset it.
00:53:15 => Close and reopen your terminal to start using nvm or run the following to use it now:
00:53:15 
00:53:15 export NVM_DIR="/home/buildfarm/.nvm"
00:53:15 [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"  # This loads nvm
00:53:17 nvm is not compatible with the npm config "prefix" option: currently set to "/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/debian/ros-indigo-fetch-pbd-interaction/home/buildfarm/.nvm/versions/node/v7.2.0"
00:53:17 Run `nvm use --delete-prefix v7.2.0 --silent` to unset it.
00:53:17 N/A: version "N/A" is not yet installed.
00:53:17 
00:53:17 You need to run "nvm install N/A" to install it before using it.
00:53:18 v7.2.0 is already installed.
00:53:19 nvm is not compatible with the npm config "prefix" option: currently set to "/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/debian/ros-indigo-fetch-pbd-interaction/home/buildfarm/.nvm/versions/node/v7.2.0"
00:53:19 Run `nvm use --delete-prefix v7.2.0` to unset it.
00:53:19 ./install_web_dependencies.sh: line 7: npm: command not found
00:53:19 ./install_web_dependencies.sh: line 8: npm: command not found
00:53:19 /usr/bin/env: node: No such file or directory
00:53:19 make[4]: *** [CMakeFiles/install_web_dependencies] Error 127
00:53:19 make[4]: Leaving directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:19 make[3]: *** [CMakeFiles/install_web_dependencies.dir/all] Error 2
00:53:19 make[3]: Leaving directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:19 make[2]: *** [all] Error 2
00:53:19 make[2]: Leaving directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/obj-x86_64-linux-gnu'
00:53:19 dh_auto_install: make -j1 install DESTDIR=/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6/debian/ros-indigo-fetch-pbd-interaction AM_UPDATE_INFO_DIR=no returned exit code 2
00:53:19 make[1]: *** [override_dh_auto_install] Error 2
00:53:19 make[1]: Leaving directory `/tmp/binarydeb/ros-indigo-fetch-pbd-interaction-0.0.6'
00:53:19 make: *** [binary] Error 2
00:53:19 dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
00:53:19 E: Building failed
00:53:19 Traceback (most recent call last):
00:53:19   File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 133, in build_binarydeb
00:53:19     subprocess.check_call(cmd, cwd=source_dir)
00:53:19   File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
00:53:19     raise CalledProcessError(retcode, cmd)
00:53:19 subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-indigo-fetch-pbd-interaction']' returned non-zero exit status 1
00:53:19 # END SUBSECTION
00:53:19 
00:53:19 --------------------------------------------------------------------------------------------------
00:53:19 `apt-src build ros-indigo-fetch-pbd-interaction` failed.
00:53:19 This is usually because of an error building the package.
00:53:19 The traceback from this failure (just above) is printed for completeness, but you can ignore it.
00:53:19 You should look above `E: Building failed` in the build log for the actual cause of the failure.
00:53:19 --------------------------------------------------------------------------------------------------
00:53:19 

Running this package on a simulated robot

Hi,
I am currently working on a project in lfd for which I would like to use this interface. I don't have access to the fetch robot. However, I would like to simulate the robot in gazebo and control it using the web interface. Right now, I am not able to do that as the robot model would not appear in the web interface. I have set the ROBOTHOSTNAME to localhost currently.

How to control the speed of the Fetch's arm?

Hi all,

I have some code set up that can move the robot's arm (well, end-effector) to a pose. When I do this to the physical Fetch robot, the arm moves very quickly. Thus, I am hoping to control the speed to slow it down.

My code is similar to this code base, but I'm thinking of now using stuff from fetch_pbd and merging it into my code, which is why I'm reading this code repository. I noticed a few observations:

• The arm.py code here uses a velocities argument in the move_to_joints_plan method. However, it's commented out. Is this supposed to be something that could control the speed of the robot?

• move_to_pose has a self._move_group.set_planning_time(1.0) call. I assume this is for the planning time but does not affect how fast the arm moves once the plan is set. Is that correct?

• fetch_pbd_interaction/arm_control.py has some module-level constants: MOVE_TO_JOINTS_SLEEP_INTERVAL and TRAJECTORY_COMPLETE_SLEEP_INTERVAL . But the code comments there imply that these are invoked after the arm has finished moving, and does not control the speed while the arm is moving.

I wonder if there is an example script somewhere that could show an example of controlling the velocity of the robot? There seems to be some code re-use between several arm-related scripts so I also appreciate any code recommendations or best practices. Or maybe the best practice is to try and put a rospy.sleep(...) somewhere during the move methods?

Thanks all! I'm happy to clarify my questions as needed.

Runtime socket.error when creating a fetch_pbd couchdb

I'm trying to use the launch file from starting fetch_pbd node, but I'm getting this socket.error error. From looking at the traceback, it looks like its having trouble creating a instance of a couchbd:

Traceback (most recent call last):
  File "/home/ruffsl/workspaces/fetchrobotics/src/fetch_pbd/fetch_pbd_interaction/nodes/pbd_interaction_node.py", line 28, in <module>
    interaction = Interaction()
  File "/home/ruffsl/workspaces/fetchrobotics/src/fetch_pbd/fetch_pbd_interaction/src/fetch_pbd_interaction/interaction.py", line 61, in __init__
    self._im_server)
  File "/home/ruffsl/workspaces/fetchrobotics/src/fetch_pbd/fetch_pbd_interaction/src/fetch_pbd_interaction/session.py", line 67, in __init__
    self._db = self._couch.create('fetch_pbd')
  File "/usr/lib/python2.7/dist-packages/couchdb/client.py", line 193, in create
    self.resource.put_json(validate_dbname(name))
  File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 405, in put_json
    status, headers, data = self.put(*a, **k)
  File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 384, in put
    return self._request('PUT', path, body=body, headers=headers, **params)
  File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 419, in _request
    credentials=self.credentials)
  File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 239, in request
    resp = _try_request_with_retries(iter(self.retry_delays))
  File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 205, in _try_request_with_retries
    raise e
socket.error: [Errno 111] Connection refused
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
[fetch_pbd_interaction-12] process has died [pid 9659, exit code -6, cmd /home/ruffsl/workspaces/fetchrobotics/src/fetch_pbd/fetch_pbd_interaction/nodes/pbd_interaction_node.py __name:=fetch_pbd_interaction __log:=/home/ruffsl/.ros/log/58547894-f22f-11e6-9460-d8cb8a401fcc/fetch_pbd_interaction-12.log].
log file: /home/ruffsl/.ros/log/58547894-f22f-11e6-9460-d8cb8a401fcc/fetch_pbd_interaction-12*.log

Full session output I'll link here:
http://pastebin.com/WFyLanWQ

I poked about the web interface, and it's rendering a display and interfaces, but I can't create any actions as obviously the database is up.

default vs bundled?

Following the installation instructions both in the README and on docs.fetchrobotics I ended up with no web UI working. I traced it down to the following: serve_web_interface.sh (in fetch_pbd_interaction/scripts) is pointed at a directory called "bundled" -- but I only have "default" for some reason. Is this a bug? Or a missing configuration?

Add option for poses to be relative to torso or base.

When you save poses it should be possible to choose either of these frames. Currently only the base is an option.

Clarifying build and run instructions

Building from a fetch platform, I found I was missing a lot of runtime dependencies, that catkin build was not warning me about. Might be useful to remind users and the use of :

rosdep install --from-paths src --ignore-src --rosdistro=indigo -y

Looks like the only missing runtime dependency from the package.xml was the couchdb package,
As noted in #5

Also for runtime,
It looks like you are installing nvm locally using those creationix/nvm scripts, which is cool as it avoids the hole debian nodejs packging debocle. But it subtly adds the paths for nvm and nodejs to the tail of the user's ~/.bashrc. So you may want to remind users to resource their ~/.bashrc so the javascript executables can find nodejs before you try and roslaunch fetch_pbd_interaction.

Where is `robot.py` as mentioned in the documentation?

Hi,

In two locations in this README, you mention robot.py:

Arm Control Node
The pbd_arm_control_node.py is how the robot's arm is controlled to execute actions/primitives. It provides a lower level service interface to move the arm. The interaction node interacts with this through the interface in robot.py.

Social Gaze Node
The social_gaze_server.py handles the movements of the robot's head. This is also controlled through the robot.py interface. The sounds are also provided through this interface.

However, I cannot find any robot.py file in this repository.

Do you know where this file is located? It sounds like it would be very useful for programming the Fetch.

Thank you!

Expose un_relax_arm as a service call?

Sometime I'd like the arm to hold a pose and lock the joints.
So it would be nice if a service call for un_relax_arm was exposed,
just as there is one for relax_arm. Nitch use case I suppose.

rospy.Service('relax_arm', Empty, self._relax_arm)
+ rospy.Service('un_relax_arm', Empty, self._un_relax_arm)

Although, in my quick attempt at exposing un_relax_arm as a service and calling the private function, it looks like still doesn't lock the arm to the same extent as if I finished or interrupted the tuck_arm interface. Also setting the ControllerState to STOPPED or all _gravity_comp_controllers doesn't have the same effect.

Joystick mappings for the PbD Commands

Be able to optionally run a node on startup to receive PbD-specific commands from the joystick.