personalrobotics/benchmarks

Benchmarks to measure OpenRAVE's performance.

benchmarks's Issues

Remove herbpy dependency

Currently, for herb benchmarks we load herb via herbpy.initialize. Remove this once issue #243 in prpy is resolved.

JSON dump instead of YAML for benchmark_execute

The BenchmarkCollisionResult class that is intended, will need to also save the env file that the planning call is made in so that the collision checks can be done for that environment. I want to keep the last block of outfile dumping to be common, rather than two separate blocks depending on whether collision checks are to be logged or planning times. It is far more wasteful to save the env to a .yaml file than a .json file.
@gilwoolee can we change the if outfile is not None: block in execute_benchmark to do a JSON dump rather than a YAML dump?

Benchmark suggestions

@cdellin requests:

1. more structured representation of the planning query
2. more structured representation of the world
3. more concise input specification (i.e. duplication between different results)

NamedPlanner does not support PlanToConfiguration

When I try to run execute_benchmark on a log generated from a WAM testing environment (here - https://github.com/personalrobotics/benchmarks/tree/new_planning_benchmarks_example/prpy_benchmarks/example)
the following line (https://github.com/personalrobotics/benchmarks/blob/feature/new_planning_benchmarks/prpy_benchmarks/src/prpy_benchmarks/benchmark_utils.py#L52)
gives an error:
RuntimeError: That planner does not support planning method PlanToConfiguration!
when either CBiRRT or Snap is used

ODE collision checking bug

I'm seeing a bug with the ODE collision checker in OpenRAVE that occurs very rarely (~0.0001% of configurations) where ODE reports a HERB configuration as invalid (in collision) when it is actually valid (free). I've found four occurrences of this so far (I found this while comparing the performance of ODE and FCL for generating self-checked roadmaps). I have not found an examples of the converse problem (ODE reporting no collision even if one exists). One example is this configuration:

The right arm configuration which provokes the bug is: [ 4.2735537841327362, 0.4740221519094111, -2.4119405981641928, 1.9935127248613886, -1.7596538407270064, -0.14357628170248468, -1.4142931202799787 ].

In this case, the spurious collision is between the /right/wam3 link (upper arm) and the /right/wam5 link (small piece near the wrist); these links are actually ~25cm away from each other at their closest point. The bug is very sensitive; e.g. changing the elbow joint in the 14th digit in either direction changes the result so that it is correct. PQP and FCL both do not suffer from this bug.

Here's a minimal test case: https://gist.github.com/cdellin/b743425f5dd96da92b0c

I don't have the time to investigate this further. Here are some other notes:

• The bug persists even if all links except /right/wam3 and /right/wam5 are disabled.
• The bug persists on either robot.CheckSelfCollision or env.CheckCollision(link1,link2).
• Opening a qtcoin viewer after the spurious check and clicking once on the robot to perform a collision check reproduces the bug (i.e. the robot is bounded by an orange box, and a verbose message is printed reporting the /right/wam3-/right/wam5 collision. However, clicking a second time produces the correct result. I'm not sure what clicking does behind the scenes (recomputes kinematics?) but this might be a place to look.
• I tried saving the exact transforms of the offending geometries and loading/posing the underlying STL files directly into OpenRAVE, but was not able to reproduce the bug in this case. Either I'm not recreating the transforms accurately enough, or the bug has to do with the fact that the links are posed kinematically inside the robot.

I'm adding this here because I don't know where else to add it. If we find that it's a bug in ODE or OpenRAVE, we should open issues with those projects. I suppose it's also possible that the HERB description (mesh files?) has a bug somewhere.