danieltakeshi/deformable-ravens

Code for ICRA 2021 paper: https://arxiv.org/abs/2012.03385

deformable-ravens's Issues

Training Problem

Hi, sorry to interrupt!

I used the following code for training with the same data.

python main.py --gpu=0 --task=cable-shape --agent=transporter --crop_bef_q=0 --num_demos=1000

When num_demos is set to 0,1,10,100, the training completes without a problem, but when num_demos is set to 1000, each training will force a kill at 1000 + iterations.

My GPU is an RTX 4090 with 24GB memory. I looked into whether GPU memory was causing the problem and tweaked --gpu_mem_limit, but it still didn't work. I didn't make any other changes to the provided main.py. Do we need to make some parameter changes to main.py when num_demos=1000, or what else is causing this?

Looking forward to your reply!

NOTE TO SELF: make task names consistent with what's in the paper!

The code internally has some different task names than what we reported in the paper, since we were changing what the paper reports for ease of readability, but changing task names in the code requires a little more care. To be specific:

• The three "fabric" tasks in the paper have "cloth" instead, e.g., "fabric-flat" in the paper is "cloth-flat" in this code.
• The "bag-items-1" and "bag-items-2" in the paper are referred to as "bag-items-easy" and "bag-items-hard" in the code.
• The "block-notarget" task in the paper (see the appendix) is referred to as "insertion-goal" in the code.

Complicating matters is that there are a few places in the code which check by detecting if a task name has 'cloth' in it, etc.

For goal-conditioned task, how does the environment judge that the task is successful?

Hi，
I have a question here. For goal-conditioned tasks, how does the environment judge that the task is successful? Because the inputs of agent are current observation image and goal image. Does the environment judge success by comparing the current observation image with the target image? Or is the environment judge success by comparing particle positions, and the image is only used as input to the agent?

The results are inconsistent with those in the paper

Hi!

I used your code to train the Cable-Shape model with Transporter. In the final test, I found that the success rate was quite different from that in the paper. When I learned 1000 demos, the success rate in the paper was 86.5%, but the highest success rate was only 70%.I used the load.py file in the test and tested 100 cases.

Why are my results so different from yours?

Did you use the default 20 cases in your code when testing, or did you choose 100?

The following results were tested with a model of 25,000, 30,000, 35,000 and 40,000 steps respectively.

Question about Modeling of deformable objects

Hi! I would like to know about the problem of deformable object modeling in your paper. Is your obj file modeled by Blender software?

Error message of python packages(tensorflow package version may have mismatched)

I created anaconda environment using your instruction in README(install_ubuntu.sh), and try to execute dataset.py.
But some errors of tensorflow packages. If you know the solution, could you share me?
I put the below that the error message when I tried " python ravens/dataset.py --path goals/bag-color-goal/".

Thank you,

pybullet build time: Sep 22 2020 00:55:20
2021-10-04 18:00:30.592384: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
Detected TensorFlow version: 2.4.1
Traceback (most recent call last):
File "ravens/dataset.py", line 9, in
from ravens import utils as U
File "/home/user/deformable-ravens/ravens/init.py", line 2, in
import ravens.agents as agents
File "/home/user/deformable-ravens/ravens/agents/init.py", line 1, in
from ravens.agents.dummy import DummyAgent
File "/home/user/deformable-ravens/ravens/agents/dummy.py", line 10, in
from ravens.models import Attention, Transport
File "/home/user/deformable-ravens/ravens/models/init.py", line 6, in
from ravens.models.conv_mlp import ConvMLP
File "/home/user/deformable-ravens/ravens/models/conv_mlp.py", line 5, in
import tensorflow_hub as hub
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_hub/init.py", line 88, in
from tensorflow_hub.estimator import LatestModuleExporter
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_hub/estimator.py", line 62, in
class LatestModuleExporter(tf.compat.v1.estimator.Exporter):
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 62, in getattr
module = self._load()
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 45, in _load
module = importlib.import_module(self.name)
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_estimator/init.py", line 10, in
from tensorflow_estimator._api.v1 import estimator
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_estimator/_api/v1/estimator/init.py", line 13, in
from tensorflow_estimator._api.v1.estimator import tpu
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_estimator/_api/v1/estimator/tpu/init.py", line 14, in
from tensorflow_estimator.python.estimator.tpu.tpu_estimator import TPUEstimator
File "/home/user/anaconda3/envs/py3-defs/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 108, in
_tpu_estimator_gauge = tf.compat.v2.internal.monitoring.BoolGauge(
AttributeError: module 'tensorflow._api.v2.compat.v2.internal' has no attribute 'monitoring'

Using the 2-finger gripper?

Thanks a lot for sharing the amazing work,

I can run the demo using self.ee = 'suction', but I failed to do it with Robotiq2F85.

I see this is the line where to load specific end-effector. For example, using insertion.py task, from here I changed it to self.ee = 'gripper'.

The gripper could be loaded but the robot doesn't move correctly to the target object, the end-effector always moves upside down. The robot seems to approach the object for picking but the gripper heads to the up direction.

When I run python main.py --disp --task insertion I get the following,

Traceback (most recent call last):
  File "main.py", line 251, in <module>
    demo_reward, episode, t, last_obs_info = rollout(task.oracle(env), env, task, args)
  File "main.py", line 114, in rollout
    (obs, reward, done, info) = env.step(act)
  File "/home/abdu/deformable-ravens/ravens/environment.py", line 268, in step
    success = self.primitives[act['primitive']](**act['params'])
  File "/home/abdu/deformable-ravens/ravens/environment.py", line 602, in pick_place
    while not self.ee.detect_contact(def_IDs) and target_pose[2] > 0:
TypeError: detect_contact() takes 1 positional argument but 2 were given
numActiveThreads = 0
stopping threads
Thread with taskId 0 exiting
Thread TERMINATED

When I print the pose0[0] and pose0[1] from here I get

Pose0[0] (0.359375, -0.21249999999999997, 0.040639832615852356)
Pose0[1] (0.0, 0.0, 0.0, 1.0)

Any help would be greatly appreciated...
Thanks

Function of the code

I want to know the functions of load.py and how to view zone-template.urdf directly in Pybullet , can you tell me,Plase?

Questions regarding network architecture

Hello Dr Seita,

First of all, thank you for making the code for your paper publicly available, and for the contribution your work has brought to the research community.
I have been reading the "goal-conditioned transporter network" paper in detail, and have a few points of confusion I was hoping I could ask about:

1. For Transporter-Goal-Stack: Although Fig 3 of the paper seems to imply that only $o_t$ is input to $\Phi_{query}$ and $\Phi_{key}$, I was reading through the code and noticed that it seems that actually the full stacked $(o_t, o_g)$ is used as the input to these two FCNs (not only as the input to $f_{pick}$). Could you please confirm this point? link
2. For Transporter-Goal-Split: Similarly, although Fig 3 of the paper seems to imply that only $o_t$ is input to $f_{pick}$ in this architecture, from the code it seems that actually the full stacked $(o_t, o_g)$ is used as input. Could you please confirm this point? link
   (It seems I may be having some misunderstandings regarding how to read Fig 3 properly, so please do let me know if this is the case.)
3. From the code it seems that rotations are actually performed (although not shown in Fig 3). My understanding is that (some input image of some size) is input to $\Phi_{query}$ and we get dense features as output from this; then $T_{pick}$ is sampled from $Q_{pick}$, and then we create rotated crops from the extracted dense features, around $T_{pick}$. We then convolve each crop to get $T_{place}$ as the location with maximum placing success, calculated over all locations and over all rotations. Is this correct?
4. Related to the previous question. Is it correct to understand $Q_{place}$ as a set of num_rotations maps where each map is associated with a rotation, and the pixel values in the map are correlated with predicted placement success given this pick and rotation?

I'm sorry for so many questions but I would really like to better understand the paper's details.
Thank you very much for your time!

Unable to run `main.py`: "No module named 'tensorflow.python.types'"

I followed the instructions (I think):

$ cd deformable-ravens
$ git log -n 1 --oneline --no-decorate
6ff2443 remove outdated bag files (we're using the other ones)
$ conda create -n py3-bullet python=3.7
$ conda activate py3-bullet
$ ./install_python_ubuntu.sh

Then I got the following error:

$ python main.py -h
pybullet build time: Sep 22 2020 00:55:20
Detected TensorFlow version:  2.2.0
Traceback (most recent call last):
  File "./main.py", line 45, in <module>
    from ravens import Dataset, Environment, agents, tasks
  File ".../deformable-ravens/ravens/__init__.py", line 2, in <module>
    import ravens.agents as agents
  File ".../deformable-ravens/ravens/agents/__init__.py", line 1, in <module>
    from ravens.agents.dummy import DummyAgent
  File ".../deformable-ravens/ravens/agents/dummy.py", line 10, in <module>
    from ravens.models import Attention, Transport
  File ".../deformable-ravens/ravens/models/__init__.py", line 6, in <module>
    from ravens.models.conv_mlp import ConvMLP
  File ".../deformable-ravens/ravens/models/conv_mlp.py", line 5, in <module>
    import tensorflow_hub as hub
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_hub/__init__.py", line 29, in <module>
    from tensorflow_hub.estimator import LatestModuleExporter
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_hub/estimator.py", line 64, in <module>
    class LatestModuleExporter(tf_v1.estimator.Exporter):
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 62, in __getattr__
    module = self._load()
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow/python/util/lazy_loader.py", line 45, in _load
    module = importlib.import_module(self.__name__)
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1 import estimator
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/_api/v1/estimator/__init__.py", line 10, in <module>
    from tensorflow_estimator._api.v1.estimator import experimental
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/_api/v1/estimator/experimental/__init__.py", line 10, in <module>
    from tensorflow_estimator.python.estimator.canned.dnn import dnn_logit_fn_builder
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 31, in <module>
    from tensorflow_estimator.python.estimator import estimator
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 51, in <module>
    from tensorflow_estimator.python.estimator import model_fn as model_fn_lib
  File "{home}/.local/opt/miniconda3/envs/py3-bullet/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/model_fn.py", line 29, in <module>
    from tensorflow.python.types import core
ModuleNotFoundError: No module named 'tensorflow.python.types'

See output of conda list --export here:
https://gist.github.com/EricCousineau-TRI/50284aba406e965956a2827de37745b1

The parameter meaning of load.py output.We look forward to your reply.

Excuse me, I'm eager to know why the RESNET network structure used by the network structure chose to give up the BN layer? We look forward to your replyResNet43_8s(input_shape, output_dim, **include_batchnorm=False**, batchnorm_axis=3, prefix='', cutoff_early=False):