Error running CuRobo examples from isaac sim docker

[D-1shu]

Error running CuRobo examples from isaac sim docker

[balakumar-s]

Can you try in isaac sim 2022.2.1 while we look at issues with 2023.1.1?

[balakumar-s]

Can you clear your docker build cache and try again? We pushed a fix for this issue yesterday but your docker might be using an older commit.

[D-1shu]

Thanks!! a new docker build solved the earlier issue of running curobo examples But I am not able to run the curobo issac sim examples.
To give you a bit of context I am running the issac sim docker on aws server and streaming it using omniverse streaming client. Seems like CuRobo is running as I can see activity in console after running the following command.

/isaac-sim/python.sh simple_stacking.py --headless_mode native

2024-01-25 18:08:19 [523,919ms] [Warning] [curobo] Running in headless mode: native
[523.981s] [ext: omni.kit.asset_converter-2.0.12] startup
[524.102s] [ext: omni.kit.tool.asset_importer-2.4.18] startup
[524.213s] [ext: omni.isaac.asset_browser-0.3.1] startup
[524.292s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[524.296s] [ext: omni.kit.livestream.core-2.2.1] startup
[524.303s] [ext: omni.kit.streamsdk.plugins-2.3.7] startup
[524.304s] [ext: omni.kit.livestream.native-2.3.0] startup

Active user not found. Using default user [kiosk]Streaming server started.
reached picking target:  cube_1
reached placing target:  cube
reached picking target:  cube_2
reached placing target:  cube_1
reached picking target:  cube_3
reached placing target:  cube_2
reached picking target:  cube_4
reached placing target:  cube_3
reached picking target:  cube_5
reached placing target:  cube_4
reached picking target:  cube_6
reached placing target:  cube_5
reached picking target:  cube_7
reached placing target:  cube_6
reached picking target:  cube_8
reached placing target:  cube_7

TASK DONE

But the streaming client shows a blank screen

Logs after I terminated the code

024-01-25 18:20:42 [1,266,765ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-25 18:20:42 [1,266,765ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1266.960s] Simulation App Shutting Down
2024-01-25 18:20:42 [1,266,955ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /isaac-sim/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2024-01-25 18:20:42 [1,267,004ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.graph.action/bin/libomni.graph.action.plugin.so remained loaded after unload request.
2024-01-25 18:20:42 [1,267,005ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.

[balakumar-s]

Did you run xhost + outside the docker in aws?

[D-1shu]

No, I did not run xhost + outside the docker because I do not have X Window System server running on the aws and xhost + returns xhost: unable to open display ""
Also I can run issac sim from the same container using this ./runheadless.native.sh and visualize it using streaming client. I am not sure why i am not able to run the isaac example scripts by curobo. Any help or direction to work on will be appriciated Thank you!

[balakumar-s]

Do any of the other examples work? /isaac-sim/python.sh motion_gen_reacher.py --headless_mode native

[D-1shu]

I haven't tried all the examples but for all the examples I have tried I am facing the same issue including /isaac-sim/python.sh motion_gen_reacher.py --headless_mode native seems like curobo is initialzed and working though.

warming up...
Curobo is Ready
2024-01-26 08:14:42 [59,387ms] [Warning] [curobo] Running in headless mode: native
[59.430s] [ext: omni.kit.asset_converter-2.0.12] startup
[59.477s] [ext: omni.kit.tool.asset_importer-2.4.18] startup
[59.592s] [ext: omni.isaac.asset_browser-0.3.1] startup
[59.679s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[59.684s] [ext: omni.kit.livestream.core-2.2.1] startup
[59.688s] [ext: omni.kit.streamsdk.plugins-2.3.7] startup
[59.688s] [ext: omni.kit.livestream.native-2.3.0] startup

Active user not found. Using default user [kiosk]Streaming server started.
2024-01-26 08:14:54 [71,231ms] [Warning] [omni.client.python] Detected a blocking function. This will cause hitches or hangs in the UI. Please switch to the async version:
  File "/pkgs/curobo/examples/isaac_sim/motion_gen_reacher.py", line 376, in <module>
  File "/pkgs/curobo/examples/isaac_sim/motion_gen_reacher.py", line 215, in main
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/scenes/scene.py", line 225, in add_default_ground_plane
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 504, in get_assets_root_path
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 191, in check_server
  File "/isaac-sim/kit/extscore/omni.client/omni/client/__init__.py", line 161, in stat

2024-01-26 08:16:55 [191,971ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
2024-01-26 08:16:56 [193,600ms] [Warning] [omni.hydra] Mesh '/World/obstacles/table_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
**** Click Play to start simulation *****
**** Click Play to start simulation *****
**** Click Play to start simulation *****
**** Click Play to start simulation *****

Is this log helpful or it because I do not have have a display connected?

2024-01-26 08:13:54 [11,206ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-26 08:13:54 [11,206ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-01-26 08:13:54 [11,207ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-01-26 08:13:54 [11,208ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)

[D-1shu]

@balakumar-s From what I have observed the issue is with the isaac sim version 2023.1.0. and is not possible to live stream with a python script. Checkout this
I was trying to install CuRobo with 2023.1.1 but it is not supported directly through dockerfiles yet. I am going ahead by making bare minimum changes in the docker files to install curobo with isaac sim 2023.1.1. Let me know if you have any suggestions on it.

[D-1shu]

Update: I am able to run and live stream some of the examples like collision_checker.py on AWS server with

cuRobo installation mode: docker isaac sim
python version: 3.10.13
Isaac Sim version (if using): 2023.1.1

But with this I am running into yet another issue when I am running /isaac-sim/python.sh ik_reachability.py --headless native or /isaac-sim/python.sh motion_gen_reacher.py --headless_mode native This time CuRobo doesn't get intialized.

Traceback (most recent call last):
  File "/pkgs/curobo/examples/isaac_sim/ik_reachability.py", line 366, in <module>
    main()
  File "/pkgs/curobo/examples/isaac_sim/ik_reachability.py", line 168, in main
    robot, robot_prim_path = add_robot_to_scene(robot_cfg, my_world)
  File "/pkgs/curobo/examples/isaac_sim/helper.py", line 115, in add_robot_to_scene
    robot_p.set_solver_velocity_iteration_count(4)
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/articulations/articulation.py", line 981, in set_solver_velocity_iteration_count
    self._articulation_view.set_solver_velocity_iteration_counts(count)
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 3243, in set_solver_velocity_iteration_counts
    set_prim_property(self.prim_paths[i], "physxArticulation:solverVelocityIterationCount", counts[read_idx])
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/prims.py", line 802, in set_prim_property
    prim.GetAttribute(property_name).Set(property_value)
pxr.Tf.ErrorException: 
	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_SetValueImpl' at line 6174 in file /buildAgent/work/ac88d7d902b57417/USD/pxr/usd/usd/stage.cpp : 'Empty typeName for </World/panda.physxArticulation:solverVelocityIterationCount>'

Any help or direction to work on will be appreciated Thank you!!

[balakumar-s]

Can you comment out these lines:

curobo/examples/isaac_sim/helper.py

Line 115 in 88eac64

robot_p.set_solver_velocity_iteration_count(4)

[D-1shu]

After commenting robot_p.set_solver_velocity_iteration_count(4) and robot_p.set_solver_position_iteration_count(44)
while running /isaac-sim/python.sh ik_reachability.py --headless native I am able to start and stream the simulation but as soon as I hit the play button the process crashes with this log

2024-01-29 07:26:47 [203,932ms] [Warning] [omni.isaac.dynamic_control.plugin] Failed to find articulation at '/World/panda'
2024-01-29 07:26:47 [203,932ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetArticulationRootBody: Invalid or expired articulation handle
2024-01-29 07:26:47 [203,933ms] [Warning] [omni.physx.tensors.plugin] Failed to find articulation at '/World/panda'
2024-01-29 07:26:47 [203,933ms] [Error] [omni.physx.tensors.plugin] Pattern '/World/panda' did not match any articulations

Traceback (most recent call last):
  File "/pkgs/curobo/examples/isaac_sim/ik_reachability.py", line 366, in <module>
    main()
  File "/pkgs/curobo/examples/isaac_sim/ik_reachability.py", line 235, in main
    my_world.reset()
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 408, in reset
    self._scene._finalize(self.physics_sim_view)
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/scenes/scene.py", line 322, in _finalize
    robot.initialize(physics_sim_view)
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/articulations/articulation.py", line 294, in initialize
    self._articulation_view.initialize(physics_sim_view=physics_sim_view)
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/articulations/articulation_view.py", line 349, in initialize
    assert self._physics_view.is_homogeneous
  File "/isaac-sim/extsPhysics/omni.physics.tensors-105.1.12-5.1/omni/physics/tensors/impl/api.py", line 170, in is_homogeneous
    return self._backend.is_homogeneous
AttributeError: 'NoneType' object has no attribute 'is_homogeneous'

[Entongsu]

I encountered a similar problem in the 2023.1.1 version.