Docker Image: Non-NVIDIA image seg faults on NVIDIA system

[osrf]

Docker Image: Non-NVIDIA image seg faults on NVIDIA system

[osrf-migration]

Original comment by Yu Tsao (Bitbucket: Yu Cao).

---

Hi,

I am experiencing the same issue on segmentation fault and I got no luck with nvidia-docker either.

#!bash

~/vmrc/docker$ ./run.bash -n osrf/vmrc:nvidia_current
xauth:  /tmp/.docker.xauth not writable, changes will be ignored
xauth: (argv):1:  unable to read any entries from file "(stdin)"
chmod: changing permissions of '/tmp/.docker.xauth': Operation not permitted
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

Hi Yu,

It looks like the Nvidia runtime is not set up. Do you have an Nvidia card?

[osrf-migration]

Original comment by Yu Tsao (Bitbucket: Yu Cao).

---

Yes, I do have an Nvidia card.

#!bash
$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750 Ti  Off  | 3167fc3ec4555094a7bbd76a87b675a9e5e2d065:01:00.0  On |                  N/A |
| 30%   30C    P8     1W /  38W |    712MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

OK, great. Could you confirm that you have Nvidia Docker runtime installed (https://osrf-migration.github.io/vrx-gh-pages/#!/osrf/vmrc/wiki/tutorials/installNvidiaDocker)? Does the base image give you the same output as above when you run:

#!bash

$ docker run --runtime=nvidia --rm nvidia/cuda:9.1-devel nvidia-smi

[osrf-migration]

Original comment by Yu Tsao (Bitbucket: Yu Cao).

---

Michael,

Thanks for the help, I figure out I am using nvidia/cuda:9.0 as base image. After trying cuda:9.1-devel the problem goes away.

Cheers!

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

• changed milestone from "0.3" to "0.4"

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

@M1chaelM, is this still an open issue?

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

• changed milestone from "0.4" to "1.1"

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

@caguero,
This is still open but low priority. It doesn't have to work, but it also doesn't seem right that adding an Nvidia card should render the integrated graphics unusable. I suspect the Nvidia setup may be altering default values in some way but I haven't had a chance to hunt this down yet.

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

• set component to "Docker"

Not sure if this is a bug, or just something we should document.

On my desktop, with an NVIDIA card, I successfully run the NVIDIA version of the Docker Hub image tutorial.

If I try to run the non-NVIDIA image...

$ ./run.bash osrf/vmrc:current

The docker image downloads and runs correctly, but then seg faults when I try the example...

developer@e7912a51336d:~$ roslaunch robotx_gazebo sandisland.launch 
... logging to /home/developer/.ros/log/58bfcf3c-aa33-11e8-aa75-0242ac110002/roslaunch-e7912a51336d-58.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://e7912a51336d:38151/

SUMMARY
========

PARAMETERS
 * /robot_description: <?xml version="1....
 * /rosdistro: kinetic
 * /rosversion: 1.12.13
 * /use_sim_time: True

NODES
  /
    gazebo (gazebo_ros/gzserver)
    gazebo_gui (gazebo_ros/gzclient)
    spawn_model (gazebo_ros/spawn_model)

auto-starting new master
process[master]: started with pid [68]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 58bfcf3c-aa33-11e8-aa75-0242ac110002
process[rosout-1]: started with pid [81]
started core service [/rosout]
process[gazebo-2]: started with pid [88]
[ INFO] [1535399815.825274566]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1535399815.825703481]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
process[gazebo_gui-3]: started with pid [110]
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Warning [parser.cc:644] XML Attribute[xmlns:xacro] in element[sdf] not defined in SDF, ignoring.
Warning [parser.cc:644] XML Attribute[xmlns:xacro] in element[world] not defined in SDF, ignoring.
[ INFO] [1535399816.093618114]: Finished loading Gazebo ROS API Plugin.
process[spawn_model-4]: started with pid [177]
[ INFO] [1535399816.115683393]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
[ INFO] [1535399817.094805121]: Pattern is YRY
[ INFO] [1535399817.132733059, 0.022000000]: waitForService: Service [/gazebo/set_physics_properties] is now available.
[ INFO] [1535399817.157287113, 0.046000000]: Physics dynamic reconfigure ready.
[ INFO] [1535399817.365024684, 0.249000000]: waitForService: Service [/gazebo/set_physics_properties] is now available.
Warning [parser.cc:644] XML Attribute[xmlns:xacro] in element[plugin] not defined in SDF, ignoring.
[ INFO] [1535399817.489728272, 0.307000000]: Physics dynamic reconfigure ready.
[spawn_model-4] process has finished cleanly
log file: /home/developer/.ros/log/58bfcf3c-aa33-11e8-aa75-0242ac110002/spawn_model-4*.log
Segmentation fault (core dumped)
[gazebo_gui-3] process has died [pid 110, exit code 139, cmd /opt/ros/kinetic/lib/gazebo_ros/gzclient __name:=gazebo_gui __log:=/home/developer/.ros/log/58bfcf3c-aa33-11e8-aa75-0242ac110002/gazebo_gui-3.log].
log file: /home/developer/.ros/log/58bfcf3c-aa33-11e8-aa75-0242ac110002/gazebo_gui-3*.log
^C[gazebo-2] killing on exit
[gazebo-2] escalating to SIGTERM
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
developer@e7912a51336d:~$ 

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

• removed "1.1" milestone

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

• set milestone to "1.2"

[osrf-migration]

Original comment by Michael McCarrin (Bitbucket: m1chaelm).

---

• changed state from "new" to "resolved"

This issue can be resolved by forcing the host system to use non-nvidia graphics drivers. So, I think it’s working as expected. What appears to be happening is that systems with Nvidia graphics cards installed are configured to use Nvidia’s proprietary drivers, and the default assumption is that if you are doing this you don’t also need to use the onboard graphics drivers, so these are disabled. The Nvidia-docker image is set up to find the Nvidia drivers, but the non-nvidia image looks for the default onboard graphics drivers, fails to find them because they are disabled, then tries to load the software rendered (swrast) and fails to find that, then crashes.

Fixing this so you can run either image on an Nvidia machine would probably mean figuring out how to configure the host so you can run both graphics drivers at the same time without creating a conflict. I don’t think this is something we need to support. If someone really needs to run the non-nvidia image on a machine using Nvidia drivers, the easiest thing to do is to switch to the non-Nvidia drivers first.