The upstart service "roscore" won't start automatically after upgrading to 18.04

[fetchrobotics]

The upstart service "roscore" won't start automatically after upgrading to 18.04

[umhan35]

@erelson I found something very interesting. After I plug in the HDMI cable connected to a monitor. the roscore starts...

Here are related logs in /var/log/syslog (the Bluetooth service is also stopped... #66 In the last few lines you can see something for PS3 controller):

Apr 18 22:15:05 fetch1068 systemd[1]: Received SIGRTMIN+21 from PID 359 (plymouthd).
Apr 18 22:15:05 fetch1068 systemd[1]: Started Hold until boot process finishes up.
Apr 18 22:15:05 fetch1068 systemd[1]: Starting Set console scheme...
Apr 18 22:15:05 fetch1068 systemd[1]: Reached target Multi-User System.
Apr 18 22:15:05 fetch1068 systemd[1]: Starting Job that launches roscore once the system has started...
Apr 18 22:15:05 fetch1068 systemd[1]: Reached target Graphical Interface.
Apr 18 22:15:05 fetch1068 systemd[1]: Started Stop ureadahead data collection 45s after completed startup.
Apr 18 22:15:05 fetch1068 systemd[1]: Starting Update UTMP about System Runlevel Changes...
Apr 18 22:15:05 fetch1068 systemd[1]: Started Daemon for generating UUIDs.
Apr 18 22:15:08 fetch1068 systemd[1]: Started Job that launches roscore once the system has started.
Apr 18 22:15:08 fetch1068 systemd[1]: Started Job that launches the ps3joy node once roscore has started.
Apr 18 22:15:08 fetch1068 systemd[1]: Started Job that launches the robot drivers once roscore has started.
Apr 18 22:15:08 fetch1068 systemd[1]: Startup finished in 4.746s (firmware) + 3.961s (loader) + 1.965s (kernel) + 7min 53.686s (userspace) = 8min 4.360s.
Apr 18 22:15:08 fetch1068 systemd[1]: Stopping Bluetooth service...
Apr 18 22:15:08 fetch1068 bluetoothd[749]: Terminating
Apr 18 22:15:08 fetch1068 bluetoothd[749]: Stopping SDP server
Apr 18 22:15:08 fetch1068 bluetoothd[749]: Exit
Apr 18 22:15:08 fetch1068 NetworkManager[791]: <info>  [1555640108.7471] bluez5: NAP: removed interface 00:16:6F:F5:FC:52
Apr 18 22:15:08 fetch1068 systemd[1]: Stopped Bluetooth service.
Apr 18 22:15:09 fetch1068 spacenavd[755]: trying alternative detection, querying /dev/input/ devices...
Apr 18 22:15:09 fetch1068 upowerd[942]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.2/usb3/3-4/3-4:1.0
Apr 18 22:15:09 fetch1068 kernel: [  476.784785] input: Sony Playstation SixAxis/DS3 as /devices/virtual/input/input13

You can see from this line that the startup won't finish until I plug in a monitor...

Apr 18 22:15:08 fetch1068 systemd[1]: Startup finished in 4.746s (firmware) + 3.961s (loader) + 1.965s (kernel) + 7min 53.686s (userspace) = 8min 4.360s.

We installed Ubuntu 18.04 Desktop, do you also have Desktop installed or the server version?

@norips also discovered this issue.

[erelson]

@umhan35 @norips tl;dr, I think a fix is to do sudo systemctl set-default multi-user.target. Thanks again for finding the key to repro'ing this!

I was able to reproduce the issue on Fetch1. A symptom of this was that the multi-user.target was not started. This could be seen by looking at the output of systemctl list-units --type target.

I then presumed that multi-user.target was tied to a graphical session being started (i.e. by plugging in a display). I'm not sure that was correct. I then tried to modify the systemd unit file for roscore (/lib/systemd/system/roscore.service) to point at a few different targets, and restarting a few times, before finally realizing that the answer was to set the default to multi-user.target, whereas it was set to graphical.target. h/t to this answer, also: https://unix.stackexchange.com/a/164028 ... I now know more about systemd than I did before, and still lots to learn. Thanks for your patience, all!

[umhan35]

@erelson Now roscore automatically starts. Do you plan to write something on docs.fetchrobotics.com and publish a new version of fetch-melodic-config.deb?

[erelson]

Yup, I've added that to my currently open PR for docs. I'll be pushing that live tonight probably. Thanks for confirming!

[moriarty]

That should be setup by the package:
http://packages.fetchrobotics.com/binaries/fetch-melodic-config.deb

In this section of the install document:
https://github.com/fetchrobotics/docs/blob/master/source/indigo_to_melodic.rst#1804-install-and-installing-rosfetch-packages

It must be installed after the step:

       sudo apt install ros-melodic-fetch-calibration ros-melodic-fetch-open-auto-dock \
       ros-melodic-fetch-navigation ros-melodic-fetch-tools -y

https://github.com/fetchrobotics/docs/blame/master/source/indigo_to_melodic.rst#L95-L108

#. **NOTE**: at a later time, Fetch may host and recommend its own mirror of ROS Melodic debians.
#. Run the following to **install Fetch research debians**:

   - General packages for Fetch robots::

       sudo apt install ros-melodic-fetch-calibration ros-melodic-fetch-open-auto-dock \
       ros-melodic-fetch-navigation ros-melodic-fetch-tools -y

   - Then install packages specific to the robot type::

       export ROBOTTYPE=$(hostname | awk -F'[0-9]' '{print $1}')
       # sudo apt install $ROBOTTYPE-melodic-config  # pending future availability
       wget https://packages.fetchrobotics.com/binaries/$ROBOTTYPE-melodic-config.deb
       sudo apt install ./$ROBOTTYPE-melodic-config.deb -y

[moriarty]

@umhan35 I'll assign @erelson.

Let us know if you've installed http://packages.fetchrobotics.com/binaries/fetch-melodic-config.deb

If you try to install the fetch-melodic-config.deb before the ros-melodic-fetch-... packages, it will fail because of unmet dependencies.

[erelson]

@umhan35 thanks for the report! Unfortunately you're not the only seeing this issue, and we've been unable to reproduce it ourselves other than once during testing*.

Can you verify another symptom for us? Note that with Ubuntu 18.04, Ubuntu no longer uses upstart, but instead uses systemd. My understanding is sudo service is a wrapper script around sudo initctl (14.04) and/or sudo systemctl (18.04). The symptom I expect to see in your case is:

1. power on the robot
2. Verify roscore/robot drivers are not running
3. Run sudo systemctl start roscore
4. In testing, we found this to hang, with no errors given.

Let me know if you see the same. If you're also able to check the following, that would potentially be informative

1. power on the robot
2. Verify roscore/robot drivers are not running
3. Run sudo service roscore start
4. Run sudo systemctl start roscore
5. Did step 4 hang, return an error?
6. Run sudo service roscore stop
7. Run sudo systemctl start roscore
8. Did step 7 hang, or return an error?

Can you also let us know what version of systemd you have, via dpkg -l systemd? Thanks!

• The time we saw it during testing, we did a reinstall and this went away. However, another customer has tried reinstalls, and had no success.

[umhan35]

Yes, we installed fetch-melodic-config.deb. Its version is 0.1-0.

[umhan35]

@erelson In /var/log/syslog, there are a lot of repeated entries:

fetch1068 spacenavd[755]: trying alternative detection, querying /dev/input/ devices...

fetch1068 spacenavd[755]:   trying "/dev/input/." ...   trying "/dev/input/.." ...   trying "/dev/input/by-id" ...   trying "/dev/input/by-path" ...   trying "/dev/input/event5" ...   trying "/dev/input/event4" ...   trying "/dev/input/mouse0" ...   trying "/dev/input/event3" ...   trying "/dev/input/event2" ...   trying "/dev/input/event1" ...   trying "/dev/input/event0" ...   trying "/dev/input/mice" ... failed to find any supported devices

I searched spacenavd, is it being using for the PrimeSense sensor?

[norips]

@erelson
Hi,
We are facing the same problem as @umhan35.
We discover that if we plug in the screen, roscore and robot driver starts automatically at boot up without the need to run sudo service roscore start manually.

I followed your step and here are the results:

• sudo systemctl start roscore hangs with no error
• sudo service roscore start works
• sudo systemctl start roscore after sudo service roscore start stills hang

The version installed of systemd is 237-3ubuntu10.15

[moriarty]

I installed desktop version, but selected minimal when it gave the option (no Libre Office etc) on Fetch3 & Fetch1081

@erelson didn’t we see something weird on your Fetch - when the desktop/gui was waiting for some sort of input?

[erelson]

Hi @umhan35 @norips thanks for tracking that down! We've traditionally used the desktop install, and I've continued doing that...

An external monitor being plugged in has other effects I've noticed in the past, such as causing flash drives to auto-mount. I'll look into this tomorrow for sure. One possibility is that I'm triggering the roscore system service unit off the wrong thing (i.e. something tied to a desktop display session). Though given that syslog message, sounds like it might be a more general systemd configuration.

[erelson]

Whoops, a bit hasty there; I'll wait for you guys to confirm before closing this.

[moriarty]

@erelson could you link to your docs PR? Thanks

[moriarty]

Nvm found it- sorry I’m camping and on my phone ;)

[norips]

@erelson Thank you, roscore starts automatically now. Great job !

[erelson]

A quick follow-up note: the oneline fix noted above is added to the docs. It doesn't change the postinstall script of fetch-melodic-config, so there is not a new version of that package for now.

[erelson]

So as an important note... this may break the ability to use a monitor with your robot... This can be reverted with sudo systemctl set-default graphical.target.

I'll be doing some more research/testing the next few weeks, in parallel with working on the automated 18.04 installer, to try and find a better solution.

[moriarty]

So as an important note... this may break the ability to use a monitor with your robot... This can be reverted with sudo systemctl set-default graphical.target.

I'll be doing some more research/testing the next few weeks, in parallel with working on the automated 18.04 installer, to try and find a better solution.

That is a big “important note” and means we should not use the desktop install at all if that is really the case... but for Fetch3, and Fetch1081 which I installed Melodic on, I haven’t seen this issue at all, internally we saw it on Fetch1 which you set up- so maybe I did something out of habit which we can pin point once you’ve worked on an automated install.