qstat -a gives slurm error

[thomaskwok91]

qstat -a gives slurm error

[thomaskwok91]

Hi @mike0042 I finally figured out the issue. I did have one remnant slurm left, it was slurm-wlm-torque. Finally got it all to work! Thanks for all your assistance

[thomaskwok91]

This may or may not be because of the fail source /etc/profile.d/pbs.csh file which says unexpected end of file (ends in endif).

I instead sourced pbs.sh which had no issue but I assume I probably needed to do csh

[mike0042]

PBS Pro and SLURM are different products that fulfill the role of workload/resource managers. You should only install and run one of these packages. (We recommend PBS Pro!) You may use the which command to see where qstat is coming from...

$ which qstat
/opt/pbs/bin/qstat

[thomaskwok91]

Okay I ran which qstat and I got back
/usr/bin/qstat

How do I change it to /opt/bin/qstat?

Sorry for the stupid questions

[mike0042]

Not a problem. Run configure like this:

./configure --prefix=/opt/pbs

The "make install" will then install PBS Pro into /opt/pbs rather than /usr.

[thomaskwok91]

I just did those steps and it didn't seem to work. My guess is that when I run 'make install' it isn't actually installing. I get this message which at first I didn't realize is an error message but now I think it is.

Making install in buildutils
make[1]: Entering directory '/home/tkwok/Downloads/pbspro-19.1.2/buildutils'
make[2]: Entering directory '/home/tkwok/Downloads/pbspro-19.1.2/buildutils'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.

[mike0042]

I've not seen the problems you're encountering. First, please check your PATH to make sure it doesn't include something unexpected. Next, delete the source directory you've been using up to this point (or move it out of the way). Then go to https://github.com/PBSPro/pbspro/releases and re-download the 19.1.2 tarball. Unpack the contents and follow the instructions in the INSTALL file carefully. You may also access the instructions at https://github.com/PBSPro/pbspro/blob/release_19_1_branch/INSTALL

Aside from the warnings you see from running ./autogen.sh everything should build cleanly if your system is configured properly. If it doesn't, please indicate the first place where something unexpected happens.

[thomaskwok91]

Hi @mike0042, sorry hadn't been in the office for a few days. When I run ./autogen.sh I also get this error message:

test/fw/Makefile.am:44: warning: wildcard $(srcdir: non-POSIX variable name
test/fw/Makefile.am:44: (probably a GNU make extension)
test/fw/Makefile.am:48: warning: wildcard $(srcdir: non-POSIX variable name
test/fw/Makefile.am:48: (probably a GNU make extension)
test/fw/Makefile.am:52: warning: wildcard $(srcdir: non-POSIX variable name
test/fw/Makefile.am:52: (probably a GNU make extension)
test/fw/Makefile.am:56: warning: wildcard $(srcdir: non-POSIX variable name
test/fw/Makefile.am:56: (probably a GNU make extension)
test/fw/Makefile.am:60: warning: wildcard $(srcdir: non-POSIX variable name
test/fw/Makefile.am:60: (probably a GNU make extension)
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/fw/Makefile.am:44:   while processing 'dist_ptlpkg_bin_SCRIPTS'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/fw/Makefile.am:52:   while processing 'dist_ptlpkg_pylib_lib_PYTHON'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/fw/Makefile.am:60:   while processing 'dist_ptlpkg_pylib_plugins_PYTHON'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/fw/Makefile.am:48:   while processing 'dist_ptlpkg_pylib_top_PYTHON'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/fw/Makefile.am:56:   while processing 'dist_ptlpkg_pylib_utils_PYTHON'
test/tests/Makefile.am:39: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:39: (probably a GNU make extension)
test/tests/Makefile.am:42: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:42: (probably a GNU make extension)
test/tests/Makefile.am:45: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:45: (probably a GNU make extension)
test/tests/Makefile.am:48: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:48: (probably a GNU make extension)
test/tests/Makefile.am:51: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:51: (probably a GNU make extension)
test/tests/Makefile.am:54: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:54: (probably a GNU make extension)
test/tests/Makefile.am:57: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:57: (probably a GNU make extension)
test/tests/Makefile.am:60: warning: wildcard $(srcdir: non-POSIX variable name
test/tests/Makefile.am:60: (probably a GNU make extension)
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:39:   while processing 'dist_ptl_tests_PYTHON'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:42:   while processing 'dist_ptl_testfunctional_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:45:   while processing 'dist_ptl_testinterfaces_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:48:   while processing 'dist_ptl_testperformance_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:51:   while processing 'dist_ptl_testresilience_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:54:   while processing 'dist_ptl_testsecurity_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:57:   while processing 'dist_ptl_testselftest_DATA'
automake: warning: wildcard $(srcdir: non-POSIX variable name
automake: (probably a GNU make extension)
test/tests/Makefile.am:60:   while processing 'dist_ptl_testupgrades_DATA'

Could this be where it first starts?

[mike0042]

Hello @thomaskwok91, these warnings may safely be ignored. Automake is complaining that we are using a GNU extension (wildcard) in test/tests/Makefile.am and test/fw/Makefile.am. We are, in fact, using that extension so automake is right to complain. On other non-Linux platforms (e.g. NetBSD) this could cause an incompatibility. There was a discussion when this was added and we decided that the warnings were less important than forcing test developers to update the Makefile.am every time a new test file is added. Hence, you see these warnings.

Please try running configure and make per the installation instructions. If you see errors from either of those commands, please provide them here.

[thomaskwok91]

The configure command looks fine, don't see anything wrong. Is there a way to debug while running it so those messages stand out? Or are installed somewhere?

For make, this is the first set of error that I think I encounter. There are a few more like this.

pbsd_init.c: In function ‘call_log_license’:
pbsd_init.c:2254:3: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
   (void)write(fd, &usedlicenses, sizeof(usedlicenses));
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mv -f .deps/pbs_server_bin-pbsd_init.Tpo .deps/pbs_server_bin-pbsd_init.Po
gcc -DHAVE_CONFIG_H -I. -I../../src/include  -I../../src/include -I/usr/include/postgresql -I/usr/include/python2.7 -I/usr/include/python2.7   -g -O2 -MT pbs_server_bin-pbsd_main.o -MD -MP -MF .deps/pbs_server_bin-pbsd_main.Tpo -c -o pbs_server_bin-pbsd_main.o `test -f 'pbsd_main.c' || echo './'`pbsd_main.c
pbsd_main.c: In function ‘main’:
pbsd_main.c:1907:5: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result [-Wunused-result]
     (void)system(schedcmd);
     ^~~~~~~~~~~~~~~~~~~~~~
pbsd_main.c: In function ‘lock_out’:
pbsd_main.c:2484:5: warning: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result [-Wunused-result]
     (void)ftruncate(fds, (off_t)0);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pbsd_main.c:2486:5: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
     (void)write(fds, buf, strlen(buf));
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mv -f .deps/pbs_server_bin-pbsd_main.Tpo .deps/pbs_server_bin-pbsd_main.Po
gcc -DHAVE_CONFIG_H -I. -I../../src/include  -I../../src/include -I/usr/include/postgresql -I/usr/include/python2.7 -I/usr/include/python2.7   -g -O2 -MT pbs_server_bin-svr_migrate_data.o -MD -MP -MF .deps/pbs_server_bin-svr_migrate_data.Tpo -c -o pbs_server_bin-svr_migrate_data.o `test -f 'svr_migrate_data.c' || echo './'`svr_migrate_data.c
svr_migrate_data.c: In function ‘svr_migrate_data_from_fs’:
svr_migrate_data.c:393:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:403:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:423:6: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
      chdir(origdir);
      ^~~~~~~~~~~~~~
svr_migrate_data.c:433:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:446:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:457:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:465:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:483:6: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
      chdir(origdir);
      ^~~~~~~~~~~~~~
svr_migrate_data.c:493:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:504:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:614:5: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
     chdir(origdir);
     ^~~~~~~~~~~~~~
svr_migrate_data.c:633:6: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
      chdir(origdir);
      ^~~~~~~~~~~~~~
svr_migrate_data.c:650:4: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
    chdir(origdir);
    ^~~~~~~~~~~~~~
svr_migrate_data.c:663:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:669:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   chdir(origdir);
   ^~~~~~~~~~~~~~
svr_migrate_data.c:672:2: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
  chdir(origdir);
  ^~~~~~~~~~~~~~

[mike0042]

I should have asked this earlier... what OS are you using? I'm guessing it's something recently released, like CentOS 8. The gcc compiler continues to get more picky as it ages. My dev VM looks like this...

$ lsb_release -d
Description:    CentOS Linux release 7.6.1810 (Core) 
$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[thomaskwok91]

I am using Ubuntu 18.04.3 LTS and my gcc is Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

[mike0042]

I just fired up my Ubuntu VM and ran through a complete build and install. This is from the system I was using:

root@ubuntu:~/pbspro-19.1.2# lsb_release -d
Description:	Ubuntu 18.04.3 LTS
root@ubuntu:~/pbspro-19.1.2# gcc --version
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The warnings that you are seeing are due to a more recent version of gcc than was used when PBS Pro 19.1.2 was released. I took a look through them and they are innocuous. Did your build complete?

[thomaskwok91]

I did not get an error message that said the build did not finish so I assume it completed, is there a progress report I can pull up somewhere? I think the make and make install completed but for some reason my computer won't switch the library for qstat.

May be a dumb question but when you run which qstat do you get the correct directory when you tried to build and install?

[mike0042]

I think you have some remnants of a former SLURM installation in /usr that is being picked up in your PATH. The script /etc/profile.d/pbs.csh appends /opt/pbs/bin to your PATH when you log in. It's likely that /usr/bin appears in your PATH before /opt/pbs/bin causing /usr/bin/qstat to be called rather than /opt/pbs/bin/qstat.

The best thing would be to clean up any remaining remnants from the SLURM installation. Then your shell should find qstat in /opt/pbs/bin. Otherwise, you can put /opt/pbs/bin ahead of /usr/bin in your PATH. Lastly, you could use the full path to qstat to invoke it as /opt/pbs/bin/qstat.

[thomaskwok91]

It is odd because it is installed. If I run

sudo /opt/pbs/libexec/pbs_postinstall this is what I get

*** PBS Installation Summary
***
*** Postinstall script called as follows:
*** /opt/pbs/libexec/pbs_postinstall '' 
***
*** Existing configuration file found: /etc/pbs.conf
***
*** Saving /etc/pbs.conf as /etc/pbs.conf.pre.19.1.2.20191002194739
*** Replacing /etc/pbs.conf with /etc/pbs.conf.19.1.2
*** /etc/pbs.conf has been modified.
*** The original contents have been saved to /etc/pbs.conf.pre.19.1.2.20191002194739
***
*** Registering PBS Pro as a service.
Synchronizing state of pbs.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pbs
***
*** PBS_HOME is /var/spool/pbs
*** Existing environment file left unmodified: /var/spool/pbs/pbs_environment
***
*** The PBS Pro server has been installed in /opt/pbs/sbin.
*** The PBS Pro scheduler has been installed in /opt/pbs/sbin.
***
*** The PBS Pro communication agent has been installed in /opt/pbs/sbin.
***
*** The PBS Pro MOM has been installed in /opt/pbs/sbin.
***
*** The PBS commands have been installed in /opt/pbs/bin.
***

and when I run sudo /etc/init.d/pbs start

I get

Starting PBS
Warning: PBS Professional has detected core file(s) in PBS_HOME that require attention!!!
Warning: Please inform your administrator immediately or contact Altair customer support
PBS comm already running.
PBS mom already running.
PBS scheduler already running.
PBS Server already running.

So it is installed, for some reason qstat just doesn't switch over.

[thomaskwok91]

I purged all remnants of slurm before I made my previous post but the same error still persisted. When I look at the pbs.csh file, this is the output.

if ( $?PBS_CONF_FILE ) then
   set conf = "$PBS_CONF_FILE"
else
   set conf = /etc/pbs.conf
endif

if ( -r "$conf" ) then
   setenv __PBS_EXEC `grep '^[[:space:]]*PBS_EXEC=' "$conf" | tail -1 | sed 's/^[[:space:]]*PBS_EXEC=\([^[:space:]]*\)[[:space:]]*/\1/'`
   if ( $?__PBS_EXEC ) then
      # Define the PATH and MANPATH for the users
      if ( $?PATH && -d ${__PBS_EXEC}/bin ) then
         setenv PATH ${PATH}:${__PBS_EXEC}/bin
      endif
      if ( $?MANPATH && -d ${__PBS_EXEC}/man ) then
         setenv MANPATH ${MANPATH}:${__PBS_EXEC}/man
      endif
      if ( $?MANPATH && -d ${__PBS_EXEC}/share/man ) then
         setenv MANPATH ${MANPATH}:${__PBS_EXEC}/share/man
      endif
      if ( `whoami` == "root" ) then
         if ( $?PATH && -d ${__PBS_EXEC}/sbin ) then
            setenv PATH ${PATH}:${__PBS_EXEC}/sbin
         endif
      endif
      unsetenv __PBS_EXEC
   endif
endif

[mike0042]

The PBS Pro init scripts don't rely on PATH being set properly. They take the values from /etc/pbs.conf and use the full path names of each binary when starting the services. The core files are a concern, but that could have been the result of something being configured improperly. It does appear that the services actually started, so that's good news. Look at the timestamp on /usr/bin/qstat to figure out when it was installed. There are likely a bunch of other files with similar timestamps that were installed along with qstat. That should give you an indication of what needs to be removed.

If it were my system and I hadn't spent a significant amount of time configuring it, I'd start with a fresh instance of Ubuntu. You can also generate a .deb package rather than using "make install". First generate an RPM using the instructions here:
https://pbspro.atlassian.net/wiki/spaces/PBSPro/pages/13991940/Building+PBS+Pro+Using+rpmbuild

Thank use alien to create the .deb package as we do for our TravisCI tests:
https://github.com/PBSPro/pbspro/blob/release_19_1_branch/.travis/ubuntu_18_04.sh

[mike0042]

Once you figure out how /usr/bin/qstat got installed I think you'll be headed in the right direction if you want to salvage your current Ubuntu installation.

[thomaskwok91]

I looked and it seems that /usr/bin/qstat was installed February 5 2018 which is interesting as we had purchased this machine this year. Although there are too many changes to reinstall from scratch.

[mike0042]

What happens if you run the following:

dpkg -S /usr/bin/qstat

[mike0042]

Nice work @thomaskwok91!