500 Internal Server Error:

[ctxis]

500 Internal Server Error:

[sadreck]

Can you post any of the errors you see in syslog? That would help a lot to get to the bottom of this issue.

In the meantime, can you check if the /etc/environment file is empty? If it's empty, try replacing it with /etc/profile (in the service).

crackerjack/setup/ansible/roles/service/templates/ubuntu/systemd.j2

Line 9 in 6920304

EnvironmentFile=/etc/environment

This is the case between Debian and Ubuntu, but I've personally never used Manjiro so this could be it.

[HaylockGrant]

@sadreck
Aren't the journalctl logs I posted the system logs?
I was unable to locate either /etc/environment/ or /etc/profile/ . I do have /etc/profile.d/ , however when I changed the service to use profile.d nothing changed.

[sadreck]

You can get the exact errors if you run tail -f /var/log/syslog and then visit the page.

Also, the /etc/environment and /etc/profile are files, not folders - https://wiki.archlinux.org/title/environment_variables#Globally

[HaylockGrant]

@sadreck
both /etc/environment and /etc/profile are files. neither seems to be fixing the issue

/etc/environment

#
# This file is parsed by pam_env module
#
# Syntax: simple "KEY=VAL" pairs on separate lines
#

/etc/profile

# /etc/profile

# Set our umask
umask 022

# Append "$1" to $PATH when not already in.
# This function API is accessible to scripts in /etc/profile.d
append_path () {
    case ":$PATH:" in
        *:"$1":*)
            ;;
        *)
            PATH="${PATH:+$PATH:}$1"
    esac
}

# Append our default paths
append_path '/usr/local/sbin'
append_path '/usr/local/bin'
append_path '/usr/bin'

# Force PATH to be environment
export PATH

# Load profiles from /etc/profile.d
if test -d /etc/profile.d/; then
        for profile in /etc/profile.d/*.sh; do
                test -r "$profile" && . "$profile"
        done
        unset profile
fi

# Unload our profile API functions
unset -f append_path

# Source global bash config, when interactive but not posix or sh mode
if test "$BASH" &&\
   test "$PS1" &&\
   test -z "$POSIXLY_CORRECT" &&\
   test "${0#-}" != sh &&\
   test -r /etc/bash.bashrc
then
        . /etc/bash.bashrc
fi

# Termcap is outdated, old, and crusty, kill it.
unset TERMCAP

# Man is much better than us at figuring this out
unset MANPATH

Manjaro logs might be a bit more confusing than ubuntu logs. Unfortunately was unable to locate the log file, however, I was able to get journalctl to show me the entire systems logs and I filtered it by time.

crackerjack service started using sudo systemctl start crackerjack.service

Dec 04 14:21:25 SS-Server dbus-daemon[530]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.617' (uid=0 pid=165779 comm="sudo systemctl start crackerjack.service")
Dec 04 14:21:25 SS-Server dbus-daemon[530]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Dec 04 14:21:25 SS-Server sudo[165779]: pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Dec 04 14:21:25 SS-Server audit[165779]: USER_ACCT pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="spk" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server sudo[165779]:      spk : TTY=pts/2 ; PWD=/home/spk ; USER=root ; COMMAND=/usr/bin/systemctl start crackerjack.service
Dec 04 14:21:25 SS-Server audit[165779]: CRED_REFR pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server sudo[165779]: pam_unix(sudo:session): session opened for user root(uid=0) by spk(uid=1000)
Dec 04 14:21:25 SS-Server audit[165779]: USER_START pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:session_open grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server kernel: audit: type=1101 audit(1670185285.974:1380): pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:accounting grantors=pam_unix,pam_permit,pam_time acct="spk" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server kernel: audit: type=1110 audit(1670185285.974:1381): pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server kernel: audit: type=1105 audit(1670185285.974:1382): pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:session_open grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server systemd[1]: crackerjack.service: Ignoring invalid environment assignment 'test "${0#-}" !=sh &&   test -r /etc/bash.bashrc': /etc/profile
Dec 04 14:21:25 SS-Server systemd[1]: Started CrackerJack Gunicorn.
░░ Subject: A start job for unit crackerjack.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://forum.manjaro.org/c/support
░░
░░ A start job for unit crackerjack.service has finished successfully.
░░
░░ The job identifier is 6331.
Dec 04 14:21:25 SS-Server audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=crackerjack comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 04 14:21:25 SS-Server sudo[165779]: pam_unix(sudo:session): session closed for user root
Dec 04 14:21:25 SS-Server kernel: audit: type=1130 audit(1670185285.994:1383): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=crackerjack comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 04 14:21:25 SS-Server kernel: audit: type=1106 audit(1670185285.994:1384): pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server kernel: audit: type=1104 audit(1670185285.994:1385): pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server audit[165779]: USER_END pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:25 SS-Server audit[165779]: CRED_DISP pid=165779 uid=1000 auid=1000 ses=9 subj=unconfined msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
Dec 04 14:21:26 SS-Server gunicorn[165782]: [2022-12-04 14:21:26 -0600] [165782] [INFO] Starting gunicorn 20.1.0
Dec 04 14:21:26 SS-Server gunicorn[165782]: [2022-12-04 14:21:26 -0600] [165782] [INFO] Listening at: http://0.0.0.0:5000 (165782)
Dec 04 14:21:26 SS-Server gunicorn[165782]: [2022-12-04 14:21:26 -0600] [165782] [INFO] Using worker: sync
Dec 04 14:21:26 SS-Server gunicorn[165783]: [2022-12-04 14:21:26 -0600] [165783] [INFO] Booting worker with pid: 165783
Dec 04 14:21:26 SS-Server gunicorn[165784]: [2022-12-04 14:21:26 -0600] [165784] [INFO] Booting worker with pid: 165784
Dec 04 14:21:26 SS-Server gunicorn[165785]: [2022-12-04 14:21:26 -0600] [165785] [INFO] Booting worker with pid: 165785

Logged into the UI

Dec 04 14:23:02 SS-Server gunicorn[165785]: [2022-12-04 14:23:02,077] ERROR in app: Exception on /auth/login [POST]
Dec 04 14:23:02 SS-Server gunicorn[165785]: Traceback (most recent call last):
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/app.py", line 2447, in wsgi_app
Dec 04 14:23:02 SS-Server gunicorn[165785]:     response = self.full_dispatch_request()
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/app.py", line 1952, in full_dispatch_request
Dec 04 14:23:02 SS-Server gunicorn[165785]:     rv = self.handle_user_exception(e)
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/app.py", line 1821, in handle_user_exception
Dec 04 14:23:02 SS-Server gunicorn[165785]:     reraise(exc_type, exc_value, tb)
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/_compat.py", line 39, in reraise
Dec 04 14:23:02 SS-Server gunicorn[165785]:     raise value
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/app.py", line 1950, in full_dispatch_request
Dec 04 14:23:02 SS-Server gunicorn[165785]:     rv = self.dispatch_request()
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/venv/lib/python3.10/site-packages/flask/app.py", line 1936, in dispatch_request
Dec 04 14:23:02 SS-Server gunicorn[165785]:     return self.view_functions[rule.endpoint](**req.view_args)
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/app/controllers/auth.py", line 90, in login_process
Dec 04 14:23:02 SS-Server gunicorn[165785]:     system.run_updates()
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/app/lib/base/system.py", line 14, in run_updates
Dec 04 14:23:02 SS-Server gunicorn[165785]:     self.update_git_hash_version()
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/app/lib/base/system.py", line 42, in update_git_hash_version
Dec 04 14:23:02 SS-Server gunicorn[165785]:     git_binary = self.shell.execute(['which', 'git'], user_id=0)
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/home/spk/crackerjack/app/lib/base/shell.py", line 21, in execute
Dec 04 14:23:02 SS-Server gunicorn[165785]:     output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode().strip()
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/usr/lib/python3.10/subprocess.py", line 503, in run
Dec 04 14:23:02 SS-Server gunicorn[165785]:     with Popen(*popenargs, **kwargs) as process:
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
Dec 04 14:23:02 SS-Server gunicorn[165785]:     self._execute_child(args, executable, preexec_fn, close_fds,
Dec 04 14:23:02 SS-Server gunicorn[165785]:   File "/usr/lib/python3.10/subprocess.py", line 1847, in _execute_child
Dec 04 14:23:02 SS-Server gunicorn[165785]:     raise child_exception_type(errno_num, err_msg, err_filename)
Dec 04 14:23:02 SS-Server gunicorn[165785]: FileNotFoundError: [Errno 2] No such file or directory: 'which'

If you need additional log files let me know which of these files would contain what you're looking for

$ cd /var/log
$ sudo tree
.
├── audit
├── btmp
├── btmp.1
├── cups
│   ├── access_log
│   ├── access_log.1
│   ├── access_log.2
│   └── access_log.3
├── gssproxy
├── journal
│   ├── d4e2fbedce3b41399826381095d9aabc
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── system@816d0040a8fe443484ef3eb6ff906da2-0000000000000001-0005ea139bfe0803.journal
│   │   ├── system.journal
│   │   ├── [email protected]~
│   │   ├── [email protected]~
│   │   ├── user-1000@7b8953e13be94463aceb4cf6bfb34a09-00000000000016de-0005e9dcf24e045e.journal
│   │   └── user-1000.journal
│   └── remote
├── lastlog
├── lightdm
│   ├── lightdm.log
│   ├── lightdm.log.old
│   ├── seat0-greeter.log
│   ├── seat0-greeter.log.old
│   ├── x-0.log
│   └── x-0.log.old
├── mysql.log
├── nvidia-installer.log
├── old
├── pacman.log
├── private
├── samba
├── supervisor
├── wtmp
├── Xorg.0.log
└── Xorg.0.log.old

[sadreck]

This is what wanted to check:

FileNotFoundError: [Errno 2] No such file or directory: 'which'

This indicates that the service is not loading the PATH variable, so when crackerjack is running which git (for example) it fails.

In Ubuntu the file /etc/environment contains the following:

$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

But on Debian you have to use /etc/profile. I'm not familiar with Manjaro Linux, but it should have a similar file.

If you can't find it, you can always just create (or link if it's under another name) that file and export the PATH variable in it?

[HaylockGrant]

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
was appended to /etc/environment and crackerjack service was adjusted to use /etc/environment again. UI seems to be working now.
Thanks for the help

[sadreck]

No problem, glad we could get it to work!