Running bash over ssh with different results from local and remote run

[ohwgiles]

Running bash over ssh with different results from local and remote run

[ohwgiles]

But be careful, since you changed "EOF" to EOF, also the $? variables will be expanded before the script is passed to the server. You need to escape it like \$? or go back to "EOF"

[ohwgiles]

The difference is, test.sh on the remote server has the shebang #!/bin/bash, while this line on laminar

ssh [email protected] "env C_NAME=$C_NAME /bin/bash -xe"<<"EOF"

executes under /bin/bash -xe. -x causes each line to be printed before its executed (that's what causes all the lines beginning with +), and -e causes the script to abort on the first command that returns a non-zero exit status. See Advanced Bash Scripting Guide: Options.

Since the ping command isn't executed, I guess lxc exec pkg-builder-37 ip a add 10.214.101.2/24 dev eth0 is returning non-zero. On your test.sh, this is ignored and the script continues.

If you don't care if it returns non-zero, you can either remove the -e from the bash invocation, or change the line to something like lxc exec pkg-builder-37 ip a add 10.214.101.2/24 dev eth0 || true

By the way, if you don't need internal variables inside the heredoc, you might consider changing "EOF" to EOF. This would expand the variables before passing the script to the server and allow you to remove env C_NAME=$C_NAME

[palica]

So I changed the script now to:

#!/bin/bash

echo This script should start a container build a package and report back.

echo We are going to start container:
C_NAME=pkg-builder-$RUN
echo $C_NAME

echo Connecting to remote host and starting container

ssh [email protected] /bin/bash -xe << EOF
        echo $C_NAME
        lxc launch -e -p prf-funtoo -p default fun-generic "$C_NAME"
        sleep 3
        lxc exec "$C_NAME" ip a add 10.214.101.2/24 dev eth0
        echo $?
        lxc exec "$C_NAME" -- ping -c3 10.214.101.1
        echo $?
        echo "Waiting 3 seconds"
        sleep 3
        lxc exec "$C_NAME" -- poweroff
EOF

Laminar:

[laminar] Executing /var/lib/laminar/cfg/jobs/pkg-builder.run
This script should start a container build a package and report back.
We are going to start container:
pkg-builder-40
Connecting to remote host and starting container
pkg-builder-40
+ echo pkg-builder-40
+ lxc launch -e -p prf-funtoo -p default fun-generic pkg-builder-40
Creating pkg-builder-40


Starting pkg-builder-40
+ sleep 3
+ lxc exec pkg-builder-40 ip a add 10.214.101.2/24 dev eth0

Using remote script:
Job:

#!/bin/bash
echo Running on robor ...        
ssh [email protected] /bin/bash -xe /root/test.sh
echo Done

remote script:

#!/bin/bash   

        C_NAME=pkg-builder-1
        echo $C_NAME
        lxc launch -e -p prf-funtoo -p default fun-generic "$C_NAME"
        sleep 3
        lxc exec "$C_NAME" ip a add 10.214.101.2/24 dev eth0
        echo $?
        lxc exec "$C_NAME" -- ping -c3 10.214.101.1
        echo $?
        echo "Waiting 3 seconds"
        sleep 3
        lxc exec "$C_NAME" -- poweroff

output:

[laminar] Executing /var/lib/laminar/cfg/jobs/pkg-builder-remote.run
Running on robor ...
pkg-builder-1
+ C_NAME=pkg-builder-1
+ echo pkg-builder-1
+ lxc launch -e -p prf-funtoo -p default fun-generic pkg-builder-1
Creating pkg-builder-1


Starting pkg-builder-1
+ sleep 3
+ lxc exec pkg-builder-1 ip a add 10.214.101.2/24 dev eth0
0
+ echo 0
+ lxc exec pkg-builder-1 -- ping -c3 10.214.101.1
PING 10.214.101.1 (10.214.101.1) 56(84) bytes of data.
64 bytes from 10.214.101.1: icmp_seq=1 ttl=64 time=0.175 ms
64 bytes from 10.214.101.1: icmp_seq=2 ttl=64 time=0.067 ms
64 bytes from 10.214.101.1: icmp_seq=3 ttl=64 time=0.062 ms

--- 10.214.101.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2077ms
rtt min/avg/max/mdev = 0.062/0.101/0.175/0.052 ms
0
Waiting 3 seconds
+ echo 0
+ echo 'Waiting 3 seconds'
+ sleep 3
+ lxc exec pkg-builder-1 -- poweroff
Done

Hm. the ip a add command returns 0 when running from remote script. I don't really understand. Thanks for the tips about bash -xe and EOF.

[ohwgiles]

Yeah that is weird. Can you try removing the -e from ssh [email protected] /bin/bash -xe just to prove that this is the issue, and if so, the error code it returns might help diagnose the root cause

[palica]

Same output, no error code.

[ohwgiles]

Hmm, I'm not sure how else to help. Did you try executing the laminar .run file directly yourself (not via laminar)? Or try the heredoc directly in a console?

[palica]

Direct run also doesn't work. Oh, well not too bad. I will workaround with the remote script that works well so far. Thank you for support.

[ohwgiles]

If you eventually figure this out I'd be interested in the solution. It's apparently not a laminar issue but since laminar encourages scripts as heredocs I can imagine this coming up again.

By the way, thanks very much for all your detailed issues, this is great QA and I think has really helped improve laminar.

[palica]

You are welcome, I didn't want to a be a big pain in the ...

ad heredoc: I will if I find a solution let you know. One thing that I also noticed eg. from your example on Remote jobs is that you have to use #!/bin/bash -e on your .run script or else the job will always succeed. So might want to put that somewhere in the docs, that the scripts have to provide a valid exit code when something goes wrong.

Thank you for you help and also for fixing and improving your very nice software.

[ohwgiles]

With 792c69a I added a note about -e under Creating a job