performance issues

[mgrossmann]

performance issues

[IgorTodorovskiIBM]

Thanks! There is currently at issue that prevents us from building at O3. We're building with no optimization at the moment. That you for the analysis and numbers! We will work on this soon!

[mgrossmann]

Hello Igor,

I don‘t think this problem is related to compiler optimizations, only.
During my tests I also compared IBMs tcsh to IBMs sh.

The tcsh as slow as the different bash version, nearly.

While running the test script with different shells, I put my eyes into SDSF.

One thing I saw is much higher SIO values when running the faster sh.
Bash and tcsh had much lower values
at SIO.

So the problem may IO related?

Thanks Mike

[MikeFultonDev]

time to dig into that OPT bug i guess. Fun :)

[MikeFultonDev]

@mgrossmann i wonder if you are always getting the builtins for echo and cut and not 'failing over' to the coreutils versions?

[mgrossmann]

I can't tell you that.
How can I check?

But I think that it is actually not
about echo or cut. Tomorrow I will test another script again.

I noticed the performance problem with our build system. A number of bash scripts around make. The whole thing is very very sluggish.

Best, Mike

[MikeFultonDev]

one thought - change the script to say /bin/echo (say) and see if the perf changes?

[mgrossmann]

Will do, tomorrow.

Thank you.

[MikeFultonDev]

not sure if you're interested @mgrossmann but I would expect rebuilding at -O3 will make a huge perf difference for bash. Most of our tools are built with -O3 but we hit problems with bash and temporarily turned off optimization. Not the simplest problem in the world to track down, but a binary search approach is what I've used in the past. Break the set of compiled files in half and build half at opt, half not. Keep repeating until you isolate the file that, building at OPT fails, then you've narrowed it down to that source file (or files?). Then you can try functions although that is trickier because the optimizer can inline functions causing problems.

[mgrossmann]

Hello Mike,

I ran some more tests today.

Regarding the built-in functions topic, with enable -n echo you can control whether the built-in function should be used or not. In fact, from my point of view, the effect is not significant.

Assuming that Rocket builds their V4 with optimisations enabled, I have carried out further tests.
I explicitly compared Rocket's V4 with "our" V5. I can see slight differences here.
But even these are not as significant as the differences between SH and BASH.

For me, it seems much more as if the "sub-shelling" is the root of the problem.
I have used a slightly adapted shell script here. Comparing out=`echo ...` with out=`bash -c echo ...` showed a surprising result for me. The call of bash -c was faster than the one without.

Using SH in shebang and simple backticks for sub-shelling

#!/bin/sh
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`echo "Some test data" | cut -c1`
        count=$((count+1))
done

Result:

real    0m1.692s
user    0m0.365s
sys     0m0.122s

Using SH in shebang and sh -c in backticks for sub-shelling

#!/bin/sh
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`/bin/sh -c "echo "Some test data" | cut -c1"`
        count=$((count+1))
done

Result:

real    0m1.224s
user    0m0.307s
sys     0m0.102s

Using SH in shebang and bash -c in backticks for sub-shelling

#!/bin/sh
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`/bin/bash -c "echo "Some test data" | cut -c1"`
        count=$((count+1))
done

Result (V4):

real    0m4.770s
user    0m0.745s
sys     0m0.248s

Result (V5):

real    0m7.992s
user    0m1.203s
sys     0m0.401s

---

Using BASH in shebang and simple backticks for sub-shelling

#!/bin/bash
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`echo "Some test data" | cut -c1`
        count=$((count+1))
done

Result (V4):

real    0m8.112s
user    0m2.593s
sys     0m0.864s

Result (V5):

real    0m12.309s
user    0m4.288s
sys     0m1.429s

Using BASH in shebang and sh -c in backticks for sub-shelling

#!/bin/bash
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`/bin/sh -c "echo "Some test data" | cut -c1"`
        count=$((count+1))
done

Result (V4):

real    0m3.628s
user    0m1.018s
sys     0m0.339s

Result(V5):

real    0m5.038s
user    0m1.495s
sys     0m0.498s

Using BASH in shebang and bash -c in backticks for sub-shelling

#!/bin/bash
count=1

#enable -n echo

while [ ${count} -le 50 ]
do
        out=`/bin/bash -c "echo "Some test data" | cut -c1"`
        count=$((count+1))
done

Result (V4):

real    0m7.461s
user    0m1.480s
sys     0m0.493s

Result (V5):

real    0m10.919s
user    0m2.475s
sys     0m0.825s

---

On the subject of optimizations or general contribution. Unfortunately, I can't set up a development environment on our system. If you could give me access to the development system mentioned yesterday, I could well imagine contributing.

Regards, Mike

[MikeFultonDev]

Nice progress. Yes - we can set up an account for you. @IgorTodorovskiIBM can you cover that since I am out on business?

[IgorTodorovskiIBM]

Sure!

@mgrossmann, can you send me a public ssh key and an 8 character userid at [email protected]?

You can use ssh-keygen -t rsa -b 4096 -C "email id" -m PEM on Linux/Window/Mac to create it.

[mgrossmann]

Sent you a mail, Igor.