Result statistics broken in new version of dbt3-post-process

[olepbr]

Result statistics broken in new version of dbt3-post-process

[markwkm]

Fixed in 66bead4 and it's in v2.0.5.

[markwkm]

Hi!

I haven't spent enough time to figure out where the bug is, but to restate what I think the facts are, STREAMS (the plural) should be 2 and STREAM (singular) should be the current stream (1 or 2). My initial reaction is that line 217 isn't wrong, but perhaps there is a typo elsewhere so that STREAMS is getting clobbered?

I'll hopefully have more time to look next week, but I hope that might be helpful if that might illuminate where the source of the issue is...

[olepbr]

I haven't spent enough time to figure out where the bug is, but to restate what I think the facts are, STREAMS (the plural) should be 2 and STREAM (singular) should be the current stream (1 or 2). My initial reaction is that line 217 isn't wrong, but perhaps there is a typo elsewhere so that STREAMS is getting clobbered?

Sorry, I was perhaps a bit unclear in my initial report. STREAMand STREAMS are both fine (and I agree with your definitions of them), but the error isn't on line 217, it's on 207 (and 184 and 162 for the other aggregates). The problem is that we can't set STRING using STREAM (singular) when computing the aggregates, because then the shell pipe only gets one line to operate on (that of the last stream) instead of a set of lines (one per stream).

I'll hopefully have more time to look next week, but I hope that might be helpful if that might illuminate where the source of the issue is...

No worries! FWIW, I tested changing the three mentioned lines to my proposed regex and the resulting statistics seem correct to me :-)

[markwkm]

Aha, please go ahead and open a PR.

[markwkm]

I finally made more time to look and I understand now. So min and max were broken too when calculating the query stats. I took a slightly different approach since there could be more than 9 streams. And I tightened up the regex for the refresh stream calculations too. I'll make time soon to push a new release.

diff --git a/scripts/dbt3-post-process.in b/scripts/dbt3-post-process.in
index cf27b0b..3cb48ef 100755
--- a/scripts/dbt3-post-process.in
+++ b/scripts/dbt3-post-process.in
@@ -155,11 +155,11 @@ throughput_query() {
    printf "     Min"
    for Q in $(seq "${Q0}" "${Q1}"); do
        if [ "${Q}" -eq 23 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF1,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF1,"
        elif [ "${Q}" -eq 24 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF2,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF2,"
        else
-           STRING="^PERF\.THRUPUT\.QS${STREAM}\.Q${Q},"
+         STRING="^PERF\.THRUPUT\.QS[0-9]\+\.Q${Q},"
        fi

        if [ $HTML -eq 1 ]; then
@@ -177,11 +177,11 @@ throughput_query() {
    printf "     Max"
    for Q in $(seq "${Q0}" "${Q1}"); do
        if [ "${Q}" -eq 23 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF1,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF1,"
        elif [ "${Q}" -eq 24 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF2,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF2,"
        else
-           STRING="^PERF\.THRUPUT\.QS${STREAM}\.Q${Q},"
+         STRING="^PERF\.THRUPUT\.QS[0-9]\+\.Q${Q},"
        fi

        if [ $HTML -eq 1 ]; then
@@ -199,11 +199,11 @@ throughput_query() {
    printf "     Avg"
    for Q in $(seq "${Q0}" "${Q1}"); do
        if [ "${Q}" -eq 23 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF1,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF1,"
        elif [ "${Q}" -eq 24 ]; then
-           STRING="^PERF\.THRUPUT\.RFST.*\.RF2,"
+         STRING="^PERF\.THRUPUT\.RFST[0-9]\+\.RF2,"
        else
-           STRING="^PERF\.THRUPUT\.QS${STREAM}\.Q${Q},"
+         STRING="^PERF\.THRUPUT\.QS[0-9]\+\.Q${Q},"
        fi

        if [ $HTML -eq 1 ]; then

[olepbr]

Thank you so much for fixing this! Sorry for the poor response time on my end, scrambling to finish my master's…