[Bug] Automated Rules fail to activate/deactivate

[andrewazores]

[Bug] Automated Rules fail to activate/deactivate

[mwangggg]

The archived recordings for that target looked to be as expected so I assume it's not stopping the activation from continuing.

[andrewazores]

@mwangggg:

https://github.com/cryostatio/cryostat/blob/2b0cd5adacf8f928f8666f36663b0a5b31e0e76b/src/main/java/io/cryostat/configuration/Variables.java#LL68C37-L68C37

cryostat/run.sh

Line 138 in 2b0cd5a

-e CRYOSTAT_PLATFORM="$CRYOSTAT_PLATFORM" \

Let's try this out next. The patch above prevents the vertx-fib-demo applications' Agents from connecting and registering themselves as target applications. If you need to apply that patch, you can do:

1. git checkout smoketest.sh first to discard any other changes you may have, if necessary
2. Copy and paste the patch above into a file p.patch
3. patch -p1 < p.patch

If you also do CRYOSTAT_PLATFORM=io.cryostat.platform.internal.DefaultPlatformStrategy sh smoketest.sh (that comes from https://github.com/cryostatio/cryostat/blob/main/src/main/java/io/cryostat/platform/internal/DefaultPlatformStrategy.java), this should disable the Podman API discovery so you won't have the Podman Realm either. After doing these steps, I have a discovery tree like this:

$ https --auth=user:pass :8181/api/v2.1/discovery
HTTP/1.1 200 OK
content-encoding: gzip
content-length: 911
content-type: application/json

{
    "data": {
        "result": {
            "children": [
                {
                    "children": [
                        {
                            "id": 1856770420,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:10000/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "/deployments/quarkus-run.jar",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "/deployments/quarkus-run.jar",
                                        "PORT": "10000",
                                        "REALM": "JDP"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:10000/jmxrmi",
                                "jvmId": "WPXVyIdK47YXaFuRwcwlFiSk3nU92SxT031e29_4ljc=",
                                "labels": {}
                            }
                        },
                        {
                            "id": 1635587257,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "io.cryostat.Cryostat",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "io.cryostat.Cryostat",
                                        "PORT": "9091",
                                        "REALM": "JDP"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
                                "jvmId": null,
                                "labels": {}
                            }
                        },
                        {
                            "id": -435260576,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "es.andrewazor.demo.Main",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "es.andrewazor.demo.Main",
                                        "PORT": "9093",
                                        "REALM": "JDP"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi",
                                "jvmId": "iU3TUGe3lc3wnCIwUThJauNNgmDTFsyT8ukEMrWsuvQ=",
                                "labels": {}
                            }
                        },
                        {
                            "id": 2019498512,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:9094/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "es.andrewazor.demo.Main",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "es.andrewazor.demo.Main",
                                        "PORT": "9094",
                                        "REALM": "JDP"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9094/jmxrmi",
                                "jvmId": null,
                                "labels": {}
                            }
                        },
                        {
                            "id": 1647692055,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:9095/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "es.andrewazor.demo.Main",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "es.andrewazor.demo.Main",
                                        "PORT": "9095",
                                        "REALM": "JDP"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9095/jmxrmi",
                                "jvmId": null,
                                "labels": {}
                            }
                        }
                    ],
                    "id": 1301967007,
                    "labels": {
                        "REALM": "fd4bd7d5-0fef-4234-b5ea-3fc5b0c56cf2"
                    },
                    "name": "JDP",
                    "nodeType": "Realm"
                },
                {
                    "children": [
                        {
                            "id": -432899869,
                            "labels": {},
                            "name": "service:jmx:rmi:///jndi/rmi://cryostat:9097/jmxrmi",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "quarkus-test-agent",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "/deployments/quarkus-run.jar",
                                        "PID": "1",
                                        "PORT": "-1",
                                        "REALM": "quarkus-test-agent",
                                        "START_TIME": "1684934382"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9097/jmxrmi",
                                "jvmId": "vjtgbNxwFMxemB3YBCAXiR317KeEEJGx5x55Lag52CU=",
                                "labels": {}
                            }
                        }
                    ],
                    "id": -208503340,
                    "labels": {
                        "REALM": "47d67480-23dc-4fe3-a937-3a2d1519b198"
                    },
                    "name": "quarkus-test-agent",
                    "nodeType": "Realm"
                },
                {
                    "children": [
                        {
                            "id": 1056509530,
                            "labels": {},
                            "name": "http://localhost:9988/",
                            "nodeType": "JVM",
                            "target": {
                                "alias": "quarkus-test-agent",
                                "annotations": {
                                    "cryostat": {
                                        "HOST": "cryostat",
                                        "JAVA_MAIN": "/deployments/quarkus-run.jar",
                                        "PID": "1",
                                        "PORT": "9988",
                                        "REALM": "quarkus-test-agent",
                                        "START_TIME": "1684934382"
                                    },
                                    "platform": {}
                                },
                                "connectUrl": "http://localhost:9988/",
                                "jvmId": "p8CaVOm7-st7PgUuFMDJE2dUtsZX09kOhhmsO990EQ4=",
                                "labels": {}
                            }
                        }
                    ],
                    "id": 193647808,
                    "labels": {
                        "REALM": "b56fa50b-a250-443b-9ea8-ac4f657f679d"
                    },
                    "name": "quarkus-test-agent",
                    "nodeType": "Realm"
                },
                {
                    "children": [],
                    "id": -1330562508,
                    "labels": {
                        "REALM": "5aae4f79-7dd8-48d7-85c8-1b1c2893a3e3"
                    },
                    "name": "Custom Targets",
                    "nodeType": "Realm"
                }
            ],
            "id": 1823591626,
            "labels": {},
            "name": "Universe",
            "nodeType": "Universe"
        }
    },
    "meta": {
        "status": "OK",
        "type": "application/json"
    }
}

[andrewazores]

Aha, okay, so it sounds like there is a bug in deactivation, which is what I thought I had seen when I first filed this bug. Let's focus on that for now. I will file a separate bug for the way we trigger rules and handle targets reachable via multiple URLs. Problems in that area keep coming up over time in different ways.

[andrewazores]

Okay, so it's sounding more and more like there may be a (heisen)bug in rule deactivation, but #1497 is the real bug. I will leave this issue open for now until we can confirm more reports of a rule deactivation issue and get a better picture of what may be happening there.

[mwangggg]

{
   "name":"test",
   "description":"",
   "matchExpression":"target.connectUrl == \"service:jmx:rmi:///jndi/rmi://localhost:9093/jmxrmi\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":0,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

I created this rule with the connection url as the match expression and the rule didn't trigger for the desired target.

[mwangggg]

{
   "name":"test_jvmId",
   "description":"",
   "matchExpression":"target.jvmId == \"nVSsTM3QcYa-M0kTfps5Vb2MlnesJqcCyR5VXYFuUkk=\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

I create this rule with the JVM ID as the the match expression and the rule works as expected, but throws the following error:
Report generation failure: io.cryostat.net.AgentJFRService$UnimplementedException caused by AgentJFRService.UnimplementedException:
when opening the active recording for vertx-fib-demo-1 Realm

[andrewazores]

Aha, that one makes some sense. Targets discovered by the Agent registering itself may have HTTP URLs, which currently only expose a readonly API - recordings cannot be dynamically created over HTTP, hence the UnimplementedException.

I think it's OK for the activation to fail in that case, but we need to check if that exception being thrown actually stops the activation from continuing. It should catch and log that exception but continue on.

[andrewazores]

If that target had multiple URLs it was reachable by then it could have still stopped the activation, in the case that the HTTP URL was the last one that it tried. You're probably right, but it's worth taking a closer look.

[mwangggg]

{
   "name":"test_jvmId_and_alias",
   "description":"should only target http://localhost:8910/ and service:jmx:rmi:///jndi/rmi://localhost:9093/jmxrmi",
   "matchExpression":"/vertx-fib-demo/.test(target.alias) && target.jvmId == \"VBX_ZhDP8qs8wEf1el7lOv-_PVcoAh_jbOMvt7Snf9U=\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

I created this rule which checks both the JVM ID and the alias. The rule does not appear in any of the targets.

[mwangggg]

{
   "name":"test_jvmId_or_alias",
   "description":"should have 7 targets: all vertx-fib-demos and service:jmx:rmi:///jndi/rmi://cryostat:9093/jmxrmi",
   "matchExpression":"/vertx-fib-demo/.test(target.alias) || target.jvmId == \"VBX_ZhDP8qs8wEf1el7lOv-_PVcoAh_jbOMvt7Snf9U=\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

I created this rule which applies if the JVM ID or the alias is satisfied. The rule should appear in 7 targets, but only applies in the targets that satisfy the JVM ID.

[andrewazores]

For that last case, in the standard smoketest.sh setup with no additional steps, the vertx-fib-demo- 2 and 3 will normally fail to connect over the JMX connection because they require JMX credentials, and IIRC 3 also requires Cryostat to trust the application's SSL cert. So the JMX URLs that point to those targets, both in the JDP Realm and the Podman Realm, should fail to activate the rule, I think. Let's take note of that particular case and I'll help walk you through it later so we can see if adding credentials and/or trusting the SSL cert changes the behaviour.

[andrewazores]

There should be no actual differences in the container image at this point, but we should test that this bug affects not just the latest main development image but also the released 2.3.0 image.

@mwangggg if you just spin up the test script like CRYOSTAT_IMAGE=quay.io/cryostat/cryostat:2.3.0 sh smoketest.sh then it'll be the right release version.

[mwangggg]

I tried running the test script for the 2.3.0 image, but it's been stuck on
Copying blob sha256:9234344aa613a50c8eee0dd15c71b539c68c986c040bd48b22d52b73c542eb8b for the past 5-10 minutes

[andrewazores]

Weird. If it's still stuck, maybe just try Ctrl-cing the process in your terminal and trying again. It should make progress and download the image, and it isn't that big that it should take more than 5 minutes just to download a single layer.

[andrewazores]

It should not make any difference but you can also try to do podman pull quay.io/cryostat/cryostat:2.3.0 in a terminal directly, just so you can see the output from that command in isolation rather than as part of the scripted setup.

[mwangggg]

it works! Just took awhile

[andrewazores]

diff --git a/smoketest.sh b/smoketest.sh
index f0ec590b..9515b376 100755
--- a/smoketest.sh
+++ b/smoketest.sh
@@ -112,15 +112,6 @@ runDemoApps() {
         --name vertx-fib-demo-1 \
         --env HTTP_PORT=8081 \
         --env JMX_PORT=9093 \
-        --env CRYOSTAT_AGENT_APP_NAME="vertx-fib-demo-1" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
-        --env CRYOSTAT_AGENT_WEBSERVER_HOST="localhost" \
-        --env CRYOSTAT_AGENT_WEBSERVER_PORT="8910" \
-        --env CRYOSTAT_AGENT_CALLBACK="http://localhost:8910/" \
-        --env CRYOSTAT_AGENT_BASEURI="${protocol}://localhost:${webPort}/" \
-        --env CRYOSTAT_AGENT_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
         --pod cryostat-pod \
         --label io.cryostat.connectUrl="service:jmx:rmi:///jndi/rmi://localhost:9093/jmxrmi" \
         --rm -d quay.io/andrewazores/vertx-fib-demo:0.12.2
@@ -130,15 +121,6 @@ runDemoApps() {
         --env HTTP_PORT=8082 \
         --env JMX_PORT=9094 \
         --env USE_AUTH=true \
-        --env CRYOSTAT_AGENT_APP_NAME="vertx-fib-demo-2" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
-        --env CRYOSTAT_AGENT_WEBSERVER_HOST="localhost" \
-        --env CRYOSTAT_AGENT_WEBSERVER_PORT="8911" \
-        --env CRYOSTAT_AGENT_CALLBACK="http://localhost:8911/" \
-        --env CRYOSTAT_AGENT_BASEURI="${protocol}://localhost:${webPort}/" \
-        --env CRYOSTAT_AGENT_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
         --pod cryostat-pod \
         --label io.cryostat.connectUrl="service:jmx:rmi:///jndi/rmi://localhost:9094/jmxrmi" \
         --rm -d quay.io/andrewazores/vertx-fib-demo:0.12.2
@@ -149,15 +131,6 @@ runDemoApps() {
         --env JMX_PORT=9095 \
         --env USE_SSL=true \
         --env USE_AUTH=true \
-        --env CRYOSTAT_AGENT_APP_NAME="vertx-fib-demo-3" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
-        --env CRYOSTAT_AGENT_WEBSERVER_HOST="localhost" \
-        --env CRYOSTAT_AGENT_WEBSERVER_PORT="8912" \
-        --env CRYOSTAT_AGENT_CALLBACK="http://localhost:8912/" \
-        --env CRYOSTAT_AGENT_BASEURI="${protocol}://localhost:${webPort}/" \
-        --env CRYOSTAT_AGENT_TRUST_ALL="true" \
-        --env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
         --pod cryostat-pod \
         --label io.cryostat.connectUrl="service:jmx:rmi:///jndi/rmi://localhost:9095/jmxrmi" \
         --rm -d quay.io/andrewazores/vertx-fib-demo:0.12.2

This patch removes the -agent configuration from all the vertx-fib-demo sample applications. The Agent is still loaded but it is missing required environment variables, so it stops immediately, does not register with Cryostat, etc. This removes the http:// discovery target nodes from the deployment. These are not properly handled since they do not allow dynamic recording creation and they may trigger the Rule processing instead of the JMX URL doing so. The rule processing tries to only trigger once per target by using the JVM ID to determine uniqueness of targets, so if the HTTP URL comes up first in that iteration, then the whole trigger for that target will silently fail, even though there are usable JMX URLs available.

Next thing to try with @mwangggg for tomorrow: apply this patch, and then also set the discovery environment variable to turn off the Podman API discovery so we go back to only using JDP discovery. After this we should be back in a scenario where all targets are reachable only by one unique discovery URL, which simplifies the testing scenario and we can narrow down what part of the bug is actually broken rule behaviour vs what is failure to resolve targets properly, which is somewhat separate.

[mwangggg]

{
   "name":"test_all_aliases",
   "description":"",
   "matchExpression":"/es.andrew/.test(target.alias)",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

I created a match expression using the alias for the targets and the rule works as expected. The original bug seemed to be fixed the first time I ran this, but after running it a second time, the bug still persists- after deleting the rule, recordings are still being archived.

[mwangggg]

{
   "name":"test_jvmId",
   "description":"",
   "matchExpression":"target.jvmId == \"aVjrvkhI2Jv1TCg-TgrxTW74K-c56N5N0nwJA3ol6Hw=\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

Without podman realm -> correct targets, no more archived rules after deleting

[mwangggg]

{
   "name":"test_aliases",
   "description":"",
   "matchExpression":"target.alias == \"es.andrewazor.demo.Main\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":3,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":true
}

without podman realm -> correct targets, no more archived recordings after disabling or deleting

[mwangggg]

{
   "name":"test_aliases_2",
   "description":"",
   "matchExpression":"target.alias == \"es.andrewazor.demo.Main\"",
   "eventSpecifier":"template=Continuous,type=TARGET",
   "archivalPeriodSeconds":30,
   "initialDelaySeconds":30,
   "preservedArchives":2,
   "maxAgeSeconds":35,
   "maxSizeBytes":0,
   "enabled":false
}

with podman realm -> vertx-fib-demos in podman realm also get recordings, no more archived recordings after disabling or deleting

[andrewazores]

I observed this in a crc test just now, and it corresponded to a rule failing to activate. May be a red herring.

INFO: Observing new target: io.cryostat.platform.ServiceRef@773906e8[alias=quarkus-test-7bc456f446-5rt9z,annotations=io.cryostat.platform.ServiceRef$Annotations@11084b0c[cryostat={HOST=10.217.0.94, PORT=9097, NAMESPACE=myapps1, POD_NAME=quarkus-test-7bc456f446-5rt9z, REALM=KubernetesApi},platform={seccomp.security.alpha.kubernetes.io/pod=runtime/default, openshift.io/scc=restricted-v2, k8s.v1.cni.cncf.io/network-status=[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.217.0.94"
],
"default": true,
"dns": {}
}]}],jvmId=<null>,labels={app=quarkus-test, pod-template-hash=7bc456f446},serviceUri=service:jmx:rmi:///jndi/rmi://10-217-0-94.myapps1.pod:9097/jmxrmi]
Exception in thread "ForkJoinPool.commonPool-worker-1" java.lang.UnsupportedOperationException: MODIFIED
at io.cryostat.recordings.RecordingMetadataManager.lambda$accept$9(RecordingMetadataManager.java:439)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

[andrewazores]

SEVERE: Exception thrown
java.util.concurrent.ExecutionException: io.cryostat.recordings.RecordingNotFoundException: Recording auto_myrule2 was not found in the target [active recordings].
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.cryostat.rules.PeriodicArchiver.performArchival(PeriodicArchiver.java:141)
at io.cryostat.rules.PeriodicArchiver.run(PeriodicArchiver.java:123)
at io.cryostat.rules.RuleProcessor.lambda$activate$5(RuleProcessor.java:323)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:927)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:903)
at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55)
at io.vertx.core.impl.ContextBase.emit(ContextBase.java:239)
at io.vertx.core.impl.ContextInternal.emit(ContextInternal.java:194)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:921)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:159)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

[andrewazores]

This will be "fixed" after #1646 - there are multiple factors at play, some of which will be fixed directly in #1646 and the remainder are actually #1497 itself.