Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER

[k-e-r-n-e-l-p-a-n-i-c]

Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER

[k-e-r-n-e-l-p-a-n-i-c]

Hi,

Can you please provide some help on this issue?

[baum]

Hello @arunprasadmudaliar 🖖

I have tried to reproduce this issue with both v5.10.1 and the latest master branch operator/core installation, I do not see the issue:

➜  noobaa-operator git:(master) ✗ ./build/_output/bin/noobaa-operator-local -n noobaa api events_api read_activity_log '{}'
INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✅ Exists: Secret "noobaa-admin"
INFO[0000] ✈️  RPC: events.read_activity_log() Request: map[]
WARN[0000] RPC: GetConnection creating connection to wss://localhost:65366/rpc/ 0xc0003b6b40
INFO[0000] RPC: Connecting websocket (0xc0003b6b40) &{RPC:0xc00045dd60 Address:wss://localhost:65366/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0000] RPC: Connected websocket (0xc0003b6b40) &{RPC:0xc00045dd60 Address:wss://localhost:65366/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0000] ✅ RPC: events.read_activity_log() Response OK: took 44.9ms
logs:
- desc:
  - Drive /noobaa_storage of noobaa-default-backing-store-noobaa-pod-e25426c8 in pool
    noobaa-default-backing-store is online
  event: node.connected
  id: 63397dee5c53f6002fc965a0
  level: info
  node:
    linkable: true
    name: noobaa-default-backing-store-noobaa-pod-e25426c8#1
    pool: noobaa-default-backing-store
  time: 1664712174965
- actor:
    email: [email protected]
    linkable: true
  desc:
  - A managed pool noobaa-default-backing-store with 1 nodes was created by [email protected]
  event: resource.create
  id: 63397d225c53f6002fc9659c
  level: info
  pool:
    linkable: true
    name: noobaa-default-backing-store
    resource_type: HOSTS
  time: 1664711970827
- account:
    email: [email protected]
    linkable: true
  desc:
  - '[email protected] was created '
  event: account.create
  id: 63397cc45c53f6002fc96593
  level: info
  time: 1664711876291
- actor:
    email: [email protected]
    linkable: true
  desc:
  - noobaa was created by [email protected]
  event: conf.create_system
  id: 63397cb65c53f6002fc9658f
  level: info
  time: 1664711861995

Could you please provide more info?
Also, it might be a good idea to verify that the CLI and the core versions match.

v5.10.1

➜  noobaa-operator git:(6f571fd) ✗  ./build/_output/bin/noobaa-operator-local version
INFO[0000] CLI version: 5.10.1
INFO[0000] noobaa-image: noobaa/noobaa-core:5.10.1
INFO[0000] operator-image: noobaa/noobaa-operator:5.10.1
➜  noobaa-operator git:(6f571fd) ✗ k get  noobaa noobaa -n noobaa -o yaml | grep Core
  readme: "\n\n\tWelcome to NooBaa!\n\t-----------------\n\tNooBaa Core Version:     5.10.1-e8c9b09\n\tNooBaa

Latest master

➜  noobaa-operator git:(master) ✗ ./build/_output/bin/noobaa-operator-local version
INFO[0000] CLI version: 5.12.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20220913
INFO[0000] operator-image: noobaa/noobaa-operator:5.12.0
➜  noobaa-operator git:(master) ✗ k get  noobaa noobaa -n noobaa -o yaml | grep Core
  readme: "\n\n\tWelcome to NooBaa!\n\t-----------------\n\tNooBaa Core Version:     5.12.0-cfbf726\n\tNooBaa

Please share your thoughts.
Best regards.

[k-e-r-n-e-l-p-a-n-i-c]

Hello @baum,

Thank you for responding. Based on your comment, I verified the version of the client and the operator and indeed they were different in my case. ( operator was 5.8.0 and the client was 5.10.0).

I have now downgraded my client version to 5.8.0 to match the operator:

CLI
INFO[0000] CLI version: 5.8.0
INFO[0000] noobaa-image: noobaa/noobaa-core:5.8.0
INFO[0000] operator-image: noobaa/noobaa-operator:5.8.0

Operator version ( as shown in web-console)

But I still get the same error:

INFO[0002] ✅ Exists: NooBaa "noobaa"
INFO[0002] ✅ Exists: Service "noobaa-mgmt"
INFO[0002] ✅ Exists: Secret "noobaa-operator"
INFO[0003] ✅ Exists: Secret "noobaa-admin"
INFO[0003] ✈️ RPC: events.read_activity_log() Request: map[]
WARN[0003] RPC: GetConnection creating connection to wss://localhost:39703/rpc/ 0xc000f0f4a0
INFO[0003] RPC: Connecting websocket (0xc000f0f4a0) &{RPC:0xc00003a5a0 Address:wss://localhost:39703/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
INFO[0004] RPC: Connected websocket (0xc000f0f4a0) &{RPC:0xc00003a5a0 Address:wss://localhost:39703/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
ERRO[0024] ⚠️ RPC: events.read_activity_log() Response Error: Code=INVALID_SCHEMA_REPLY Message=INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log
FATA[0024] ❌ INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log

I have to connect to the server through a VPN connection and initially I thought might be its due to VPN. But later I realized that the websocket connection shows connected and as well as other api queries related to bucket works fine. It's only the events api that throws an error.

I tried the export_activity_log but again I get an error. Using the -o flag does not help here. Looks like it is failing due to type casting.

noobaa api events_api export_activity_log '{}' -o yaml
INFO[0005] ✅ Exists: NooBaa "noobaa"
INFO[0006] ✅ Exists: Service "noobaa-mgmt"
INFO[0006] ✅ Exists: Secret "noobaa-operator"
INFO[0006] ✅ Exists: Secret "noobaa-admin"
INFO[0008] ✈️ RPC: events.export_activity_log() Request: map[]
WARN[0008] RPC: GetConnection creating connection to wss://localhost:38625/rpc/ 0xc000521a90
INFO[0008] RPC: Connecting websocket (0xc000521a90) &{RPC:0xc000460690 Address:wss://localhost:38625/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
INFO[0011] RPC: Connected websocket (0xc000521a90) &{RPC:0xc000460690 Address:wss://localhost:38625/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
ERRO[0069] ⚠️ RPC: events.export_activity_log() Call failed: json: cannot unmarshal string into Go struct field .reply of type map[string]interface {}
FATA[0069] ❌ json: cannot unmarshal string into Go struct field .reply of type map[string]interface {}

Please provide any suggestions that you want me to try. Upgrading the operator to latest version is bit difficult as it is maintained by another team.

[baum]

Hello @arunprasadmudaliar 🖖

• I do see an issue with export_activity_log, see the #7087 PR.
• Re read_activity_log, could you try with v5.10.1 CLI?

Best regards.

[k-e-r-n-e-l-p-a-n-i-c]

Hello @baum,

Thanks for sharing the above issue details for export_activity_log.

I tried to run read_activity_log with 5.10.1 cli but the error still remains the same.

ERRO[0009] ⚠️ RPC: events.read_activity_log() Response Error: Code=INVALID_SCHEMA_REPLY Message=INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log
FATA[0009] ❌ INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log

root@~/noobaa5.10# ./noobaa version
INFO[0000] CLI version: 5.10.1
INFO[0000] noobaa-image: noobaa/noobaa-core:5.10.1
INFO[0000] operator-image: noobaa/noobaa-operator:5.10.1

So, if I could summarize, I have a noobaa server (operator) of version 5.8.0 and I had tried CLI version's 5.8.0 ( to match server version) and 5.10.1. But I still receive the same error. As far as I understood, the cli and operator are same binaries and so do you think I should try updating the noobaa server (operator) to latest version?

Regards,