Opensearch is not available after provision certificate

[graylog2]

Opensearch is not available after provision certificate

[janheise]

@cuserdzy I'm glad to hear that it works for you! Maybe we should add some info regarding minimal RAM to the docs/readme. Maybe we should also make the docs more clear: skipping provisioning basically means: "I don't want to use the DataNode but want to roll my own." But I guess we have to state that clearly.

[cuserdzy]

After switch to t4g.medium, 2core, 4g, the opensearch process returns exit code 78 with error "[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]".
Just set map_map_count by "sysctl -w vm.max_map_count=262144", then restart by "docker-compose restart" and re-provisioning, everything works.

The account is "admin" and the password is the original value of sha2 in .env file

[dorokhin]

@janheise
I just tried to run it on my MacBook Pro with 32 gigabytes of RAM.
and everything started successfully.
The virtual machine I previously tried to run had 4 gigabytes of memory, ~2Gb of which were used by another service.
thanks for the help

[janheise]

@cuserdzy Hi, which step exactly are you skipping? And why?

[dorokhin]

The same error:

2023-11-30 11:34:19,735 INFO : org.graylog2.bootstrap.preflight.PreflightJerseyService - Shutting down HTTP listener at <0.0.0.0:9000>
2023-11-30 11:34:19,736 INFO : org.graylog2.periodical.Periodicals - Shutting down periodical [org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical].
2023-11-30 11:34:19,737 INFO : org.graylog2.periodical.Periodicals - Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2023-11-30 11:34:19,737 INFO : org.graylog2.periodical.Periodicals - Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2023-11-30 11:34:19,759 INFO : org.glassfish.grizzly.http.server.NetworkListener - Stopped listener bound to [0.0.0.0:9000]
2023-11-30 11:34:19,999 INFO : org.graylog2.plugin.system.FilePersistedNodeIdProvider - Node ID: 8d98b995-6eac-4aec-ae30-f0598bbd4cac
2023-11-30 11:34:20,007 INFO : org.graylog2.storage.versionprobe.VersionProbe - OpenSearch/Elasticsearch is not available. Retry #1
2023-11-30 11:34:25,009 INFO : org.graylog2.storage.versionprobe.VersionProbe - OpenSearch/Elasticsearch is not available. Retry #2
2023-11-30 11:34:30,015 INFO : org.graylog2.storage.versionprobe.VersionProbe - OpenSearch/Elasticsearch is not available. Retry #3

[janheise]

At both of you: the error indicates, that no OpenSearch instance is running.

In the first post, this is also what the snippet from the datanode logs indicate. My question is: which step in the setup did you skip?

@dorokhin : what exactly did you do, up to the error? Did you skip any steps?

In both cases, I recommend doing a docker compose down -v and then start over from the beginning. Don't skip any setup steps, make sure to press the resume button at the end.

If you still get errors, please in any case, add always logs from graylog & the datanode. also, which docker-compose.yml are you using? Open? Enterprise? Cluster?

[dorokhin]

I used docker compose open-core file from this commit e44b25c (Using Graylog Data Node instead of OpenSearch)
which has no opensearch instances.

I'am also trying to remove all related to this docker compose volumes (with docker compose down -v)
and start from beginning, but no luck

[cuserdzy]

Hi @janheise, I used "open-core" file and skipped below step by clicking "Skip provisioning", then the graylog keep hang.

I also tried to do "provisioning" but failed as well, shows below error:

graylog node:

2023-12-01 02:43:30,282 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
2023-12-01 02:43:30,286 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
2023-12-01 02:43:42,085 INFO : org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical - Starting connectivity check with node d6979ccd-2e74-4f9f-949c-61fa38b23723, silencing error messages for 60 seconds.
2023-12-01 02:44:42,242 WARN : org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical - Error trying to connect to data node  d6979ccd-2e74-4f9f-949c-61fa38b23723: Failed to connect to 111afc9d1f5f/172.23.0.2:9200, retrying (attempt #21)

data node:

2023-12-01T02:43:30.534Z INFO  [CustomCAX509TrustManager] CA changed, refreshing trust manager
2023-12-01T02:43:30.534Z INFO  [CustomCAX509TrustManager] CA changed, refreshing trust manager
2023-12-01T02:43:39.208Z INFO  [DataNodeConfigurationPeriodical] created CSR for this node
2023-12-01T02:43:40.727Z INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 111afc9d1f5f, 0:0:0:0:0:0:0:1, 172.23.0.2
2023-12-01T02:43:40.744Z INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 111afc9d1f5f, 0:0:0:0:0:0:0:1, 172.23.0.2
2023-12-01T02:43:40.776Z INFO  [TruststoreCreator] Adding certificate transport-chain-CA-root to the truststore
2023-12-01T02:43:40.786Z INFO  [TruststoreCreator] Adding certificate http-chain-CA-root to the truststore
2023-12-01T02:43:40.972Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-aarch64/bin/opensearch
2023-12-01T02:43:41.544Z INFO  [JerseyService] Server configuration changed, restarting Data node REST API to apply security changes
2023-12-01T02:43:41.545Z INFO  [JerseyService] Shutting down HTTP listener at <0.0.0.0:8999>
2023-12-01T02:43:41.574Z INFO  [NetworkListener] Stopped listener bound to [0.0.0.0:8999]
2023-12-01T02:43:41.840Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:8999]
2023-12-01T02:43:41.842Z INFO  [HttpServer] [HttpServer-1] Started.
2023-12-01T02:43:41.842Z INFO  [JerseyService] Started REST API at <0.0.0.0:8999>
2023-12-01T02:43:44.658Z WARN  [OpensearchProcessImpl] Opensearch process failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
        at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
        at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
        at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-12-01T02:43:44.682Z INFO  [ProcessWatchdog] Detected terminated process, restarting. Attempt #1
2023-12-01T02:43:45.484Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-aarch64/bin/opensearch
2023-12-01T02:43:46.677Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 276 unavailable. Cause: Connection refused

[cuserdzy]

The error code 137 seems caused by insufficient memory, my machine is t4g.small, 2core, 2g

[janheise]

@dorokhin please post some log files and info on your machine so that I can help you!