Comments (24)
Yup, seems that fixed it - thanks!
from rabbitmq-server.
Hi! Thank you for the report.
How do you specify the path to the inetrc file?
The peer discovery subsystem starts a temporary intermediate node to query some properties of remote nodes. I suspect that this temporary node doesn't use this inetrc file.
from rabbitmq-server.
No, this is a limitation in the way that temporary node is started probably. I will try to reproduce locally.
from rabbitmq-server.
Thank you very much! Yes, it definitely helps.
I fixed the command in my comment, thanks.
from rabbitmq-server.
@dormanze it's incredibly rude to nudge open source maintainers to hurry up to deliver a fix in a tool you very likely do not pay for, and never have. We have a PR ready but it has shown an improvement only in some environments.
If you are desperately in need of a fix, you are welcome to contribute it, pay for commercial support, or patiently wait for the maintainers to do it.
from rabbitmq-server.
I verified the addition of the double quotes and my problem was indeed solved. Thank you very much.@dumbbell
from rabbitmq-server.
I’ve merged the patch to the main
branch and will leave it there for a week or so. If no regressions are spotted, I will backport it to 3.13.x. Sorry for the time it took to merge, there were unrelated failures in CI that blocked it.
If you have a chance, please test the main
branch :-)
from rabbitmq-server.
The patch was integrated into the 3.13.x release branch and will be part of RabbitMQ 3.13.2.
from rabbitmq-server.
Here is the container log: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/container_logs/rabbitmq.txt
We just put erl_inetrc under /etc/rabbitmq/erl_inetrc with this content: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/erl_inetrc
Is there something more we should be doing?
from rabbitmq-server.
I suspect that this temporary node doesn't use this inetrc file
I'm sure you're right @dumbbell
from rabbitmq-server.
This may be worth delaying 3.13.1
over.
from rabbitmq-server.
I'm starting to look at this issue today, bu I can't find where this /etc/rabbitmq/erl_inetrc
file is configured. RabbitMQ itself doesn't look at a default location, so it must come from something outside of RabbitMQ.
Could you please share the output of the following two commands?
rabbitmqctl eval 'os:getenv().'
rabbitmqctl eval 'application:get_all_env(kernel).'
from rabbitmq-server.
we set this via the environment, see the jinja template here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 (line 10)
I don't currently have shell access to the CI env where this fails, but as you can see from the git history this part of the code didn't change since 2019, so here are the commands you wanted the output from, from a similar environment, but with a slightly older rmq release.
you can see the erl_inetrc
is loaded there as well, I just edited out some stuff like hostnames and IPs:
rabbitmqctl eval 'os:getenv().'
["KOLLA_DISTRO_PYTHON_VERSION=3.10",
"RABBITMQ_CLUSTER_COOKIE=REDACTED",
"RABBITMQ_LOG_DIR=/var/log/kolla/rabbitmq","ROOTDIR=/usr/lib/erlang",
"SHLVL=0","SYS_PREFIX=","RABBITMQ_NODENAME=rabbit@REDACTED",
"RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid",
"RABBITMQ_HOME=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23",
"RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq","RABBITMQ_DIST_PORT=25672",
"OLDPWD=/","PROGNAME=erl","PWD=/var/lib/rabbitmq",
"PATH=/usr/lib/erlang/erts-13.2.2.3/bin:/usr/lib/erlang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PLUGINS_DIR=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
"LANG=en_US.UTF-8","MNESIA_BASE=/var/lib/rabbitmq/mnesia",
"KOLLA_SERVICE_NAME=rabbitmq","HOME=/var/lib/rabbitmq",
"KOLLA_BASE_DISTRO=ubuntu","KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
"HOSTNAME=REDACTED","KOLLA_BASE_ARCH=x86_64","ERL_EPMD_ADDRESS=REDACTED",
"ERL_INETRC=/etc/rabbitmq/erl_inetrc","ERL_MAX_ETS_TABLES=50000",
"ERL_MAX_PORTS=65536",
"ERL_LIBS=/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
"ERL_EPMD_PORT=4369","DEBIAN_FRONTEND=noninteractive",
"ENABLED_PLUGINS_FILE=/etc/rabbitmq/enabled_plugins",
"ERL_CRASH_DUMP=/var/log/kolla/rabbitmq/erl_crash.dump","EMU=beam",
"BINDIR=/usr/lib/erlang/erts-13.2.2.3/bin",
"CONFIG_FILE=/etc/rabbitmq/rabbitmq",
"ADVANCED_CONFIG_FILE=/etc/rabbitmq/advanced.config"]
and the other command (your command missed a ' at the end):
rabbitmqctl eval 'application:get_all_env(kernel).'
[{shell_docs_ansi,auto},
{net_ticktime,60},
{shutdown_func,{rabbit_prelaunch,shutdown_func}},
{inet_default_connect_options,[{nodelay,true}]},
{connect_all,true},
{logger_level,notice},
{inet_dist_listen_min,25672},
{inetrc,'/etc/rabbitmq/erl_inetrc'},
{logger_sasl_compatible,false},
{logger,
[{handler,default,logger_std_h,
#{config => #{type => standard_io},
formatter =>
{logger_formatter,
#{legacy_header => true,single_line => false}}}}]},
{inet_dist_listen_max,25672},
{inet_dist_use_interface,{REDACTED}},
{net_tickintensity,4},
{prevent_overlapping_partitions,false}]
HTH in the debugging process.
from rabbitmq-server.
As a side question, why do you set both $ERL_INETRC
and -kernel inetrc …
?
This won’t change the debuggint or fix anything, I’m just curious.
from rabbitmq-server.
@mnasiadka, @artificial-intelligence: I prepared a patch in pull request #10759. I’m not sure yet how to create a testcase for this. Could you please give it a try in your environment?
from rabbitmq-server.
@dumbbell I'm afraid we have no experience in building a package with that PR content - is there an option you would make an rpm or deb package available for testing?
Sorry for the delayed response.
from rabbitmq-server.
@mnasiadka: Sure. I took a shortcut and replaced the rabbit_peer_discovery.beam
module inside the RabbitMQ 3.13.0 Debian package :-) This is the only module modified in this patch, so I expect it to work. Here it is:
https://people.freebsd.org/~dumbbell/rabbitmq/rabbitmq-server_3.13.0-1_all.deb
from rabbitmq-server.
@mnasiadka we plan on shipping 3.13.1 next Tuesday, it would be great to get some feedback from you by then. Note that one-off .deb
package that @dumbbell has produced for you.
Thank you!
from rabbitmq-server.
Related issue: #10840
from rabbitmq-server.
I used the new ported code and it didn't solve my problem. The error message is as follows:
2024-03-26 13:47:31.437298+08:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> BOOT FAILED
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> ===========
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> Exception during startup:
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> error:{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}}
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_peer_discovery:maybe_add_inetrc_arguments/1, line 442
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_peer_discovery:query_node_props/1, line 392
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_peer_discovery:sync_desired_cluster/3, line 188
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_db:init/0, line 65
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_boot_steps:-run_step/2-lc$^0/1-0-/2, line 51
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_boot_steps:run_step/2, line 58
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_boot_steps:-run_boot_steps/1-lc$^0/1-0-/1, line 22
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0> rabbit_boot_steps:run_boot_steps/1, line 23
2024-03-26 13:47:31.437559+08:00 [error] <0.248.0>
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> crasher:
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> initial call: application_master:init/4
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> pid: <0.247.0>
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> registered_name: []
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> exception exit: {{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}},
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> {rabbit,start,[normal,[]]}}
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> in function application_master:init/4 (application_master.erl, line 142)
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> ancestors: [<0.246.0>]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> message_queue_len: 1
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> messages: [{'EXIT',<0.248.0>,normal}]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> links: [<0.246.0>,<0.44.0>]
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> dictionary: []
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> trap_exit: true
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> status: running
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> heap_size: 2586
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> stack_size: 28
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> reductions: 176
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0> neighbours:
2024-03-26 13:47:32.438634+08:00 [error] <0.247.0>
2024-03-26 13:47:32.446210+08:00 [notice] <0.44.0> Application rabbit exited with reason: {{case_clause,{ok,'/usr/local/rabbitmq_server-3.13.0/etc/rabbitmq/erl_inetrc'}},{rabbit,start,[normal,[]]}}
my erl_inetrc:
{inet6, true}.
{file, hosts, "/etc/hosts"}.
{file, resolv, "/etc/resolv.conf"}.
For other configurations, see: #10840
from rabbitmq-server.
@dormanze: Thank you for testing!
The patch was not ready anyway and should not have been merged. I reverted it and will work on an improved patch.
from rabbitmq-server.
@dumbbell Thank you for your reply. Please let me know if you need me to cooperate with uploading any configuration.
from rabbitmq-server.
Hi @dumbbell , any progress on this issue? We desperately need to adapt to 3.13. If there is a fix, please let me know in time to cooperate with the verification. Thank you very much.
from rabbitmq-server.
@dormanze: The bug fix is probably ok, but I'm still working on the testsuite to have more confidence. It is challenging to test the part of peer discovery that triggered the regression @mnasiadka and you hit. The current test I have for IPv6 looks good, but having the equivalent for TLS inter-node communication is slightly harder to get right.
Your specific error is interesting. You used single quotes to quote the inetrc
filename. Erlang uses this quoting for a type that is not a string. We could tell you to use double quotes and be done with it. But I think RabbitMQ should handle the use of single quotes from users likely used to Bourne shell quoting and do what’s needed internally to use the parameter as a filename. I’m considering this in the testsuite.
from rabbitmq-server.
Related Issues (20)
- HTTP API: GET /api/queues/{vhost}/{name} can return duplicate keys for quorum queues HOT 3
- 3.11: two Raft replicas are in the timeout state, one is a candidate HOT 5
- OpenID compliance check is not based on the final specification
- Possible race condition in classic queue deletion/declaration handling for multi-node clusters
- Quorum queue replica Erlang process names can conflict between queues in different virtual hosts HOT 2
- OpenID Connect RP-Initiated Logout should be optional HOT 1
- Optimise how rdq files are scanned at CQ shared message store recovery startup HOT 21
- Management UI creates classic queue instead of virtual host default queue type in some circumstances HOT 2
- When I create a stream, an error message "could not connect osiris to replica" appears in the log. HOT 1
- Make it possible to configure OpenId Connect endpoints rather than discover them dynamically via OpenId Connect Discovery endpoint
- 3.13.0 - 3.13.2: Dead-letter cycle detection can wrongly drop messages
- 3.13.0 - 3.13.2: Wrong warning messages that dead letter messages get dropped HOT 1
- Streams: consider allowing non-numerical values for publishingId HOT 3
- Log tls handshake timeouts HOT 1
- Include x-death header in Stream messages HOT 1
- 4.x: reduce default maximum message size further (e.g. to 64 or 50 MiB) HOT 12
- Publish `amqp_client` 3.13.x GA versions to hex
- Emit event if configured queue length is reached
- Don't close connection if channel without finished publish is closed HOT 3
- [Prometheus] Reduce number of series for per-exchange/per-queue metrics HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-server.