Comments (9)
Please provide full readable stacktrace as a text. Otherwise will not be assessed.
from rdkafka-ruby.
Also please prepare a workable POC if your expectation is to get help. Help me, so I can help you.
from rdkafka-ruby.
rdkafka-ruby-stacktrace.txt
https://github.com/jiteshonce/rdkafka-syncronize
Here is the attached backtrace and repo link
We are getting issue with our puma 5.6.7 but not getting with puma 5.0
from rdkafka-ruby.
@jiteshonce can you explain how this reproduction should be used? exact steps to trigger the behaviour etc?
from rdkafka-ruby.
@jiteshonce your stacktrace points be to app/entities/destination_entity.rb
- I cannot find such object in the example repro app. Once again I will kindly ask you to provide me with all information and clear path for me to be able to help you.
from rdkafka-ruby.
I am getting this exception on running rails console. I added the related code in https://github.com/jiteshonce/rdkafka-syncronize/blob/main/config/initializers/kafka_queues_init.rb
from rdkafka-ruby.
Ok I can reproduce. It is not rdkafka-ruby, it is you running topic creation on initialization and catching errors without explicit admin close.
This is not something you want to do anyway in an initializer the way you do. You should create an explicit rake task for topics mangement or you can use karafka declarative topics available in OSS: https://karafka.io/docs/Declarative-Topics/
that provide bundle exec karafka topics migrate
and a DSL for topics definition:
class KarafkaApp < Karafka::App
routes.draw do
topic :a do
config(
partitions: 6,
replication_factor: 3,
'retention.ms': 86_400_000 # 1 day in ms,
'cleanup.policy': 'delete'
)
consumer ConsumerA
end
topic :b do
config(
partitions: 2,
replication_factor: 3
# The rest will be according to the cluster defaults
)
consumer ConsumerB
end
end
end
if you really want to do what you are doing and I highly discourage you from doing so, you should ensure admin is closed:
# Create Kafka Topic if it does not exist
Rdkafka::Config.logger = Rails.logger
begin
admin = Rdkafka::Config.new({:"bootstrap.servers" => "localhost:9092"}).admin
create_topic_handle = admin.create_topic('our.new.topic2', 1, 1)
create_topic_handle.wait(max_wait_timeout: 1.0)
rescue Rdkafka::RdkafkaError => e
Rails.logger.warn 'Kafka topic already exists'
rescue StandardError => e
Rails.logger.error "Kafka error- (kafka_queues_init.rb): #{e.message}"
ensure
admin.close
end
from rdkafka-ruby.
also please note that by doing what you are doing likely you will hit this bug as well: #266
from rdkafka-ruby.
Thanks @mensfeld for help, on closing the admin our problem got resolved.
from rdkafka-ruby.
Related Issues (20)
- Retire support for Ruby 2.7 after 2024-06-30
- Connection timeouts when creating topic but not when listing topics HOT 1
- Probide ability to label handlers
- Ship librdkafka with the package to allow for offline usage and prevent potential checksum issues
- Segfault on "rd_kafka_new" HOT 28
- Kafka config sasl.password with special characters treated improperly HOT 2
- Consumer can hang during cooperative-sticky HOT 1
- Use Mutex lock released on delivery instead of a sleep wait with timeout and a loop HOT 9
- rdkafka-ruby and librdkafka matrix HOT 4
- Invalid ACL resources termination HOT 1
- Post-fork initialization crash due to libcurl dynamic load from a fork HOT 1
- Use the existing background thread to refresh metadata state for partition key usage HOT 1
- Partition cache caches invalid nil result for `PARTITIONS_COUNT_TTL`
- Optimize `partition_count` iterating on big set of topics
- Use multi-producer topics partitions count cache
- Consider defaults alter on producer due to potential DDOS HOT 1
- how to configure oauthbearer token refresh HOT 5
- Background logger stops working after forking causing memory leaks HOT 7
- Modernize producers thread usage via `rd_kafka_queue_cb_event_enable`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rdkafka-ruby.