jcustenborder/puppet-confluent

Puppet Module for installing and configuring the Confluent Platform

Introduction

This puppet module is used to install and configure the Confluent Platform. The documentation is available here.

Note on versioning

Puppet only allows version numbers in <major>.<minor>.<version> format. Due to this limitation the versioning schema matches the major and minor versions of the Confluent Platform. Meaning 3.2.1234 of the puppet module is for 3.2.x of the Confluent platform.

Usage

Zookeeper

Class installation

class{'confluent::zookeeper':
  zookeeper_id => '1',
  heap_size => '4000M'
}

Hiera Installation

include ::confluent::zookeeper

confluent::zookeeper::zookeeper_id: '1'
confluent::zookeeper::config:
  server.1:
    value: 'zookeeper-01.example.com:2888:3888'
  server.2:
    value: 'zookeeper-02.example.com:2888:3888'
  server.3:
    value: 'zookeeper-03.example.com:2888:3888'
confluent::zookeeper::heap_size: '4000M'

Kafka Broker

Class installation

class{'confluent::kafka::broker':
  broker_id => '1',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  heap_size => '4000M'
}

Heira installation

include ::confluent::kafka::broker

confluent::kafka::broker::broker_id: '1'
confluent::kafka::broker::heap_size: '4000M'
confluent::kafka::broker::zookeeper_connect:
  - 'zookeeper-01:2181',
  - 'zookeeper-02:2181',
  - 'zookeeper-03:2181'
confluent::kafka::broker::data_path: '/var/lib/kafka'

Kafka Connect

Distributed

Class Installation

class{'confluent::kafka::connect::distributed':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::distributed

 confluent::kafka::connect::distributed::heap_size: '4000M'
 confluent::kafka::connect::distributed::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::distributed::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Standalone

Class Installation

class{'confluent::kafka::connect::standalone':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::standalone

 confluent::kafka::connect::standalone::heap_size: '4000M'
 confluent::kafka::connect::standalone::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::standalone::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Schema Registry

Class installation

class {'confluent::schema::registry':
  heap_size => '1024M',
  kafkastore_connection_url => [
      'zookeeper-01:2181',
      'zookeeper-02:2181',
      'zookeeper-03:2181'
  ]
}

Hiera installation

include ::confluent::schema::registry

confluent::schema::registry::heap_size: '1024M'
confluent::schema::registry::kafkastore_connection_url:
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'

Confluent Control Center

Class installation

class {'confluent::control::center':
  heap_size => '6g',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  connect_cluster => [
    'kafka-connect-01.example.com:8083',
    'kafka-connect-02.example.com:8083',
    'kafka-connect-03.example.com:8083'
  ]
}

Hiera installation

include ::confluent::control::center

confluent::control::center::heap_size: '6g'
confluent::control::center::zookeeper_connect: 
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'
confluent::control::center::bootstrap_servers: 
  - 'broker-01:9092'
  - 'broker-02:9092'
  - 'broker-03:9092'
confluent::control::center::connect_cluster: 
  - 'kafka-connect-01.example.com:8083'
  - 'kafka-connect-02.example.com:8083'
  - 'kafka-connect-03.example.com:8083'

Run tests

rake spec

Rebuild github pages

rake strings:gh_pages:update

puppet-confluent's Issues

SystemD location for Debian based systems is most likely not correct.

confluent::systemd::unit is hardcoding the path for systemd unit files to /usr/lib/systemd/system/. This most likely is not correct on Debian and Ubuntu.

SystemD needs to be refreshed before starting service.

All the user to specify if services should be restarted over changes to logging.

Expose Metrics on Kafka Broker

Hi,

first of all thanks for this module. I've been playing with it for some days now and it works great! Saved me a lot of work!

I have the Control Center up and running and I wanted to see the broker health metrics. I see, your default config contains
'confluent.support.metrics.enable' => true,

puppet-confluent/manifests/kafka/broker.pp

Line 83 in c17152a

'confluent.support.metrics.enable' => true,

However, in order to actually see metrics, I needed to do two more things:

Firstly, extend the broker config by this:
"metric.reporters" => "io.confluent.metrics.reporter.ConfluentMetricsReporter",

and furthermore, add a

  package { 'confluent-rebalancer':
    ensure => latest,
    tag    => 'confluent',
  }

Am I missing something, or is this the right way to do it? Would you want a patch to extend that in your module?

Cheers,
Constantin

Ensure that all system units set TimeoutStopSec

Puppet 3.x

Last master branch broke puppet 3.x compatibility
Error 400 on SERVER: Syntax error at 'Integer'; expected ')'
Var type declaration only works with puppet 4.x

Enable autopurge in zookeeper by default.

Ensure that kafka-connect shuts down gracefully.

Connect-standalone should require configuration files to be specified.

Documentation Issues

I was helping someone new to puppet setup this module, and we found that several of the documentation examples have issues. I might be able to convince him to submit a PR, or if not, I will try to fit it in. Just in case someone else is trying to figure out why things are not working when they paste in the example code.

I will try to itemize the ones I see for now:

• There are several places where the code expects an "Integer" but the example code has '1', which is a "String" ... This results in the following error:

Error: Evaluation Error: Error while evaluating a Function Call, Class[Confluent::Kafka::Broker]: parameter 'broker_id' expects an Integer value, got String (file: /etc/puppetlabs/code/environments/vagrant/site/profile/manifests/confluent_kafka/broker.pp, line: 3, column: 3) on node kafka01.vagrant.local`)

It can be solved by removing the quotes around the 1, like:

confluent::zookeeper::zookeeper_id: 1

or

class{'confluent::zookeeper':
  zookeeper_id => 1,
}

NOTE: I feel like there should be some "automated" way to determine this ID by default, but that is another story.

• The hieradata lists should not have commas or quotes (some of the hiera examples appear to be copied from the Puppet code example, with minimal reformatting). It should look like:

confluent::kafka::broker::zookeeper_connect:
  - zookeeper01:2181
  - zookeeper02:2181
  - zookeeper03:2181

• The hashes are represented incorrectly in the examples and readme. It should just be key: value. Like:

confluent::zookeeper::config:
  server.1: zookeeper01:2888:3888
  server.2: zookeeper02:2888:3888
  server.3: zookeeper03:2888:3888

Heap should be a top level parameter on all classes.

Bump to Confluent 4.0

The package version installed should be configurable.

Support for Confluent Kafka 5.0

Confluent Kafka 5.0 was released at the end of July, it would be great for this module to support that version.

https://www.confluent.io/blog/introducing-confluent-platform-5-0

There should be consistent logging across all of the systems.

Ensure that control center shuts down gracefully.

java_property does not allow multiple applications with same property name

Ensure that schema-registry shuts down gracefully.

Support managing the repositories

Ensure that zookeeper shuts down gracefully.

Changes to configs should cause services to restart.

Use Stdlib::Unixpath instead of Stdlib::Absolutepath

Windows isn't supported.

Create manifest for kafka-rest

Specs should test to ensure that the repositories are included.

SystemD configuration should leverage the new SystemD units that Confluent is shipping

Change logger for reflections to error.

Zookeeper should refuse to start in standalone mode (triple split-brain)

If the server.1, server.2 and server.3 Hiera properties are not provided (or mapped incorrectly) Zookeeper will start in standalone mode. The way this Puppet module is structured, three standalone Zookeepers is an erroneous state.

This is due to the fact that the ZK manifest will proceed with or without the server list when creating the conf file for a ZK instance. According to the way ZK interpret its config, the absence of a server list will implicitly assume standalone mode. This will result in a triple split-brain that leads to data corruption among the Brokers.

Zookeeper v3.5.0 adds a config standaloneEnabled that will stop ZK from assuming standalone mode when no server list is specified. As of currently, this is not yet available in the Confluent Platform and this Puppet module. Until then, I suggest that the zookeeper.pp profile explicitly checks for the presence of the server list configuration and fail hard without it.

Connect is missing required default values.

Please tag your releases

It would be helpful if you would use git tags to "tag" each release. That way, when we are looking at the code in GitHub, we can ensure we are seeing the same code that is deployed to the puppet server by R10k/code-manager.

Thanks in advance,
Tommy

Kafka Connect standalone stores offsets in /tmp.

The config is currently configuring offset.storage.file.filename=/tmp/connect.offsets. A directory needs to be created and the offsets placed there. This will allow it survive a reboot.

Update kafka_package_name to confluent-kafka-2.12

In

puppet-confluent/manifests/params.pp

Line 174 in 9bc2da5

$kafka_package_name = 'confluent-kafka-2.11'

, it hardcodes a specific (outdated) version of the confluent-kafka package. We suggest that it should be updated to 2.12. I am not sure why the version number is embedded into the package name, as that makes maintenance a pain, but whatever :)

~tommy

Correct paths for logs.

Broker data_path should support an array for JBOD.

Timeout for stopping a Kafka Broker should be configurable. Default to 300

Add support for mirror maker

Move required configuration to top level parameters for classes.

There should be an easy way to enable the enterprise features

Move systemd units to require network-online.target instead of network.target

This will ensure that the network is up and online rather than just up.

Changing confluent::kafka::broker::log_path should cause the LOG_DIR environment variable to be updated.

SystemD files are being generated with a space between the separator.

Support puppet 5 types for parameters.

Update gemfile dependencies.

automatic tagging causes the ordering defined in init.pp to be applied to every module in some situations

Because of automatic tagging, https://puppet.com/docs/puppet/5.3/lang_tags.html#automatic-tagging the tag "confluent" gets applied to all modules included in a profile and applies the order specified in init.pp to all resources. This breaks runs when using this module and for e.g forge sssd module in a role/profile setup. I've tested the changes in #67 in our environment and it works for us.

Example log output:
Failed to apply catalog: Found 2 dependency cycles: (Exec[authconfig-mkhomedir] => File[sssd.conf] => Service[oddjobd] => Class[Sssd::Dependencies] => Class[Sssd::Config] => File[sssd.conf])

Our role:

class role::confluent::kafka::standalone_control_center {
  include profile::base
  include profile::java
  include profile::confluent::standalone::control_center
  include profile::yum::confluent
}

Profile:

class profile::confluent::control::center {
  include ::confluent::control::center
  include ::profile::nginx

  nginx::resource::server { 'confluent-control-center':
    server_name => [ $facts['networking']['fqdn'] ],
    listen_port => 80,
    proxy       => 'http://localhost:9021/',
  }
}

If you're not able to reproduce and need/want a debug output I can provide you an anonymized one.

This is the sssd module we're using https://forge.puppet.com/sgnl05/sssd

Ensure that control center has a top level property to configure the rocksdb storage.

Remove dependency on ini_file for properties environment files.