travisjeffery/terraform-provider-kafka

Terraform provider for managing Kafka topics.

Terraform Kafka Topic Provider

This is a Terraform provider for managing Kafka topics with Terraform.

Why use this Kafka provider?

• Supports adding partitions and altering configs
• Supports TLS/SASL
• Uses Kafka's new admin APIs rather than shelling out to old bash scripts

Installation

1. Download the latest compiled binary from GitHub releases.

2. Unzip/untar the archive.

3. Move it into $HOME/.terraform.d/plugins:

   $ mkdir -p $HOME/.terraform.d/plugins
   $ mv terraform-provider-kafka $HOME/.terraform.d/plugins/terraform-provider-kafka

4. Create your Terraform configurations as normal, and run terraform init:

   $ terraform init

   This will find the plugin locally.

Usage

1. Create a Terraform configuration file:

   provider "kafka" {
   	hosts = ["localhost:9092"]
   }
   
   resource "kafka_topic" "example" {
   	name: "example"
   	num_partitions: "8"
   	replication_factor: "1"
   	config_entries: {
   		retention.bytes: "102400"
   		cleanup.policy: "compact
   	}
   }

   There's parameters to set if you use TLS/SASL.

2. Run terraform init to pull in the provider:

   $ terraform init

3. Run terraform plan and terraform apply to interact with the filesystem:

   $ terraform plan
   
   $ terraform apply

Importing topics

This provider supports importing externally created topics by their name. Assuming you've already created a topic declaration like the one above, you can get Terraform to manage the state of the existing topic:

$ terraform import kafka_topic.example example

Examples

For more examples, please see the examples folder in this repository.

License

MIT

---

• travisjeffery.com
• GitHub @travisjeffery
• Twitter @travisjeffery
• Medium @travisjeffery

terraform-provider-kafka's Issues

feature request: sasl scram support

As the title says, SASL SCRAM support would be nice. SCRAM is possible with Samara.

Terraform v0.11.14 - TERRAFORM CRASH

terraform {
  backend "azurerm" {
    storage_account_name = "iacstates"
    container_name = "terrformstates"
    key = "kafka-topics.tfstate"
  }
}

provider "kafka" {
	hosts = ["x"]
}

resource "kafka_topic" "example" {
	name= "example"
	num_partitions="8"
	replication_factor="1"
	config_entries= {}
}

kafka_topic.example: Refreshing state... (ID: example)

Error: Error refreshing state: 1 error occurred:
* kafka_topic.example: kafka_topic.example: unexpected EOF

panic: runtime error: index out of range
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka:
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: goroutine 113 [running]:
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: github.com/travisjeffery/terraform-provider-kafka/topic.read(0xc420254e70, 0xe4c5e0, 0xc420534120, 0xc420254e70, 0x0)
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: /Users/travis/dev/src/github.com/travisjeffery/terraform-provider-kafka/topic/resource.go:169 +0x64f
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc4201e6070, 0xc4201fa820, 0xe4c5e0, 0xc420534120, 0xc420106978, 0x4bfb01, 0xca2c40)
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: /Users/travis/dev/src/github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:354 +0x167
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc4201e60e0, 0xc4201fa7d0, 0xc4201fa820, 0x18, 0x18, 0xc42045f740)
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: /Users/travis/dev/src/github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc42000a120, 0xc420223dd0, 0xc420570040, 0x0, 0x0)
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: /Users/travis/dev/src/github.com/travisjeffery/terraform-provider-kafka/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: reflect.Value.call(0xc4200e7740, 0xc4202140e8, 0x13, 0xe639ca, 0x4, 0xc4200d8f18, 0x3, 0x3, 0xc420202600, 0x0, ...)
2019-06-21T11:30:58.508-0300 [DEBUG] plugin.terraform-provider-kafka: /usr/local/go/src/reflect/value.go:447 +0x969
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: reflect.Value.Call(0xc4200e7740, 0xc4202140e8, 0x13, 0xc4204eaf18, 0x3, 0x3, 0x0, 0x0, 0x0)
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: /usr/local/go/src/reflect/value.go:308 +0xa4
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: net/rpc.(*service).call(0xc4200e3480, 0xc42020a2d0, 0xc4200321d0, 0xc420032200, 0xc4203cad00, 0xc42045e8e0, 0xca2c00, 0xc420223dd0, 0x16, 0xca2c40, ...)
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: /usr/local/go/src/net/rpc/server.go:384 +0x14e
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: created by net/rpc.(*Server).ServeCodec
2019-06-21T11:30:58.509-0300 [DEBUG] plugin.terraform-provider-kafka: /usr/local/go/src/net/rpc/server.go:480 +0x43a
2019/06/21 11:30:58 [ERROR] root: eval: *terraform.EvalRefresh, err: kafka_topic.example: unexpected EOF
2019/06/21 11:30:58 [ERROR] root: eval: *terraform.EvalSequence, err: kafka_topic.example: unexpected EOF
2019/06/21 11:30:58 [TRACE] [walkRefresh] Exiting eval tree: kafka_topic.example
2019/06/21 11:30:58 [TRACE] dag/walk: upstream errored, not walking "provider.kafka (close)"
2019-06-21T11:30:58.510-0300 [DEBUG] plugin: plugin process exited: path=/home/jonatasfreitasv/.terraform.d/plugins/terraform-provider-kafka
2019/06/21 11:30:58 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-06-21T11:30:58.620-0300 [WARN ] plugin: error closing client during Kill: err="connection is shut down"

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform1 so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!