pdmthorsrud/terraform-nomad-presto

Terraform module to set up presto on nomad

Terraform-nomad-presto

---

Module contains a nomad job ./conf/nomad/presto.hcl with presto sql server.

Additional information:

• consul-connect integration
• nomad docker driver

Contents

0. Prerequisites
1. Compatibility
2. Usage
   1. Requirements
      1. Required software
   2. Providers
3. Inputs
4. Outputs
5. Examples
6. Authors
7. License
8. References

Prerequisites

Please follow this section in original template

Compatibility

Software	OSS Version	Enterprise Version
Terraform	0.13.1 or newer
Consul	1.8.3 or newer	1.8.3 or newer
Vault	1.5.2.1 or newer	1.5.2.1 or newer
Nomad	0.12.3 or newer	0.12.3 or newer

Usage

make up

Check the example of terraform-nomad-presto documentation here.

Example contains csv, json, avro, protobuf file types.

Requirements

Required software

See template README's prerequisites.

All software is provided and run with docker. See the Makefile for inspiration.

Providers

This module uses the Nomad provider.

Inputs

Name	Description	Type	Default	Required
nomad_provider_address	Nomad provider address	string	"http://127.0.0.1:4646"	yes
nomad_data_center	Nomad data centers	list(string)	["dc1"]	yes
nomad_namespace	[Enterprise] Nomad namespace	string	"default"	yes
nomad_job_name	Nomad job name	string	"presto"	yes
shared_secret_provider	Provider for the shared secret: user or vault	string	"user"	no
shared_secret_user	Shared secret provided by user(length must be >= 12)	string	"asdasdsadafdsa"	no
shared_secret_vault	Set of properties to be able fetch shared cluster secret from vault	object	default = { vault_kv_policy_name = "kv-secret", vault_kv_path = "secret/data/presto", vault_kv_secret_key_name = "cluster_shared_secret"}`	no
service_name	Presto service name	string	"presto"	yes
port	Presto http port	number	8080	yes
docker_image	Presto docker image	string	"prestosql/presto:333"	yes
container_environment_variables	Presto environment variables	list(string)	[""]	no
use_canary	Uses canary deployment for Presto	bool	false	no
consul_connect_plugin	Deploy consul connect plugin for presto	bool	true	no
consul_connect_plugin_artifact_source	Artifact URI source	string	"https://oss.sonatype.org/service/local/repositories/releases/content/io/github/gugalnikov/presto-consul-connect"	no
hivemetastore.service_name	Hive metastore service name	string	"hive-metastore"	yes
hivemetastore.port	Hive metastore port	number	9083	yes
minio.service_name	minio service name	string		yes
minio.port	minio port	number		yes
minio.access_key	minio access key	string		yes
minio.secret_key	minio secret key	string		yes

Outputs

Name	Description	Type
presto_service_name	Presto service name	string
presto_port	Presto port	number

Examples

module "presto" {
  depends_on = [
    module.minio,
    module.hive
  ]

  source = "github.com/fredrikhgrelland/terraform-nomad-presto.git?ref=0.0.1"

  nomad_job_name    = "presto"
  nomad_datacenters = ["dc1"]
  nomad_namespace   = "default"

  service_name = "presto"
  port         = 8080
  docker_image = "prestosql/presto:341"

  #hivemetastore
  hivemetastore = {
    service_name = module.hive.service_name
    port         = 9083
  }

  # minio
  minio = {
    service_name = module.minio.minio_service_name
    port         = 9000
    access_key   = module.minio.minio_access_key
    secret_key   = module.minio.minio_secret_key
  }
}

For detailed information check example/ directory.

Verifying setup

You can verify successful run with next steps:

Option 1 [hive-metastore and nomad]

• Go to http://localhost:4646/ui/exec/hive-metastore
• Chose metastoreserver -> metastoreserver and click enter.
• Connect using beeline cli

# from metastore (loopback)
beeline -u jdbc:hive2://

• Query existing tables (beeline-cli)

SHOW DATABASES;
SHOW TABLES IN <database-name>;
DROP DATABASE <database-name>;
SELECT * FROM <table_name>;

# examples
SHOW TABLES;
SELECT * FROM iris;
SELECT * FROM tweets;

Option 2 [presto and nomad]

• Go to http://localhost:4646/ui/exec/presto
• Chose standalone -> server and click enter.
• Connect using presto-cli

presto

• Query existing tables (presto-cli)

SHOW CATALOGS [ LIKE pattern ]
SHOW SCHEMAS [ FROM catalog ] [ LIKE pattern ]
SHOW TABLES [ FROM schema ] [ LIKE pattern ]

# examples
SHOW CATALOGS;
SHOW SCHEMAS IN hive;
SHOW TABLES IN hive.default;
SELECT * FROM hive.default.iris;

Option 3 [local presto-cli]

NB! Check required software section first.

• in a terminal run a proxy and presto-cli session

make presto-cli

• Query tables (3 tables should be available)

show tables;
select * from <table>;

To debug or continue developing you can use presto cli locally. Some useful commands.

# manual table creation for different file types
presto --server localhost:8080 --catalog hive --schema default --user presto --file ./example/resources/query/csv_create_table.sql
presto --server localhost:8080 --catalog hive --schema default --user presto --file ./example/resources/query/json_create_table.sql
presto --server localhost:8080 --catalog hive --schema default --user presto --file ./example/resources/query/avro_tweets_create_table.sql

Authors

License

This work is licensed under Apache 2 License. See LICENSE for full details.

---

References

• Blog post
• Presto, so far (release 340), supports only varchar columns