hashicorp / nomad-pack Goto Github PK

License: Mozilla Public License 2.0

Makefile 0.92% Go 85.76% HCL 10.69% Smarty 2.09% Shell 0.19% Dockerfile 0.35%

nomad-pack's Introduction

Nomad Pack

Nomad Pack is a templating and packaging tool used with HashiCorp Nomad. Nomad Pack is currently Generally Available (GA) as of nomad-pack v0.1.0.

Nomad Pack is used to:

Easily deploy popular applications to Nomad
Re-use common patterns across internal applications
Find and share job specifications with the Nomad community

Installation

To simplify the getting started experience, you can download a precompiled binary and run it on your machine locally. After downloading Nomad Pack, unzip the package. Make sure that the nomad binary is available on your PATH. You can inspect the locations available on your path by running this command:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The output is a list of locations separated by colons. You can make Nomad Pack available by moving the binary to one of the listed locations, or by adding Nomad Pack's location to your PATH.

Nomad Pack is also available as a Docker image. With Docker installed on your local machine, you can pull the latest image by running the following command:

$ docker pull hashicorp/nomad-pack:0.0.1-techpreview1

Nightly Builds and Docker Image Preview

Nomad Pack is under constant updates, so every day the nightly release is updated with binaries built off the latest code in the main branch. This should make it easier for you to try new features and bug fixes.

Each commit to main also generates a preview Docker image that can be accessed from the hashicorppreview/nomad-pack repository on Docker Hub.

Usage

Dependencies

Nomad Pack users must have Nomad running and accessible at the address defined in the NOMAD_ADDR environment variable.

If Nomad ACLs are enabled, a token with proper permissions must be defined in the NOMAD_TOKEN environment variable.

Basic Use

First, run the list command to see which packs are available to deploy.

nomad-pack list

If you don't see the default registry (nomad registry list), it can be added with the registry add command.

nomad-pack registry add default github.com/hashicorp/nomad-pack-community-registry

To deploy one of these packs, use the run command. This deploys each jobs defined in the pack to Nomad. To deploy the hello_world pack, you would run the following command:

nomad-pack run hello_world

Each pack defines a set of variables that can be provided by the user. To get information on the pack and to see which variables can be passed in, run the info command.

nomad-pack info hello_world

Values for these variables are provided using the --var flag.

nomad-pack run hello_world --var message=hola

Values can also be provided by passing in a variables file. See the variables section of the Detailed usage guide for details.

tee -a ./my-variables.hcl << END
message="bonjour"
END

nomad-pack run hello_world -f ./my-variables.hcl

If you want to remove all of the resources deployed by a pack, run the destroy command with the pack name.

nomad-pack destroy hello_world

Adding Non-Default Pack Registries

When using Nomad Pack, the default registry for packs is the Nomad Pack Community Registry. Packs from this registry will be made automatically available.

You can add additional registries by using the registry add command. For instance, if you wanted to add an example Gitlab registry, you would run the following command to download the registry.

nomad-pack registry add example gitlab.com/mikenomitch/pack-registry

To view the packs you can now deploy, run the registry list command.

nomad-pack registry list

Packs from this registry can now be deployed using the run command.

Writing your own Packs

Nomad Pack is valuable when used with official and community packs, but many users will also want to use their own.

Converting your existing Nomad job specifications into reusable packs is achievable in a few steps, see the Writing Packs documentation for more details.

Pack Registries

Packs are organized into "registries" which contain multiple packs and shared templates.

The Nomad Pack Community Registry is a public registry for community-maintained packs. Nomad community members are encouraged to share their packs and collaborate with one another in this repo.

Pull Requests and feedback on both repositories are welcome!

Upcoming Features and Changes

Support for Volumes and ACLs
Support for other Version Control Systems
Pack search command
Integration into the official Nomad CLI

Additional Resources

Tutorials

nomad-pack's People

Contributors

Stargazers

Watchers

Forkers

rickygrassmuck mrborderline thatsk kihaya acornies derekstrickland vasilegroza pgporada quinndiggity doytsujin sungaohui zamyatin kennith kseckinc angelonfira eveld angrycub isabella232 majklovec harunor attachmentgenie bernardovale piotmni codyja giomj daleki robloxrob paladin-devops jsp-hashicorp rob-opsi gucio destinf yigene brandonromano ttys3 slonopotamus stind orucakpinar aetheriaxai andybaran bacchusjackson nightwolfz david-yu lyrandy

nomad-pack's Issues

refactor context & registry initialization check logic to happen in every command

on every command, it should validate the global cache, and initialize/download the default or specified registry

implementation:

this all goes into the base command code
possibly refactor for better cache helpers

tests: fix cli tests

A couple issues when running the entire cli test file:

We're running into what looks like issues with concurrency and the job names, which is nomad_example in all our tests (either hardcoded or default name)
We need to fix the ui output--it looks like maybe we want to switch to something thread safe like the mockUI in nomad or maybe even suppress the output since it's very verbose

deps: rename Go module to reflect true name

Change default tempalte delimiters to double square-brackets

The default template delimiters for Go Template are "{{" and "}}", this conflicts with HCL and with templates used within the jobs themselves.

This can get unfortunate looking and confusing. For instance, look at this config - https://github.com/hashicorp/nomad-pack-community-registry/blob/main/nginx/templates/nginx.nomad.tpl#L34

If we change to "[[" and "]]", we'll be less likely to conflict with HCL & other nested templates, and match Levant.

meta pack name uses local path for name

"Meta": {
    "pack-version": "01590aa",
    "pack": "/Users/mike/.nomad/packs/mike/hello_world_service",
    "pack-deployment-name": "hello_world_service@01590aa",
    "pack-deployment-timestamp": "2021-10-01 16:14:49.548709 +0000 UTC",
    "pack-job": "hello_world"
  },

it took the local path for the name, which is a bug

chore: organise repository packages before public release

the internal package doesn't need a pkg sub-dir
cli, flag, and terminal should be moved into the internal package

Destroy conflict check has invalid logic

The logic for checking for conflicts in Destroy looks wrong. At the very least the first check won't work, because if you query a job by ID and there isn't one the API will return a 404 not found error.

cli: plan should run canonicalization on job specification before planning

When planning a deployed job without changes, the returned diff includes a removal for the custom Nomad Pack meta parameters. This is because job canonicalization adds additional data to a job specification including Nomad Pack metadata and has the potential to set Consul/Vault tokens and namespaces.

The plan command should therefore be updated to include additional flags that can modify the job specification, alongside calling the job canonicalization function, before submitting the plan.

On init, the `--from` flag appears to be ignored

When I tried switching the upstream repo URL while running nomad-pack init the --from flag appears to have no effect:

nomad-pack on  main via 🐹 v1.16.5 
❯ ./bin/nomad-pack init --from https://github.com/hashicorp/nomad-pack-registry.git
  created global cache directory: /Users/lennon/.nomad/packs
  Initializing registry...
  Downloading source from [email protected]:hashicorp/nomad-pack-registry.git
  Installing into: /Users/lennon/.nomad/packs/default

The local clone created confirms the the upstream is using the SSH ([email protected]:...) URL, despite the --from line specifying an HTTP origin:

❯ cat ~/.nomad/packs/default/.git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	ignorecase = true
	precomposeunicode = true
[remote "origin"]
	url = ssh://[email protected]/hashicorp/nomad-pack-registry.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
	remote = origin
	merge = refs/heads/main

This is on main as of a7ef2357

enhancement: Move version specifier to separate flag and rebrand as revision

The @Version specifier in the registry add --from flag is cumbersome to users and leads to a lot of error-prone string parsing. We should make this it's own flag. We should also rename it to revision as that is what it more accurately represents - the git revision. We'll have to think about how this works or doesn't work with filesystem-based registries.

ci automation w/ circleci

Handle metadata updates in plan diff

If you plan an already extant job, you get output along the lines of this:

+/- Job: "example"
-   Meta[pack-deployment-name]:      "testing"
-   Meta[pack-deployment-timestamp]: "2021-09-20 22:31:04.999429 +0000 UTC"
-   Meta[pack-job]:                  "example"
-   Meta[pack-version]:              "278c005"
-   Meta[pack]:                      "nom/repo/packs/example"
+/- Task Group: "cache" (1 create/destroy update)
  +/- Task: "redis" (forces create/destroy update)
    +/- Config {
      +/- image:    "redis:latest" => "redis:6.0"
          ports[0]: "db"
        }

This can get confusing because it shows Meta being deleted, instead of the individual fields staying the same/getting mutated, which is what actually happens (here, e.g. the only field getting changed should be timestamp). We should incorporate the metadata in the diff and also get rid of timestamp since it's not needed and makes the runs not idempotent

chore: Implement validators on MetadataApp and MetadataPack

chore: be consistent in quoting job/deployment names

We sometimes put single quotes around job names, deployment names, etc.

We should be consistent and quote everywhere, or quote nowhere. Also, we might want to consider being consistent with what we do in nomad, which uses double quotes instead of single quotes.

update Meta stanza

nomad-pack currently injects a Meta stanza in the nomad jobspec. The ask is to change the keys to conform to the standard that nomad follows, eg

  "Meta": {
    "pack.name": "nginx",
    "pack.version": "0.0.1"
  }

This is required to have Nomad UI display data about packs

chore: make indentation consistent

Polish: when nomad-pack prints out Info and Success messages, the indentation level is different (it looks like success has none, and info has 2 spaces?). You can see this if you destroy a pack, e.g.:

  EvalID: "2aaf20a7-a664-4ebb-863f-c832e16556a1"
Job "foo" destroyed
Pack "nomad_example" destroyed

This looks a little wonky and we should try to be consistent in padding/indentation

packs: update CLI and registries to use nested packs directory for packs

The packs within a registry should live under a packs sub-directory. We therefore need to update the CLI, both registries, and any documentation to reflect this.

move ci automation to github actions

depends on #23

enhancement: Make `registry list` hierarchical

Since the registry list command outputs both the registries and their packs, it would be nicer if the tabular output were hierarchical by the registry.

stop/destroy ignores var overrides (and is greedily destructive)

Destroy currently only checks pack name and deployment name, and then deletes all jobs that match. This means that if you try to scope destroy to a single job, it will still destroy all jobs in a given pack and deployment, even if the job you're trying to destroy doesn't actually exist.

e.g. if you have jobs "foo" and "bar" running in deployment "dev" and try to destroy job "baz" (nomad-pack destroy nomad_example --name dev --var job_name=baz), nomad-pack will destroy jobs foo and bar.

If there are multiple jobs, we may want to consider whether just saying nomad-pack destroy nomad_example --name dev destroys all jobs in the pack, or whether we want to return an error and ask the user to specify which one.

`nomad pack render` only accepts flags after pack name

$ nomad-pack version
Nomad Pack v0.0.1-alpha (befafc4)

$ nomad-pack render --var region=global dev:hello_world@latest
! Failed To Find Pack

        Error:   stat /Users/laoqui/.nomad/packs/default/--var: no such file or directory
        Type:    *fs.PathError
        Context:
                 - Pack Name: --var
                 - Repo Name: default
                 - Registry Path: /Users/laoqui/.nomad/packs/default
                 - Pack Path: /Users/laoqui/.nomad/packs/default

bug: Registry add with target shows wrong pack in output

chore: remove use of term repo and replace with registry

Repo/Repository is an implementation detail, we should use the term registry to avoid confusion.

admin: configure repository labels using team-nomad Terraform automation

Before making the repository public, we should remove the default labels and configure our standard, plus custom labels using the Terraform modules held with the team-nomad repo.

Nomad Pack - Tech Preview Feedback

Thank you for trying Nomad Pack during this technical preview release. We are really keen to hear any and all feedback related to the Nomad Pack application, Nomad Pack Registries, and Nomad packs. This issue is designed to track general feedback from the community that will help drive design changes as we move towards a beta release. When providing feedback, including any use cases and requirements will be hugely beneficial.

If you have found a bug please raise this as a separate issue against the repository so that we can track this independently. Please also use the Nomad Discuss forum for general usage questions.

Thanks from the Nomad Team!

manager: the pack manager should return errors.WrappedUIContext

The manager package is responsible for parsing the variables, reading the pack, and rendering the templates. It currently returns a standard error type which means complex HCL diagnostics are returned as a single string. In order to present any errors in a better format, the manager should return an array of errors.WrappedUIContext. Callers can then loop over this array and use the UI errorwithcontext function.

Add Download section to Readme

Near the top of the readme (see code comments), add a link to download Nomad Pack and/or explanation of how to do so (and how to add to PATH).

refactor list command for multi-source

Readme and Guide updates

Update the Readme and guides:

Add info on adding registries and listing
Add status docs to Advanced Usage guide
Confirm every step
Change delimiter in write your own pack
Add docs on how to locally write and test packs (to the write your own guide)
Change the example registry link (and add to the example registry)
Add information on how to download
Add Contribution doc (including how to build locally)

update registries

placedholder ticket to merge existing pulls in community & default registries

docs: add documentation generation command for packs

When authoring packs we would like an easy way to ensure some basic documentation is present. Nomad Pack already includes methods for parsing the variables file along with the metadata file, therefore it would be possible to add a command which could generate a basic README file using a template and parsing the pack.

move copied nomad code to glint.Table components

Some of our output is copied directly from nomad and formats strings (e.g. createStatusListOutput in cli/stop.go) whereas our newer code uses glint.Table components, e.g. We should standardize our components for a consistent UI

Can get into stuck state when redeploying job

I took down a pack, then tried to redeploy it later. I wasn't able to run it because of a conflict, but I also wasn't able to destroy it and start again.

Add var-file command to initialize a variables file

nomad-pack var-file <PACK-NAME> ./optional/path/to/file.vars

Given a pack, a user can run a command to generate a variable file to use with that pack (The file that is used to input variables instead of CLI input variables, NOT the variable declaration)
The overrides file will be populated with the default variables from the pack

bug: Mark name as required in registry delete

This flag needs to be marked as required.

Add `nomad-pack scaffold pack`

Creates an example pack in the working directory (or path provided)

chore: Look at refactoring cli helpers to domain specific pkgs

The CLI helpers file has grown to include a lot of domain-specific methods. Should we look at refactoring to pkgs?

dependencies between registries

cli: add generate-docs command to generate skeleton pack readme

When authoring packs, we want to encourage writing useful documentation whilst making every effort to streamline this process. Luckily, Nomad Pack has both a way to read the root variables file, and render arbitrary templates. We could therefore use these internal processes to render a basic README based on the passed root variables and pack information.

variables needs description field

feature: Add registry subcommands with version support

Users should be able to install packs from the same registry at different versions and consume packs at different versions on a per-target project basis.

Requirements:

connect local registry to local directory

i.e. they may want to use a pack they are developing locally without having to push & pull from github

makefile: enhance functionality to add checking and lint

In order to make development both internally and externally easier, the makefile should include checks and lints. These help reduce friction when PRs are raised and also helps catch issues before they are merged into main.

checking of the Go module to ensure no changes are needed eg. Nomad Autoscaler
linting of code base using golangci-lint, ideally with staticcheck enabled eg. Nomad
addition of help command eg. Nomad

plan
run
destroy

registry auto-refresh

related to work on versioning #8

nomad pack status

gets list of what packs it's deployed & relevant metadata
- job name
- deployment name
- pack name
allows users to destroy stuff with nomad-pack destroy