grafana / tanka Goto Github PK
View Code? Open in Web Editor NEWFlexible, reusable and concise configuration for Kubernetes
Home Page: https://tanka.dev
License: Apache License 2.0
Flexible, reusable and concise configuration for Kubernetes
Home Page: https://tanka.dev
License: Apache License 2.0
Can we include the namespace in this?
!!! Applying to cluster 'gke_grafanalabs-global_europe-west2-c_europe-west2' at 'https://35.197.236.86' using context 'eu-west2'.
!!! Please type 'yes' to perform: yes
And maybe remove the exclamations, makes it seem more serious than it is
The --target
argument is extremely useful, allowing one to selectively apply changes.
However, if a tk diff
shows pages of output (e.g. one huge diff that you don't want to apply and lots of smaller ones that you do), it can be hard to gather the names of the resources you wish to apply.
A tk diff --diff-strategy=abbreviated
which just says "these resources will change" but excludes the details of the changes, would be exceptionally helpful in this scenario.
$ tk diff -v environments/default/hg-free-us-central1.default
Warning: `namespace` is deprecated, use `spec.namespace` instead.
Warning: `server` is deprecated, use `spec.apiServer` instead.
Warning: `team` is deprecated, use `metadata.labels.team` instead.
$ echo $?
16
For automation.
It looks weird.
If I do cd environments/
then press tab, I will see listed the directories inside environments
.
If I do the same with tk diff environments
, I will see environments/default/blah
. It would be more helpful if it stripped off the environments
portion, as I have already committed to that part of the path.
Also, it currently lists all of the paths that have a main.jsonnet
. This is neat, but gives too many options. I would expect a sequence such as:
$ tk diff env<TAB>
$ tk diff environments/
$ tk diff environments/<TAB>
default hosted-grafana cortex loki cassandra blah
$ tk diff environments/cor<TAB>
$ tk diff environments/cortex/
$ tk diff environments/cortex/<TAB>
eu-west2.dev eu-west2.prod ops-tools1-us-east4.cortex blah
$ tk diff environments/cortex/ops<TAB>
$ tk diff environments/cortex/ops-tools1-us-east4.cortex
Here it has stopped completing, i.e. didn't put a /
at the end of the dir, because there is a main.jsonnet
file in that directory.
This would allow me to navigate swiftly to where I want to be without overfilling my screen.
We currently have to install Jb to vendor jsonnet libraries.
It would be nice if Tanka has a command to deal with this without installing jb.
tk bundle vendor or something similar
Looking at this project, I think it should be possible to make this interface with the kubectl plugins and distribute via krew.
https://github.com/kubernetes-sigs/krew
Let me know what you think!
Just out of interest, what would have had to be done to migrate to Tanka? We designed tk to be very similar to ks so that it should be a flawless experience for most users .. could you elaborate a bit so we could improve? :D
The learning curve for ksonnet was pretty steep, so to be honest I just figured out enough to get the applications going that I needed it for and then I edited away without putting much effort into understanding how things worked underneath.
I think I'm mostly getting stuck because I don't understand what the "magic" that ksonnet did and how to replace it. For example with ks
the params.libsonnet files in environments/{beta, prod} and components/ were automatically loaded and merged. Or rather, the environment specific one was loaded and then finished with
{
components: {
[x]: envParams.components[x] + globals
for x in std.objectFields(envParams.components)
},
}
to do the merging. However, in my components I could just ask for local params = std.extVar('__ksonnet/params').components.foobar;
and get the foobar
parameters. I'm not sure how to make this work with tanka.
With ksonnet all the specified components were automatically, I think, imported and merged together. It's probably a straightforward bit of jsonnet to add to my main.libsonnet
, but it's not obvious to me how.
Would be good to hide these warnings:
2019/08/08 11:22:11 Warning: `namespace` is deprecated, use `spec.namespace` instead.
2019/08/08 11:22:11 Warning: `server` is deprecated, use `spec.apiServer` instead.
2019/08/08 11:22:11 Warning: `team` is deprecated, use `metadata.labels.team` instead.
We're not going to migrate to the new env format any time soon (will break others), so they're inactionable.
Using the 0.6.0 release, I'm getting the following error message when running tk apply
: error: no objects passed to apply
.
This doesn't happen with 0.5.0.
Like git diff
does.
As experienced in #95, due to recent fixes to https://github.com/jsonnet-bundler/jsonnet-bundler it is not possible anymore to install single files of a package but only the whole package at once.
However, we required this feature (which was actually a bug) to install https://github.com/ksonnet/ksonnet-lib (k.libsonnet
).
While jb
is still able to install those libraries, they end up at another import path:
- import "k.libsonnet"
+ import "ksonnet.beta.3/k.libsonnet"
This is a breaking change for most users and most imported packages, as ksonnet previously magically allowed this import style which diverts from what jsonnet-bundler
specified in its design doc.
I have no strong opinion how to tackle this issue yet, however here are some ideas around the topic for discussion:
k.libsonnet
), which equals to a orphaned file directly at the root of vendor/
. However, jb
takes a more explicit approach, where an import is usually named <package>/<file>.libsonnet
. IMHO this is the better way, as it avoids such a confusion. Golang for example even took this idea further and included the packages origin as well (<remote>/<package>/<file>.go
)ks
to tk
hard, as adopting Tanka would immediately break ksonnet. I feel like we must avoid thiss at any cost.jsonnet.Importer
: a custom implementation could wrap jsonnet.FileImporter
to alias k.libsonnet
to the appropriate path at vendor/
. In -verbose
mode it could print warnings about the deprecated import styletk lint
(linter that doesn't exist yet): Could have a lint rule warning about the deprecated importtk fix
(code-rewriter that doesn't exist either): Could have a rule to automatically adapt all import pathsWith such a combination (support of the deprecated format, plus tooling to help users switch to the new one) we could avoid a fully breaking change.
Everybody is invited and encouraged to discuss the above ideas to find the best possible solution :D
/cc @rfratto @slim-bean @cyriltovena @tomwilkie @malcolmholmes @gouthamve @brancz @metalmatze @bgagnon @beorn7
See grafana/jsonnet-libs#178 for an example. In short:
Alertmanager gossiping, similar to DNS, uses both UDP and TCP (presumably the latter as a fallback for the former). In K8s (and with ksonnet), it is no problem to define two ports with different protocol and name for the same port number. The example linked above results in the following port definition with ks
:
"ports": [
{
"name": "alertmanager-http-metrics",
"port": 80,
"targetPort": 80
}
{
"name": "alertmanager-gossip-udp",
"port": 9094,
"protocol": "UDP",
"targetPort": 9094
}
{
"name": "alertmanager-gossip-tcp",
"port": 9094,
"targetPort": 9094
}
],
Doing the same thing with tk
munges together the both alertmanager-gossip-… ports. The result is different depending on the order the two ports are defined. I assume this has to do with the way Tanka flattens the result.
Perhaps there is something we can tweak in the diff to truncate these lines?
Or perhaps subset shouldn't copy them over? Not sure why they are appearing...
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"consul","namespace":"cortex-ops"},"spec":{"minReadySeconds":10,"replicas":1,"revisionHistoryLimit":10,"template":{"metadata":{"annotations":{"consul-hash":"33b38d71a22a1025f6414c57c050d655"},"labels":{"name":"consul"}},"spec":{"affinity":{"podAntiA...
Using 0.3.0 image with an existing environment from ksonnet.
Steps to reproduce:
tk init -f
in the existing ksonnet project.tk show <environment>
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x882fd8]
goroutine 1 [running]:
github.com/grafana/tanka/pkg/kubernetes.(*Kubernetes).Reconcile(0x0, 0xc0018803f0, 0x0, 0x0, 0x0, 0xc00048bce8, 0x5b8175, 0xc0000b8500, 0xc00048bcc0, 0xa6f1af)
/drone/src/pkg/kubernetes/kubernetes.go:67 +0x118
main.showCmd.func1(0xc0000ce500, 0xc000164e90, 0x1, 0x1)
/drone/src/cmd/tk/workflow.go:145 +0x122
github.com/spf13/cobra.(*Command).execute(0xc0000ce500, 0xc00001e1a0, 0x1, 0x1, 0xc0000ce500, 0xc00001e1a0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000ce000, 0xc000164e20, 0xc00011e270, 0x0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/drone/src/cmd/tk/main.go:106 +0x43f
Not a huge dealio, and the proper init workflow works fine it's something I encountered looking to switchover from ks to tk.
Also, a few spurious diffs (that don't appear in ks diff):
diff -u -N /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff397372895/LIVE-v1.ConfigMap.dev.overrides /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff397372895/MERGED-v1.ConfigMap.dev.overrides
--- /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff397372895/LIVE-v1.ConfigMap.dev.overrides 2019-08-08 11:22:22.000000000 +0100
+++ /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff397372895/MERGED-v1.ConfigMap.dev.overrides 2019-08-08 11:22:22.000000000 +0100
@@ -5,7 +5,7 @@
"5545":
ingestion_burst_size: 50000
ingestion_rate: 25000
- max_samples_per_query: 1000000
+ max_samples_per_query: 1e+06
max_series_per_metric: 100000
max_series_per_query: 100000
max_series_per_user: 500000
@@ -13,7 +13,7 @@
accept_ha_samples: true
ingestion_burst_size: 50000
ingestion_rate: 25000
- max_samples_per_query: 1000000
+ max_samples_per_query: 1e+06
max_series_per_metric: 100000
max_series_per_query: 100000
max_series_per_user: 500000
And
diff -u -N /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff445619636/LIVE-v1.ConfigMap.dev.schema-ce56a9517e373650ad4bd9b4462c8151 /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff445619636/MERGED-v1.ConfigMap.dev.schema-ce56a9517e373650ad4bd9b4462c8151
--- /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff445619636/LIVE-v1.ConfigMap.dev.schema-ce56a9517e373650ad4bd9b4462c8151 2019-08-08 11:22:22.000000000 +0100
+++ /var/folders/g5/3pb5_td927l61zm9xfmmtcp80000gn/T/diff445619636/MERGED-v1.ConfigMap.dev.schema-ce56a9517e373650ad4bd9b4462c8151 2019-08-08 11:22:22.000000000 +0100
@@ -2,7 +2,7 @@
data:
config.yaml: |
configs:
- - from: 1969-12-31
+ - from: "1969-12-31"
index:
prefix: dev_index
schema: v9
@@ -10,7 +10,7 @@
- chunks:
period: 168h
prefix: dev_chunks_
- from: 2017-10-26
+ from: "2017-10-26"
index:
period: 168h
prefix: dev_index_
@@ -19,7 +19,7 @@
- chunks:
period: 168h
prefix: dev_chunks_
- from: 2019-02-09
+ from: "2019-02-09"
index:
period: 168h
prefix: dev_index_
@@ -29,7 +29,7 @@
- chunks:
period: 168h
prefix: dev_chunks_
- from: 2019-03-05
+ from: "2019-03-05"
index:
period: 168h
prefix: dev_index_
Some yaml flags we can tweak to make these consistent?
I followed the steps at https://github.com/grafana/tanka and it fails with the following error :
tk show environments/loki
Evaluating jsonnet: RUNTIME ERROR: couldn't open import "ksonnet-util/kausal.libsonnet": no match locally or in the Jsonnet library paths
/Users/cameronbraid/loki-config/vendor/promtail/promtail.libsonnet:1:11-49 thunk <k> from <$>
/Users/cameronbraid/loki-config/vendor/promtail/promtail.libsonnet:5:1-2 $
main.jsonnet:1:18-54 thunk <promtail> from <$>
main.jsonnet:3:1-9 $
During evaluation
I can see that ksonnet-util is a dependency in vendor/promtail/jsonnetfile.json but it doesn't seem to be installed
last i heard maintainers of ksonnet were looking for someone willing to take it over - maybe reach out?
when doing tk show
it shows me resource types, such as ConfigMap
in camel case. but when i try to do tk diff
and use the -t
parameter to select one specific resource i need to specify the resource type in lower case (configmap
) for it to work, when i write it ConfigMap
then tk
seems to not find the resource. I think this should be case insensitive.
Being able to just deploy specific objects from an env would be super useful eg tk apply env/foo/var -target configmap.consul -target deployment.consul
Slash might be a better separator for <type>/<name>
.
What happened:
Cannot pull tanka docker image because of unknown manifest.
What you expected to happen:
Tanka docker image pulling down properly.
How to reproduce it (as minimally and precisely as possible):
docker pull grafana/tanka:v0.5.0
Result:
> docker pull grafana/tanka:v0.5.0`
Error response from daemon: manifest for grafana/tanka:v0.5.0 not found: manifest unknown: manifest unknown
Anything else we need to know?:
Environment:
The dashboard config maps have gotten too big to fit in the annotation that kubectl apply
uses (NB ks doesn't use the annotation, so still works).
Tanka is kubernetes focused but it's surprising that we need to download the k8s.libsonnet anyways using bundler. I would expect that tk init
already contains the jsonnet required.
Some kubectl versions refuse to touch objects not created by themselves but by ksonnet.
Applying once with --validate=false
solves this
Limit to a single object only (i.e. only a single Deployment)
I had a typo in my spec.json. As a result, I got the below error:
While parsing config: invalid character 'd' after object key
It was not clear that this error was in spec.json, so I spent ages tracking through my ksonnet code for anything vaguely relevant, and was totally bemused for a while.
Had it said, While parsing spec.json
or something like that, I might have realised that 'config' meant json config, not the ksonnet itself.
The code says if !version.GreaterThan("1.13.0") { useSubsetDiff() }
. In the marginal case where someone is actually using 1.13.0, they will get subset, rather than native. This should really say if version.LessThan("1.13.0")
.
[root@test config]# tk env add loki --namespace loki
[root@test config]# ls environments/
default
[root@test config]# tk env list
NAME NAMESPACE SERVER
default default
loki loki
[root@test config]# ls
environments jsonnetfile.json lib loki vendor
[root@test config]# ls loki/
main.jsonnet spec.json
The new env dir should under environments
.
For helping people migrate to tk
from ks
, it would be useful for tk
to implement the env
commands that ks
has:
tk env add
tk env list
tk env rm
tk env set
This would be particularly useful in the Loki docs (grafana/loki#982).
/cc @sh0rez
At present, Tanka will not remove Kubernetes resources that have been removed from the respective Jsonnet configuration, and such removals need to be done manually.
Terraform (for example) handles this by maintaining a record of 'intended state'. If an object is present in the state file, but not in the Jsonnet configuration, then we can deduce that it needs to be removed.
Given that Kubernetes already maintains its own state, and that that is consumed for ks diff
etc, we could likely achieve deletions if we simply stored a list of resources created by this Tanka environment, within for example, a single CRD. This CRD could be named within the spec.json
file.
Thus, when either tk diff
or tk apply
are executed, Tanka can look for any resources that are present in this CRD but are not present in the Jsonnet output. Such resources are candidates for deletion. tk diff
could include them in its output, and tk apply
could use kubectl delete
to remove them.
We should consider whether deletions should be a standard feature of Tanka, or whether they would need enabling via a command line switch.
As I understand it, writing a Golang utility that includes Tanka isn't possible (in an elegant way) because it doesn't expose an API.
It would be very useful if the commands were separated out into separate packages available by import into other go code.
The https://github.com/grafana/tanka/releases/tag/v0.6.0 release says you can get the docker image using
$ docker pull grafana/tanka:v0.6.0
But the v0.6.0
tag doesn't exist, but 0.6.0-amd64
does. It seems when you switched to support multiple arches you began suffixing the image tags. .drone/release-note.md
is what looks like it needs updating.
Idea...
So users don't accidentally apply the wrong thing.
Terraform went round and round on this and eventually settled on doing it.
To improve the CLI UX, it should be possible to do e.g.
$ tk {show,diff,apply} <TAB>
default/foo default/foo prod/baz
How this exactly looks depends on #4 (Directory vs env)
Completion should probably look for folders containing a main.jsonnet
and offer these as targets
Nice suggestion by @lawrencejones: metadata.name
is crucial to every Kubernetes object. It should be added to our walkJSON
func which identifies valid objects to extract them.
Furthermore it would make sense to not only check for the presence of apiVersion
, kind
and metadata.name
, but also validate them (check that they are strings)
For testing, I sometimes launch a temporary local single-node Kubernetes cluster which receives a different IP address every time it is created. This makes deploying to it with tanka a bit difficult as the apiServer
field in spec.json requires a hostname.
I'd like to be able to optionally specify clusters by name rather than the IP address.
For example, given the ~/.kube/config
file of:
apiVersion: v1
clusters:
- cluster:
certificate-authority: /Users/robert/.minikube/ca.crt
server: https://192.168.64.3:8443
name: minikube
# ...
I would like to be able to refer to that API server in the spec.json
in one of these ways:
{
"apiVersion": "tanka.dev/v1alpha1",
"kind": "Environment",
"spec": {
// option 1: use cluster name in apiServer
"apiServer": "minikube",
// option 2: a separate key that is used if apiServer is empty
"clusterName": "minikube",
"namespace": "default"
}
}
WDYT @sh0rez?
Hi team,
We're trying out tanka for managing our core kubernetes cluster configuration but have hit a few roadbumps.
Most of our problems appear to stem from lack of understanding about what type of object tanka is expecting your jsonnet to generate from the environments file.
From looking at your examples, it appears the implicit expectation is that tanka will recursively descend your jsonnet structure, assuming that your structure is a tree with kubernetes object leaf nodes. So that assumes that you have one global object which is a map of string to object, where that object is either a kubernetes resource or another tree.
data Resource = Resource {
apiVersion :: String, kind :: String, metadata :: Metadata
}
data Environment = Resource || [(string, Environment)]
If this is so, it would be great to document it. I suspect we might be missing something though, as tanka panics when we construct several otherwise valid configs, such as:
{
a: {
apiVersion: 'v1',
kind: 'List',
items: [/* valid k8s resources */],
},
b: self.a,
}
Lacking a clear explanation about what this format is makes it difficult to adopt tanka as a tool. All the advertising says tanka is a great function
, but we don't know what the type signature is.
Can you help us out? Cheers.
tanka always read apiServer from environments/default/spec.json
, and get kubeconfig by kubectl config view -o json
, even localhost equal to ip in kubeconfig, it return:
Diffing: no cluster that matches the apiServer `https://localhost:6443` was found. Please check your $KUBECONFIG
It would be an improvement, if you like this, i feel free to file a PR for this.
One of my biggest gripes with ksonnet is difficulty understanding what's been previously applied to an environment and by whom, and this directly relates to the other reason I would like a feature like this: rollback.
It would be nice if there were a centralized transaction log (or audit log?) which could be used to both see who's been making changes to ask them questions as well as make it easy to rollback a change gone wrong.
I figured I would start an issue to discuss the pros/cons and difficulties of such a feature.
Perhaps we should only apply object we detect a diff in? WDYT?
I'd love to be able to pull secrets from vault when I apply. We do this already with tooling we have but would like to start using tanka. Any reason why we shouldn't or can't have this? Or how are others handling their secrets with tanka?
For example when I apply this jsonnet tanka would use the standard vault envvars VAULT_ADDR
and VAULT_TOKEN
to create a client for use with the native jsonnet function:
{
secret: {
apiVersion: 'v1',
kind: 'Secret',
type: 'Opaque',
metadata: {
name: 'secret',
},
data: {
password: std.native('vault')('secret/path', 'password'),
},
},
}
There are guidelines how usage strings on the CLI should be formatted: https://en.wikipedia.org/wiki/Command-line_interface#Command_description_syntax
Tanka does not accord to them, but should.
Somebody might want to grep -R "Use:
the project and reformat all Use
messages accordingly.
Help is welcome!
Currently, the baseDir
is evaluated against the current pwd
.
To simplify scripts, it should be possible to directly specify the environment on the CLI:
tk ACTION default/foo
or as an alternative
tk ACTION environments/default/foo
While the first option is shorter, the second one is more general, as the argument is an actually valid path on the local file system. This integrates better with the current approach of not enforcing a directory structure, apart from the presence of main.jsonnet
and jsonnetfile.json
quoting @lawrencejones:
Worth considering if you want to enable other kubectl flags. Several popular libraries (I'm looking at you, cert-manager) currently require you to set --validate=false to apply the manifests. You could use jsonnet to make the vendored resources valid, but I think many people may want to follow the official docs?
We should allow to invoke kubectl
with --validate=false
by exposing that exact flag on the tk apply
operation.
Note: This should be done after #111 is merged
reproduce:
# GO111MODULE=on go mod download
# go build ./cmd/tk
../../sh0rez/go-jsonnet/desugarer.go:27:2: cannot find package "github.com/google/go-jsonnet/parser" in any of:
/usr/lib/golang/src/github.com/google/go-jsonnet/parser (from $GOROOT)
/deploy/go/src/github.com/google/go-jsonnet/parser (from $GOPATH)
It might sound appealing to users to do the following:
$ tk show | kubectl apply -f
While this appears to be straightforward, it has a severe shortcoming: It evades context handling, by relying on a correctly set up kubectl
. Usually, tk
parses the output of kubectl config view
and automatically selects the correct context
based on the spec.kubernetes.apiServer
field of spec.json
.
This means in the worst case, a quick tk show | kubectl apply -f
could result in the disastrous action of applying a config to the wrong cluster.
This will most likely have a severe impact on the running workloads and should be mitigated by design. Possible solutions:
tk show
. The ANSI color chars will make piping impossibletk show
emit to stderr. While the redirection is easy, it is counterintuitive and thus a step in the right direction⚠️ Do not pipe the output of tk show. Use tk apply instead ⚠️
messages to the output.While these should be the default, it is fair to add a --allow-dangerous-piping
flag which will result in the current behaviour.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.