herrjulz / aviator Goto Github PK
View Code? Open in Web Editor NEWMerge YAML/JSON files in a in a convenient fashion based on a configuration file called aviator.yml
License: MIT License
Merge YAML/JSON files in a in a convenient fashion based on a configuration file called aviator.yml
License: MIT License
(To be entirely honest, I'm not 100% sure what the workflow for this would look like yet, just putting an issue here as a way to point this out. Feel free to close this whenever you feel fit.)
I'm investigating using ytt
as a way to templatize my concourse pipelines. I thought this might be an interesting project to check out and see if/how it could integrate to aviator
.
Thanks for your time!
I would like to use aviator in a meta (concourse) pipeline creating other pipelines. For that to be possible fly needs to be run in non-interactive mode (with -n argument).
I would appreciate adding an option along the lines of:
fly:
non-interactive: true
that would result appending '-n' to fly args
When triggering aviator -v
, I see the following:
Aviator version 0.1.0
But I would expect to see the latest version(which I installed from https://github.com/JulzDiverse/aviator/releases/tag/v0.2.0
Example:
input.yml
foo:
bar: 2000
avaitor.yml
- base: input.yml
modify:
set:
- path: .foo.baz
value: [qux]
Expected output:
foo:
bar: 2000
baz: [qux]
/cc @gdankov
Hi folks,
testing aviator behaviour and going through the code I noticed that the except
piece isn't into account when include_sub_dirs
is present. Example:
- base: base.yml
for_each:
in: configs
include_sub_dirs: true
regexp: ".*.yml"
except: [test.yml]
to_dir: ++tmp/base_output/++
I am looking to use aviator to build a large number of similar pipelines. The current example shows building multiple apps and merging them into a single pipeline, followed by a call to fly to install that pipeline.
I would like to have a single aviator file that builds multiple pipelines and calls fly on each generated pipeline.
As an example - take the cf sample and remove the last merge - you would have multiple app pipelines that would need to be installed.
https://github.com/JulzDiverse/aviator-cf-apps/blob/master/aviator.yml
Currently, if you set expose: true
your pipeline will be hidden. Setting it to false, though, does nothing. I've opened a PR at #34 which enables previously exposed pipelines to be hidden again.
e.g. change chain
to plan
Hi,
Trying out this tool and I have a stupid question. Why doesn't merge merge?
thing1.yaml:
a:
b: c
thing2.yaml:
a:
d: e
spruce merge
does what I expect:
$ spruce merge thing1.yaml thing2.yaml
a:
b: c
d: e
But aviator
seems to ignore the second file.
$ cat aviator.yml
spruce:
- base: thing1.yaml
merge:
- with:
files:
- thing2.yaml
to: things.yaml
$ aviator
SPRUCE MERGE:
thing1.yaml
to: things.yaml
$ cat things.yaml
a:
b: c
Why didn't the contents of thing2.yaml
get merged into the output?
Hi,
Not sure if my finding is correct but it looks to me that if fly errors out, then (not every?) exit code from fly gets passed down as aviator exit code.
Since my use case is CI, I care for the pipeline to fail in case of error.
The behavior I am observing:
Add a "put" to pipeline definition with misspelled name, or in any other way not existing among the pipeline resources, say:
- put: docker-imag
params:
build: build-result
Run set-pipeline with fly directly:
fly -t target sp -p pipeline-foo -c pipeline-foo.yml -n
Result:
[...]
apply configuration? [yN]: y
error: invalid configuration:
invalid jobs:
jobs.pipeline-foo.plan[5].put.docker-imag refers to a resource that does not exist
$ echo $?
1
Doing the same with aviator. My aviator-foo.yml for testing:
fly:
target: target
name: pipeline-foo
config: pipeline-foo.yml
non_interactive: true
Running:
aviator -f aviator-foo.yml
Result:
[...]
apply configuration? [yN]: y
error: invalid configuration:
invalid jobs:
jobs.pipeline-foo.plan[5].put.docker-imag refers to a resource that does not exist
$ echo $?
0
For errors from fly, like this one, it would help me if aviator exited with non-zero code. Is it possible to do? Thanks in advance.
for_each
is useful for enumerating a list of targets. I use it to loop over multiple Debian and Ubuntu releases, say ["buster", "bullseye", "bionic", "focal"]
. So I have a distros/
folder with one yaml file in it per target distribution, each with the tag names and other details that are necessary for building for that distro.
I'd like to add target architectures to that, but it does not seem possible. I want to add a second loop over the target architecutres, say ["aarch64", "powerpc64", "x86_64"]
, which works the same way: an arches/
folder with one yaml file in it per target architecture, each with the details necessary for building for that architecture.
The desired result would be a nested pair of for-loops, one for distro, one for architecture. Like this pseudo-code:
for arch in ["aarch64", "powerpc64", "x86_64"]:
for distro in ["buster", "bullseye", "bionic", "focal"]:
arch_fn = "arches/" + arch + ".yaml"
distro_fn = "distros/" + distro + ".yaml"
output = merge(all_inputs, arch_fn, distro_fn)
out_fn = "outdir/" + distro + "_" + arch + ".yaml"
write(out_fn, output)
This would produce the cartesian product of files in distros/
and files in arches/
; with one merged output file for each pairing. This syntax is not limited to 2 dimensions, but my current use case is 2-dimensional.
One possible aviator.yml
syntax might look like this:
spruce:
- base: resources.yaml
for_each:
- in: distros/
- in: arches/
merge:
[...]
I can kinda get functionality similar to this now, by generating a third directory full of pre-merged pairings like "focal_aarch64.yaml", "bullseye_x86_64.yaml", etc, and using for_each
to loop over those. But to accomplish this, I'm doing ugly shell loops in a Makefile... it would be nice if the tool could automate this.
Currently, aviator only supports to write merge results to files. This causes cleanup work if you use temp files.
Its there a reason why some methods used in main.go
are defined in another file call flags.go
?
When trying to generate the binary via the go build
I have the following error:
# command-line-arguments
./main.go:14: undefined: setCli
If I defined the methods from flags.go
into the main.go, then I do not have a problem anymore.
Regards,
Enrique Encalada
Hi,
I tried to build the project in Windows, but got following error. Does the project support windows?
github.com/JulzDiverse/aviator/vendor/code.cloudfoundry.org/commandrunner/linux_command_runner
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:27:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:30:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:39:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:42:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:51:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:54:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)
Sometimes it would useful to be able to render to stdout instead of having to create a file.
I am managing yaml configs with spruce / aviator but also there are some non-yaml static config files that I am copying along. I imagine this applies to some other peoples workflows as well. It would help greatly if I could do things like copy a list of files from place to place. Something like for_each
with its properties combined with to_dir
but without spruce merge
.
https://github.com/JulzDiverse/aviator#the-fly-section-optional
Similar to how the kubectl
executor has a --dry-run
mode, perhaps it would be beneficial to have the fly
executor only do a validate-pipeline
and not actually fly up the pipeline.
Thanks for your time and consideration.
Hey Julz,
The suggestion here is to perform a standard merge, but instead of having to always specify files in the with
section, we can directly write the yaml code we would like to merge.
For example, let's say we have the following aviator file:
spruce:
- base: path/to/base.yml
merge:
- with_obj:
foo:
bar: 42
baz: true
to: result.yml
That would merge the base file with the following:
foo:
bar: 42
baz: true
Why? Sometimes you want to merge something really tiny (2-3 lines) and creating a separate file for that can be an overkill.
Thanks ๐
Hello,
I'm using aviator
(version 1.9.0) to create a complex, or rather lengthy, pipeline.
The pipeline consists of over a dozen resources, and it would be hard to maintain or add to the pipeline if created by hand.
My aviator file is supposed to create an all_resources.yml
, but it's not working as intended.
The file contains the full list of resources with their respective repo-names and repo-urls, and additionally it also contains a job, where all those resources are fetched with get
as input to a compilation task.
I managed to get the part of the resources:
section working and made a really identical part for the plan:
section, but SOMEHOW, the - get
lines overwrite each other, which ends up with only the last resource in the final file.
I'm really at a dead end here, because I can't see anything I'm doing wrong.
Can you help me out here?
This is my aviator.yml
file:
spruce:
# First we generate all resources to use in the pipeline
- base: factory/meta-resource-stub.yml
prune:
- meta
for_each:
in: meta-resources/
except:
- example-resource.yml
regexp: ".*.(yml)"
to_dir: temp/resources/
# Merge all resources into one file
- base: factory/all-resources-stub.yml
merge:
- with_all_in: temp/resources/
regexp: ".*.(yml)"
to: temp/all-resources.yml
And these are the stubs:
meta-resource.stub.yml
---
jobs:
- get: (( grab meta.git.repo-name ))
trigger: (( grab meta.git.trigger ))
params:
depth: 1
resources:
- name: (( grab meta.git.repo-name ))
type: git
source:
uri: (( grab meta.git.http ))
username: bot-tt
password: ((sesource.bot-tt-pw))
branch: (( grab meta.git.branch ))
all-resources-stub.yml
---
jobs:
-
resources:
-
When run, it correctly produces the all-resources.yml
but inside, it failed to add each resource in the jobs
section, instead it overwrites itself, so only the last resource was added.
The resources
section correctly produces ALL generated resources.
I don't know what's wrong, since the two sections are near-identical
And also, I know that the resulting file currently is not correct in terms of concourse syntax, but I reduced it to a minimal example to figure out what's wrong.
I've just upgraded to 1.5.0 and am seeing the following behaviour:
Running aviator
, it gets to this part:
AVIATOR EXECUTE:$ fly --target concourse set-pipeline --pipeline mypipeline --config aviator_pipeline.yml
and prints out the correct diff, but then immediately prints bailing out
rather than prompting me to accept the change. I can work around this by setting non_interactive: true
in aviator.yml
. I couldn't see anything related in the recent releases that suggests I've missed some other bit of config, but am quite happy to pointed at it if there was a breaking change along the way. Thanks!
Would like to see a new release with latest spruce changes
On top of merge
aviator should support:
prune
eval
cherry_pick
This isn't a real issue. Just wanted to say thank you for your work & making this open source. This is great stuff.
Hi Julz,
I just updated to v0.18.0 and my aviator file is breaking when doing the fly
feature. The following config:
fly:
config: /tmp/final-pipeline.yml
name: $PIPELINE
target: $TARGET
vars:
- vars.yml
was working fine under v0.17.0, however when upgrading the version, that same part broke with the following:
YAML Parsing Failed: yaml: unmarshal errors:
line 15: cannot unmarshal !!seq into map[string]string
Have you seen this behaviour?
Aa aviator
uses the spruce
library, there should be an automation (CI) in place to build aviator with newer versions. (Discuss automation details)
Currently, aviator only supports env variables in file paths. They should be supported at any feasible place in the aviator.yml
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.