homeport/dyff

/ˈdʏf/ - diff tool for YAML files, and sometimes JSON

δyƒƒ /ˈdʏf/

Description

A diff tool for YAML files, and sometimes JSON.

dyff is inspired by the way the old BOSH v1 deployment output reported changes from one version to another by only showing the parts of a YAML file that change.

Each difference is referenced by its location in the YAML document by using either the Spruce dot-style syntax (some.path.in.the.file) or go-patch path syntax (/some/name=path/in/the/id=file). The output report aims to be as compact as possible to give a clear and simple overview of the change.

Similar to the standard diff tool, it follows the principle of describing the change by going from the from input file to the target to input file.

Input files can be local files (filesystem path), remote files (URI), or the standard input stream (using -).

All orders of keys in hashes are preserved during processing and output to the terminal, most notably in the sub-commands to convert YAML to JSON and vice versa.

Use cases and examples

• Show differences between the live configuration of Kubernetes resources and what would be applied (kubectl version >= v1.20.0):

  # Setup
  export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"
  
  # Usage
  kubectl diff [...]

  

  The --set-exit-code flag is required so that the dyff exit code matches kubectl expectations. An exit code 0 refers to no differences, 1 in case differences are detected. Other exit codes are treated as program issues.

  Note: Versions of kubectl older than v1.20.0 did not split the environment variable into field, therefore you cannot use command arguments. In this case, you need to wrap the dyff command with its argument into a helper shell script and use this instead.

• Show the differences between two versions of cf-deployment YAMLs:

  dyff between \
    https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.10.0/cf-deployment.yml \
    https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.20.0/cf-deployment.yml

  

• Embed dyff into Git for better understandable differences

  # Setup...
  git config --local diff.dyff.command 'dyff_between() { dyff --color on between --omit-header "$2" "$5"; }; dyff_between'
  echo '*.yml diff=dyff' >> .gitattributes
  
  # And have fun, e.g.:
  git log --ext-diff -u
  git show --ext-diff HEAD

  

• Convert a JSON stream to YAML

  sometool --json | jq --raw-output '.data' | dyff yaml -

• Sometimes you end up with YAML or JSON files, where the order of the keys in maps was sorted alphabetically. With dyff you can restructure keys in maps to a more human appealing order:

  sometool --export --json | dyff yaml --restructure -

  Or, rewrite a file in place with the restructured order of keys.

  dyff yaml --restructure --in-place somefile.yml

• Just print a YAML (or JSON) file to the terminal to look at it. By default, dyff will use a neat output schema which includes different colors and indent helper lines to improve readability. The colors are roughly based on the default Atom schema and work best on dark terminal backgrounds. The neat output is disabled if the output of dyff is redirected into a pipe, or you can disable it explicitly using the --plain flag.

  dyff yaml somefile.yml

• Convert a YAML file to JSON and vice versa:

  dyff json https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.19.0/cf-deployment.yml

  The dyff sub-command (yaml, or json) defines the output format, the tool automatically detects the input format itself.

  dyff yaml https://raw.githubusercontent.com/homeport/dyff/main/assets/bosh-yaml/manifest.json

Installation

Homebrew

The homeport/tap has macOS and GNU/Linux pre-built binaries available:

brew install homeport/tap/dyff

MacPorts

On macOS, dyff is also available via MacPorts:

sudo port install dyff

Snap

It is available in the snapcraft store in the Productivity section.

snap install dyff

Please note: Since dyff needs to use strict confinement due to otherwise manual clearance requirements, there are some limits to its usage. Most notably, users reported that in strict confinement reading files from the temporary directory does not work. This makes it impossible to use it in the kubectl diff use case. Consider using brew, or pre-built binaries instead.

Pre-built binaries in GitHub

Prebuilt binaries can be downloaded from the GitHub Releases section.

Curl To Shell Convenience Script

There is a convenience script to download the latest release for Linux or macOS if you want to need it simple (you need curl and jq installed on your machine):

curl --silent --location https://git.io/JYfAY | bash

Build from Source

Starting with Go 1.17, you can install dyff from source using go install:

go install github.com/homeport/dyff/cmd/dyff@latest

Please note: This will install dyff based on the latest available code base. Even though the goal is that the latest commit on the main branch should always be a stable and usable version, this is not the recommended way to install and use dyff. If you find an issue with this version, please make sure to note the commit SHA or date in the GitHub issue to indcate that it is not based on a released version. The version output will show dyff version (development) for go install based builds.

Contributing

We are happy to have other people contributing to the project. If you decide to do that, here's how to:

• get Go (dyff requires Go version 1.20 or greater)
• fork the project
• create a new branch
• make your changes
• open a PR.

Git commit messages should be meaningful and follow the rules nicely written down by Chris Beams:

The seven rules of a great Git commit message

1. Separate subject from body with a blank line
2. Limit the subject line to 50 characters
3. Capitalize the subject line
4. Do not end the subject line with a period
5. Use the imperative mood in the subject line
6. Wrap the body at 72 characters
7. Use the body to explain what and why vs. how

Running test cases and binaries generation

Run test cases:

ginkgo run ./...

Create binaries:

goreleaser build --clean --snapshot

License

Licensed under MIT License

dyff's Issues

Consolidate errors in one source file

Prepare all possible generic errors in one error source file and only reference the prepared error objects.

Replace terminal code with gonvenience

Replace all terminal related functions with the ones provided by gonvenience.

Can't detect duplicated list entries

dyff cannot detect duplicated list entries in two files, for example

{ "keys": [ "value1", "value2" ] }

{ "keys": [ "value1", "value2", "value1" ] }

Running dyff on these files returns no difference

Support comparing files with different numbers of documents

Have a look at YAML files to compare where each YAML file contains a different number of documents in it. This might include a change in the way paths are displayed.

Multi objects yaml separators missing after restructure

I write all resources that come under a namespace in single yaml file with "---" separators.
After using 'dyff yaml --restructure " command, noticed these multi object separators are removed in final yaml.
It will be good to preserve these separators.

[support] How to use `--filter` with `kubectl diff`?

I'm trying to only see diffs in the specs using #160 (similar to the original usecase #157 - trying to exclude metadata because it has server generated annotations I want to ignore)
the following still reports diffs in metadata -

export KUBECTL_EXTERNAL_DIFF='dyff between --filter=spec --color=on --omit-header --set-exit-code'`
kustomize build . | kubectl diff -f -

I've tried variations with`--filter=/spec' and such, but haven't got the desired result, what am I missing?

Add role-manifest to known restructure orders

The role-manifest used by https://github.com/suse/scf/ shares a similar style to BOSH deployment manifests. The restructure code should try to sort them in a better way.

Option to filter results

I'm using dyff for comparing k8s manifests which can be quite huge. So I think it'll be useful if we can filter out results using some pattern like below.

dyff between file1.json file2.json --filter deployment.spec.template.spec.containers.env

PS: Great job with the visualisation 🥇

error with kubectl diff compatibility

seems like kubectl is expecting the external diff variable to be just the executable name? is there a workaround? thanks

% export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"
% kubectl diff -f foo.yaml
error: failed to run "dyff between --omit-header --set-exit-code": executable file not found in $PATH

% kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-gke.4900", GitCommit:"2812f9fb0003709fc44fc34166701b377020f1c9", GitTreeState:"clean", BuildDate:"2021-03-19T09:19:27Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Compatiblity with KUBECTL_EXTERNAL_DIFF

First of all, thank you for a great tool!

I see recently there were a lot of changes to make an experience with k8s more straightforward. But I think it could be very useful to have a flag to enable compatibility with KUBECTL_EXTERNAL_DIFF which allows to kubectl diff use any tool specified in this env variable.

To support it the behaviour should be changed slightly:

• Exit codes should be:
  • 0 No differences were found.
  • 1 Differences were found.
  • >1 Kubectl or diff failed with an error.
• Directories comparison should be supported. From my observation, kubectl calls diff tool on folders too.

Does it makes sense to make a dyff compatible with kubectl diff?

Comparing two array of arrays with ignored order

Comparing two array of arrays, like

[1, [2, 3]]

and

[1, [3, 2]]

with option --ignore-order-changes return one difference and I believe it shouldn't because order of sub array should not matter.

Do not check order changes when list entries are not unique

For example, run

dyff between \
  https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v8.1.0/cf-deployment.yml \
  https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v9.0.0/cf-deployment.yml

to see that the order change in addons.loggregator_agent.jobs.loggr-expvar-forwarder.properties.counters is not helpful. Although the name key is used, the identifying key is actually addr in this use case.

Two suggestions:

• Disable order change detection for list, where no unique entries can be found
• Improve detection for identifiable keys by checking if there are other keys other than the default ones that would make list comparable.

arm64 support

I have some issue installing on macOS with m1:

$ curl --silent --location https://git.io/JYfAY | sudo bash
Unsupported operating system darwin or machine type arm64: Please check https://github.com/homeport/dyff/releases manually.

arm64 is supported in goreleaser goreleaser/goreleaser#1956 with go 1.16 version

dyff can't recognize fine grained differences for list items if len(list)>1

Consider following example

- hosts: localhost
  gather_facts: yes
  vars:
    - work_dir: /tmp
    - env: stage

Let's delete just env key: sed '/env:/d' old.yml > new.yml. In that case dyff shows nice and concise difference:

$ dyff between *yml
     _        __  __
   _| |_   _ / _|/ _|  between /tmp/test_dyff/cleaned/new.yml
 / _' | | | | |_| |_       and /tmp/test_dyff/cleaned/old.yml
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

0.vars
  + one list entry added:
    - env: stage

But adding just one more toplevel list item like following:

- hosts: localhost
  gather_facts: yes
  vars:
    - work_dir: /tmp
    - env: stage

- gather_facts: no

Breaks difference detection:

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/test_dyff/cleaned/new.yml
 / _' | | | | |_| |_       and /tmp/test_dyff/cleaned/old.yml
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

(root level)
- one list entry removed:   + one list entry added:
  - hosts: localhost          - hosts: localhost
  │ gather_facts: yes         │ gather_facts: yes
  │ vars:                     │ vars:
  │ - work_dir: /tmp          │ - work_dir: /tmp
                              │ - env: stage

This does not seem right.

Write use cases section in README

Update README with a use cases section to cover:

• compare two YAML files
• compare two YAML files, where in one file the root level is changed
• pretty print a YAML
• convert JSON to YAML and restructure order in keys
• Spruce merge a file and pipe the input into dyff for comparison

dyff between -o headless

dyff outputs a nice header on between command, but for post process it could be nice to removed it.
With files
fom.yaml

a: 1
b: 2
c: 3

to.yaml

a: 1
b: 22
c: 3

The command

dyff -o headless from.yaml to.yaml

Will outputs

b
  ± value change
    - 2
    - 22

Add a 4-bit colors fallback option

Add an option to translate 24-bit colors used by dyff to the nearest 4-bit ANSI alternative.

The indentation of multiple lines modification appears to be out of alignment

When output of modified value is multiple lines, It looks like there is not enough indentation after the second line.

E.g. Consider the following two YAML.

1.yaml:

foo:
  a: 1
  b: 2
bar:
  c: 3
  d: 4

2.yaml:

foo:
  - 1
  - 2
bar:

Currently, dyff between 1.yaml 2.yaml outputs bellow:

foo
  ± type change from map to list
    - a: 1
  b: 2
    + - 1
  - 2

bar
  ± type change from map to <nil>
    - c: 3
  d: 4
    + <nil>

I think the following is more appropriate:

foo
  ± type change from map to list
    - a: 1
      b: 2
    + - 1
      - 2

bar
  ± type change from map to <nil>
    - c: 3
      d: 4
    + <nil>

Preserve underscores in path names through bunt

The bunt package seems to evaluate underscores in the path names and therefore renders weird path names.

Question: How to handle "sensitive" diff ?

Any idea how to handle sensitive values in output ? Its easy to determine sensitive values based on Kind type for Kubernetes for example, but wondering what can be used for docker-compose spec.

Thinking about some argument which can help mask whole environment array/map: ( instead of values, some hash can be outputed )
dyff between docker-compose-old.yaml docker-compose-new.yaml --mask services.*.environment

or mask specific keys:
dyff between docker-compose-old.yaml docker-compose-new.yaml --mask services.*.environment.APP_TOKEN --mask services.*.environment.DATABASE_PASSWORD

with file input:
dyff between ... --mask-from-file denylist.json

or with regular support:
dyff between ... --mask-keys-regex '.*(PASSWORD|TOKEN|CERTIFICATE)'

environments spec in docker-compose can be either map or list, ^ would probably work only for the map variant.

Any ideas?

Thanks

dyff output not parsible by dyff

Dyff strips quotes off from strings with special characters and is thus not able to parse its own output.

Works with aplhanumeric chars:

echo 'foo:
  bar: "2"' | dyff yaml - | dyff yaml -

Fails with special char:

echo 'foo:
  bar: "*"' | dyff yaml - | dyff yaml -

 Error: failed to process input files
│ failed to write output _-_: failed to load input file _-_: unable to parse data from -: yaml: line 3: did not find expected alphabetic or numeric character

Seen with dyff version 1.2.3 in context of comparing kubernetes deployment files.

Add option to use JSON in human output

The human output uses YAML to display complex structures. Add an option to switch to JSON for those outputs.

dyff between can't use a local files

Greetings!

I am trying to compare two yaml files

$ cat /tmp/1.yml
glance_backend_swift: 'no'

$ cat /tmp/2.yml
glance_backend_swift: 'yes'

To do this, I use the between command from the README file

dyff between \
  https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.10.0/cf-deployment.yml \
  https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.20.0/cf-deployment.yml

But I trying use a files from file system.

The problem is that the application tries to use the HTTP format for these files and it fails.

$ dyff between /tmp/1.yml /tmp/2.yml
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
╭ Error: failed to load input files
│ unable to load data from /tmp/1.yml: Get "/tmp/1.yml": unsupported protocol scheme ""

I tried using the file schema and choosing a relative path - but it got nowhere

dyff between file:///tmp/1.yml file:///tmp/2.yml
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
╭ Error: failed to load input files
│ unable to load data from file:///tmp/1.yml: Get "file:///tmp/1.yml": unsupported protocol scheme "file"
$ cd /tmp/
$ dyff between 1.yml 2.yml
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
╭ Error: failed to load input files
│ unable to load data from 1.yml: unable to get any content using location 1.yml: it is not a file or usable URI

Is this a feature or a bug? How do I compare two files offline?

Thanks for your app and for helping.

Version:

$ snap info dyff
name:      dyff                                                                                                                                                              
summary:   YAML diff tool                                                                                                                                                    
publisher: Matthias Diester (heavywombat)                                                                                                                                    
store-url: https://snapcraft.io/dyff                                                                                                                                         
license:   MIT                                                                                                                                                               
description: |                                                                                                                                                               
  δyƒƒ /ˈdʏf/ - A diff tool for YAML files, and sometimes JSON                                                                                                               
                                                                                                                                                                             
  `dyff` is inspired by the way the old BOSH v1 deployment output reported changes from one version                                                                          
  to another by only showing the parts of a YAML file that change.                                                                                                           
                                                                                                                                                                             
  Each difference is referenced by its location in the YAML document by using either the Spruce or                                                                           
  go-patch path syntax. The output report aims to be as compact as possible to give a clear and                                                                              
  simple overview of the change.                                                                                                                                             
                                                                                                                                                                             
  Similar to the standard diff tool, it follows the principle of describing the change by going from                                                                         
  the from input file to the target to input file.                                                                                                                           
                                                                                                                                                                             
  Input files can be local files (filesystem path), remote files (URI), or the standard input stream                                                                         
  (using `-`).                                                                                                                                                               
                                                                                                                                                                             
  All orders of keys in hashes are preserved during processing and output to the terminal, most                                                                              
  notably in the sub-commands to convert YAML to JSON and vice versa.                                                                                                        
commands:                                                                                                                                                                    
  - dyff                                                                                                                                                                     
snap-id:      7nSLjJS5zRHpxmCmleuvqsfKAmgbwgbM                                                                                                                               
tracking:     latest/stable                                                                                                                                                  
refresh-date: 5 days ago, at 15:41 MSK                                                                                                                                       
channels:                                                                                                                                                                    
  latest/stable:    1.4.3 2021-06-15 (12) 2MB -                                                                                                                              
  latest/candidate: ↑                                                                                                                                                        
  latest/beta:      ↑                                                                                                                                                        
  latest/edge:      ↑                                                                                                                                                        
installed:          1.4.3            (12) 2MB - 

Add `--concourse` flag

Add an option to compare Concourse pipeline files, which can have a relative loose definition.

comparing lists of kubernetes objects -> how to influence name/identity detection?

I'm not seeing how to hint this library to appropriately care two documents that look like:

a.yaml:

- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      meta.helm.sh/release-name: foobar
    name: foobar
  spec: {} 
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      meta.helm.sh/release-name: service-2
    name: service-2
  spec: {}

b.yaml:

- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      meta.helm.sh/release-name: foobar
    name: foobar
  spec: {}
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      meta.helm.sh/release-name: service-2
    name: service-2
  spec:
    foo: bar

The only difference is in the spec for the second item.

The output for this list is:

(root level)
- one list entry removed:
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: service-2
      annotations:
        meta.helm.sh/release-name: service-2
    spec: {}

+ one list entry added:
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: service-2
      annotations:
        meta.helm.sh/release-name: service-2
    spec:
      foo: bar

Where I would hope/expect it to be:

service-2.spec
  + one map entry added:
    foo: bar

or

deployment/service-2.spec
  + one map entry added:
    foo: bar

drop "wrapper" objects / arrays

Consider the following:

---
a: foo
---
a: bar

{
  "items": [
    {"a": "foo"},
    {"a": "bar"}
  ]
}

It would be super handy to have some flag to "drop" or inspect the items array values against the yaml.

A real life use-case is comparing Kubernetes List (in json) against a yaml file with multiple objects (separated by ---). In this case the json has:

{
   "apiVersion": "v1",
   "items": [
     { ... }
   ],
   "kind": "List"
}

Format specifiers in YAML files sometimes seem to get processed

Kind of hard to explain, but it seems like dyff is somehow treating its diffs as format strings, or perhaps template strings.

I'm using v1.1.3, downloaded from the github releases page.

As a fairly minimal reproduction, with two files:

# example1.yml
example_one: "%{one}"
example_two: "two"

# example2.yml
example_one: "one"
example_two: "%{two}"

$ dyff between example.yml example-2.yml 
     _        __  __
   _| |_   _ / _|/ _|  between /home/telyn/dyff-issue-example/example.yml
 / _' | | | | |_| |_       and /home/telyn/dyff-issue-example/example-2.yml
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned two differences
        |___/

example_one
  ± value change
    - %!!(MISSING){(MISSING)one}
    + one

example_two
  ± value change
    - two
    + %!!(MISSING){(MISSING)two}

tz-laptop ~/dyff-issue-example ‹master*› » dyff version
dyff version v1.1.3

In the application I discovered this on, sometimes the result in the value change section would show %(MISSING){one} instead of %!!(MISSING){(MISSING)one}.
Weirdly, it seems that setting example_two to %{two}%{two}" in example-2.yml causes that line not to get treated this way - and is output correctly as %{two}%{two}

Add debug code to parse path functions

Add debug code to find the issue for go run cmd/dyff/main.go between https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.10.0/cf-deployment.yml https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.14.0/cf-deployment.yml --debug --chroot instance_groups --use-go-patch-style which works fine, however the human readable input location fails to parse the tree. This might be due to the fact that the root of the document was already changed.

Add check for $COLORTERM

Add an option to check for $COLORTERM to see whether it is set to truecolor before assuming true color support is available in the terminal. See this nice gist for more details: https://gist.github.com/XVilka/8346728

Replace YAML fork library

Currently, the fork is only used because the highlighting of YAML keys. Ideally, find an alternative solution without a fork to have the same effect. Maintaining the fork creates extra effort that could be avoided.

So far the investigations into the Go YAML library and options to overload or extend it failed due to the way the library is written. One option would be to copy the JSON style where dyff marshals the document while relying on the YAML library marshal code for individual marshalling.

panic: unknown and therefore unsupported scalar tag !!timestamp

dyff between cft1.yaml cft2.yaml
     _        __  __
   _| |_   _ / _|/ _|  between  xxx./cft1.yaml
 / _' | | | | |_| |_       and xxx./cft2.yaml
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned twelve differences
        |___/

(root level)
- one map entry removed:
  Description: Platform

AWSTemplateFormatVersion
panic: unknown and therefore unsupported scalar tag !!timestamp

goroutine 1 [running]:
github.com/homeport/dyff/pkg/dyff.humanReadableType(0xc000182280, 0xc0003e7680, 0x1e78548)
        /home/travis/gopath/src/github.com/homeport/dyff/pkg/dyff/output_human.go:394 +0x36a
github.com/homeport/dyff/pkg/dyff.(*HumanReport).generateHumanDetailOutputModification(0xc0001e9400, 0xb1, 0xc000182280, 0xc000119180, 0xc0001c1700, 0x104b24c, 0x10, 0x1444140)
        /home/travis/gopath/src/github.com/homeport/dyff/pkg/dyff/output_human.go:204 +0x65
github.com/homeport/dyff/pkg/dyff.(*HumanReport).generateHumanDetailOutput(0xc0001e9400, 0xb1, 0xc000182280, 0xc000119180, 0x0, 0x0, 0x20, 0x0)
        /home/travis/gopath/src/github.com/homeport/dyff/pkg/dyff/output_human.go:141 +0x23c
github.com/homeport/dyff/pkg/dyff.(*HumanReport).generateHumanDiffOutput(0xc0001e9400, 0x1554160, 0xc00001f600, 0x0, 0xc000074270, 0x1, 0x1, 0xc00000cb80, 0x1, 0x1, ...)
        /home/travis/gopath/src/github.com/homeport/dyff/pkg/dyff/output_human.go:112 +0x193
github.com/homeport/dyff/pkg/dyff.(*HumanReport).WriteReport(0xc0001e9400, 0x15549e0, 0xc0000ae008, 0x0, 0x0)
        /home/travis/gopath/src/github.com/homeport/dyff/pkg/dyff/output_human.go:94 +0x1fb
github.com/homeport/dyff/internal/cmd.glob..func1(0x17acb40, 0xc0000a6960, 0x2, 0x2, 0x0, 0x0)
        /home/travis/gopath/src/github.com/homeport/dyff/internal/cmd/between.go:128 +0x453
github.com/spf13/cobra.(*Command).execute(0x17acb40, 0xc0000a6920, 0x2, 0x2, 0x17acb40, 0xc0000a6920)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x17ad080, 0xc0000e3f18, 0x140c953, 0xc000116800)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/travis/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/homeport/dyff/internal/cmd.Execute(...)
        /home/travis/gopath/src/github.com/homeport/dyff/internal/cmd/root.go:105
main.main()
        /home/travis/gopath/src/github.com/homeport/dyff/cmd/dyff/main.go:35 +0x45

Curl convenience installation script results in a 404

Roughly two weeks ago I was able to use the convenience script to install dyff as described on https://github.com/homeport/dyff :

This is the script:
curl --silent --location https://goo.gl/DRXDVN | bash

The shorthand link resolves to https://raw.githubusercontent.com/HeavyWombat/dyff/master/scripts/download-latest.sh which gives me a 404.

bug: extra start/end document markers confuse dyff

Hey! Sweet project.

Consider the following:

$ cat foo.yaml 
---
---
a: foo

$ cat foo.json 
{
  "a": "foo"
}

Running dyff against these doesn't show what I expect as the proper output. To me the objects are the same, even though the yaml has an extra ---.

adam@~$ dyff between foo.json foo.yaml 
      _        __  __
    _| |_   _ / _|/ _|  between /Users/adam/foo.json
  / _' | | | | |_| |_       and /Users/adam/foo.yaml
 | (_| | |_| |  _|  _|
  \__,_|\__, |_| |_|   returned one difference
        |___/

(root level)
- one map entry removed:
  a: foo

Change honnef.co/go/tools/cmd/unused to its GitHub URL

Change the go get of honnef.co/go/tools/cmd/unused to its GitHub mirror.

Check for untested code

Check coverage report to identify if there are dyff package core functions that are no related to error handling, which are currently not tested by any test case.

Write a new set of test cases where (similar to Spruce) the input and the expected results are covered via files and not in code to be able to quickly write new complex tests.

dyff installation failing

When installing dyff I'm seeing the following error.

$ sudo go get github.com/homeport/dyff/cmd/dyff
# github.com/homeport/dyff/pkg/dyff
/root/go/src/github.com/homeport/dyff/pkg/dyff/core.go:713:34: not enough arguments in call to hashstructure.Hash
	have (interface {}, nil)
	want (interface {}, hashstructure.Format, *hashstructure.HashOptions)
/root/go/src/github.com/homeport/dyff/pkg/dyff/core.go:721:34: not enough arguments in call to hashstructure.Hash
	have (string, nil)
	want (interface {}, hashstructure.Format, *hashstructure.HashOptions)

I've tried both on ubuntu (20.04) and Mac (Catalina). This has worked in the past, seems to be a recent issue.

Test Issue

Add generic JSON output style

Add output style JSON which only takes the difference report as-is and prints it

Comparing 57MB files causes out of memory

When I'm trying to compare files with just 57MB in size, dyff uses more then 3GB of RAM before crashing with an "fatal error: runtime: out of memory" error.

57MB isn't that big in size, expecially for a yaml file, so I strongly suggest to optimize memory usage

go install dyff

Hello,

with Go 1.17 https://golang.org/doc/go-get-install-deprecation, we can install dyff with:

$ go install github.com/homeport/dyff/cmd/dyff@latest
go: downloading github.com/gonvenience/neat v1.3.6
go: downloading github.com/gonvenience/wrap v1.1.0
go: downloading github.com/gonvenience/bunt v1.3.2
go: downloading github.com/gonvenience/term v1.0.1
go: downloading github.com/gonvenience/ytbx v1.4.2
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/gonvenience/text v1.0.6
go: downloading github.com/sergi/go-diff v1.2.0
go: downloading github.com/texttheater/golang-levenshtein v1.0.1
go: downloading github.com/mitchellh/go-ps v1.0.0
go: downloading github.com/virtuald/go-ordered-json v0.0.0-20170621173500-b18e6e673d74
go: downloading github.com/mattn/go-ciede2000 v0.0.0-20170301095244-782e8c62fec3
go: downloading golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9

And it works:

$ which dyff
/home/user/go/bin/dyff

$ /home/ejly7374/go/bin/dyff 

δyƒƒ /ˈdʏf/ - a diff tool for YAML files, and sometimes JSON. Also, It
can transform YAML to JSON, and vice versa. The order of keys in hashes
is preserved during the conversion.

Usage:
  dyff [command]

Available Commands:
  between      Compare differences between input files from and to
  help         Help about any command
  json         Converts input documents into JSON format
  last-applied Compare differences between the current state and the one stored in Kubernetes last-applied configuration
  version      Shows the version of this tool
  yaml         Converts input documents into YAML format

Flags:
  -c, --color                        specify color usage: on, off, or auto (default auto)
  -t, --truecolor                    specify true color usage: on, off, or auto (default auto)
  -w, --fixed-width int              disable terminal width detection and use provided fixed value (default -1)
  -k, --preserve-key-order-in-json   use ordered keys during JSON decoding (non standard behavior)
  -h, --help                         help for dyff

Use "dyff [command] --help" for more information about a command.

unfortunately the version seems not correct:

$ dyff version
dyff version (development)

I think that it can be easy to fix it into next version, if you also set the correct value to version before to tag (as you defined it while building with ldflags).

dyff/internal/cmd/version.go

Line 30 in fdc6c43

var version string

ldflags

dyff/.goreleaser.yml

Line 17 in fdc6c43

- -s -w -extldflags "-static" -X github.com/homeport/dyff/internal/cmd.version={{.Version}}

Please add option for ignoring order changes

I tend to use dyff to compare long yaml dumps of implicitly unordered lists. Sometimes order of items might change, for instance human have added new record just in the end of the list and then I compare new version with auto generated dump, in which records are placed in alphabetical order. This shows order change which is actually meaningless and does not affect anything.

I'd like to have a CLI option to disable order changes check.

Customize the report to be html using go templates

I want to create a go script for detecting differences between yaml configurations and creating a html table with the results. Example:

deployment-1.yaml	deployment-2.yaml
replicas: 0	replicas: 1

JSON output with unquoted timestamps

Given the following TOML input

[metadata]
  [[metadata.dependencies]]
    deprecation_date = "2021-08-21T00:00:00Z"

  [[metadata.dependency_deprecation_dates]]
    date = 2021-08-21T13:37:00Z

the actual JSON output is invalid due to missing quotes for the timestamp:

{
  "metadata": {
    "dependencies": [
      {
        "deprecation_date": "2021-08-21T00:00:00Z"
      }
    ],
    "dependency_deprecation_dates": [
      {
        "date": 2021-08-21T13:37:00Z
      }
    ]
  }
}

Correct would be:

{
  "metadata": {
    "dependencies": [
      {
        "deprecation_date": "2021-08-21T00:00:00Z"
      }
    ],
    "dependency_deprecation_dates": [
      {
        "date": "2021-08-21T13:37:00Z"
      }
    ]
  }
}

Support Yaml Anchors and Aliases.

Hi,

This tool is really good and it helped me a lot.
My request is to actually run the diff logic on the "expanded' version after expanding all the anchors and aliases.

Thx
Gil

Add `--in-place` flag to override input file

Add --in-place flag to override input files for yaml and json sub-commands to allow use cases where someone wants to use --restructure on an existing file to persistent the reordering.

Instead of ...

dyff yaml file.yml --restructure > file.yml.tmp && mv file.yml.tmp file.yml

Use ...

dyff yaml file.yml --restructure --in-place

Enhance `between --filter` option

old.yaml

hello:
  world:
     abc: 1

new.yaml

hello:
  world:
     abc: 2
     def: 1

Requested functionality:
dyff between --filter "hello.*" "old.yaml" "new.yaml"

Expected to show dyff for hello.world.abc and hello.world.def

dyff between --filter "hello.world.def" "old.yaml" "new.yaml"

Expected to show dyff for hello.world.def
Currently hello.world.def diff is only shown on hello.world filter

dyff between --filter "*abc*" "old.yaml" "new.yaml"

Expected to show any dyff that contains a key abc

Support for diffing kubernetes configmap content

Right now, dyff treats configmaps like every other resource, which means that if there's a one-line difference in a hundred-line file, dyff will print all 100 lines of both old and new content.

It'd be nice if dyff detected that a resource was a configmap and diffed the content instead of treating values in the configmap as atomic. (I hope that makes sense.)

Move restructure code into ytbx

Move restructure code from dyff into ytbx.

Add Spruce report output style

Add output style that uses Spruce operators to define the difference

ability to generate JSON Patches (RFC 6902)

Could dyff have a feature for creating json patches for the diffs based on something like jsondiff. Would be really useful for working with things like Kustomize and admission controllers.