How viable would it be to support unencrypted fields?

[getsops]

How viable would it be to support unencrypted fields?

[jvehent]

Hi @Legomaniac, sorry for the late reply, it seems that I skipped the notification for this issue.
I've been thinking about this exact feature for a while now. I think it would be very valuable to have. Implementation is somewhat tricky, we basically have 3 options:

1. create a top level unencrypted key, under which every is in clear
2. allow prepending a suffix such as _clear to keys in the tree, which would force sops to skip that branch entirely (everything underneat it would be in cleartext)
3. support some form of pattern matching that's user configurable to define what should be in clear and/or encrypted. For example, starting sops with --unencrypted "key1|somekey2" would treat the flag value as a regex to leave both "key1" and "somekey2" in cleartext. Similarly, we could have a flag --default-unencrypted paired with --encrypt "key1|somekey2" to leave the whole document in clear except for "key1" and "somekey2". This option is a lot more complex to implement that 1 or 2, but provides more flexibility.

I have a small preference for option 2, but could be convinced that other options are better.

[twolfson]

I am impartial to option (2) as well; it allows similar configuration items to be placed side by side and is obvious to humans (whereas unencryped would require starting from root node always (bad for branches) and the --unencrypted CLI flag seems prone requires 2 steps from humans).

One other option which is YAML specific would be to define a custom YAML tag (e.g. !enc) to use in the value (e.g. !enc "hello-world"). Here's an example from Charlatan which uses !now to load the current time as a value:

https://github.com/uber/charlatan/blob/0.4.6/charlatan/file_format.py#L96

https://github.com/uber/charlatan/blob/0.4.6/charlatan/file_format.py#L40-L50

https://github.com/uber/charlatan/blob/0.4.6/charlatan/tests/data/special_tags.yaml

http://pyyaml.org/wiki/PyYAMLDocumentation#Constructorsrepresentersresolvers

[jvehent]

I'm not to thrilled about the idea of getting into custom yaml parsing. It would be tricky to implement, and it doesn't work with JSON. I think implementing option 2) is the way forward. Since I likely won't have time to implement for several weeks, if someone wants to take a crack at it, I'll be happy to review the patch.

[twolfson]

I'm not claiming this yet but to make sure we are on the same page for (2). For a file with some plaintext keys, would we keep the keys the same upon save or strip the _clear suffix and store it in metadata?

Example with _clear suffix maintained:

On-disk:

hello:
  world_clear: hi
sops:
  kms: ...

While editing via sops:

hello:
  world_clear: hi

Example with _clear suffix stripped:

On-disk:

hello:
  world: hi
sops:
  clear_keys: ['hello.world']
  kms: ...

While editing via sops:

hello:
  world_clear: hi

[jvehent]

I think we should always keep the _clear suffix in the keys directly, not in the metadata, to avoid confusing the user.

[twolfson]

Cool, and are we definitely going with the _clear suffix or should we use something more canonical like _plaintext or _plain?

Also, some people might have an edge case where they need to set a different suffix. Do we want to build a CLI flag and metadata key for that (e.g. --clear-suffix '_unencrypted' and sops.clear_suffix: _unencrpyted) or ignore it for now?

[jvehent]

_unencrypted is probably a more explicit default.
👍 on the cmdline flag to set the --unencrypted-suffix and store it in the metadata.

[twolfson]

Cool, thanks for the quick replies. To reiterate, I haven't yet claimed this issue but I will definitely put it at the top of my queue =) (I will comment if/when I start work on this to prevent duplicate work)

[twolfson]

I started working on this earlier, got a proof of concept working with tests, and currently iterating on CLI integration.

[twolfson]

I have opened up a PR to resolve this issue #47