getsops / sops Goto Github PK
View Code? Open in Web Editor NEWSimple and flexible tool for managing secrets
Home Page: https://getsops.io/
License: Mozilla Public License 2.0
Simple and flexible tool for managing secrets
Home Page: https://getsops.io/
License: Mozilla Public License 2.0
Deploying sops
to instances can be difficult because of python's complex dependencies management. Instances only need to decrypt files and don't need all the fancy editing features sops
provides, so we should write a small Go client that takes an encrypted file and decrypts it.
Can be limited to yaml
and kms
for a first shot.
Text files are currently encrypted as blobs, so there's no benefit to put a requirement on them being actual texts. Instead, we should treat any file that isn't JSON or YAML as binary, which would allow for encryption of any binary file with sops.
When opening a binary file with flag -s
and modifying the sops json data, saving the file fails with the following error:
Traceback (most recent call last):
File "/usr/local/bin/sops", line 9, in <module>
load_entry_point('sops==0.9', 'console_scripts', 'sops')()
File "/home/ulfr/git/svc/sops/sops/__init__.py", line 253, in main
if check_master_keys(tree):
File "/home/ulfr/git/svc/sops/sops/__init__.py", line 476, in check_master_keys
if 'kms' in tree['sops']:
KeyError: u'sops'
The timestamp generated when encrypting a data key currently uses unixtime with nanosecond precision. This is not optimal, we should use RFC3339 timestamps instead.
Add command line flags to add/remove master keys, such as:
--add-kms <arn>
--remove-kms <arn>
--add-pgp <fingerprint>
--remove-pgp <fingerprint>
Do not crash when encrypted yaml files where the content is entirely commented out.
When neither -e, -d or -r are passed, that means we should open an editor and let the user edit the decrypted file
sops
could inform a user that a newer version is available and should be installed via pip
. Not sure if there's a clean API to do that...
It would be useful to support extracting only a subsection of a sops
encrypted file. For example, given the file:
branchA:
keyA: valueA
branchB:
keyB: value B
The following command should extract branchA.keyA
only:
$ sops -d -q 'branchA'->'keyA' file.yaml
or something similar.
When a syntax error is detected in a document in edit mode, sops alerts and invites the user to reenter edit mode to fix it. This is broken, and sops fails to reenter edit mode with error:
$ sops /tmp/testval.yaml
/tmp/testval.yaml doesn't exist, creating it.
please wait while a data encryption key is being generated and stored securely
temp file created at /tmp/tmpRpB1Fw.yaml
Syntax error: while scanning a simple key
in "/tmp/tmpRpB1Fw.yaml", line 12, column 1
could not find expected ':'
in "/tmp/tmpRpB1Fw.yaml", line 13, column 1
Press a key to return into the editor, or ctrl+c to exit without saving.
Traceback (most recent call last):
File "/usr/local/bin/sops", line 9, in <module>
load_entry_point('sops==0.9', 'console_scripts', 'sops')()
File "/home/ulfr/git/svc/sops/sops/__init__.py", line 252, in main
restore_sops=stash['sops'])
File "/home/ulfr/git/svc/sops/sops/__init__.py", line 341, in load_file_into_tree
tree = ruamel.yaml.load(fd, ruamel.yaml.RoundTripLoader)
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/main.py", line 74, in load
return loader.get_single_data()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/constructor.py", line 57, in get_single_data
node = self.get_single_node()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 46, in get_single_node
document = self.compose_document()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 66, in compose_document
node = self.compose_node(None, None)
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 97, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 149, in compose_mapping_node
while not self.check_event(MappingEndEvent):
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/parser_.py", line 119, in check_event
self.current_event = self.state()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/parser_.py", line 491, in parse_block_mapping_key
if self.check_token(KeyToken):
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 1529, in check_token
self.fetch_more_tokens()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 177, in fetch_more_tokens
self.stale_possible_simple_keys()
File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 308, in stale_possible_simple_keys
"could not find expected ':'", self.get_mark())
ruamel.yaml.scanner.ScannerError: while scanning a simple key
in "/tmp/tmpRpB1Fw.yaml", line 12, column 1
could not find expected ':'
in "/tmp/tmpRpB1Fw.yaml", line 13, column 1
We have some tar.gz files that we encrypt and then decrypt with sops. The file was decrypting properly til recent upgrade. Now, however, (after decryption) when de-tarring the file, the contents are correctly unzipped, however tar throws an error complaining the file is terminated incorrectly. We have since switched to using tar.bz2 instead and don't see this issue with that file format.
I have users of our SOPS implementation that would prefer to use subdirectories to determine what encryption keys to use. Our project structure is as follows:
├── engineering
├── ops
├── resources
└── ssl
Right now, we have a .sops.yaml file in each directory and require users to cd to the directory before creating a new file. I get a lot of forgetful users who will create secrets from the top level.
So instead of
cd engineering
sops foo
we get
sops engineering/foo
If I could extend the configuration featureset to include a directory_regex
I can run a regex on the first filename that is checked and see if it matches. If no match, pass through to the filename_regex
as it works today.
creation_rules:
- directory_regex: ^engineering/
kms: '...'
pgp: '...'
- directory_regex: ^ops/
kms: '...'
pgp: '...'
happy to submit a patch if you are interested in accepting
I'm trying to use sops in some automation where the PGP private key I'm using for decryption has a passphrase. The passphrase can't be entered interactively. Is there a way to supply a passphrase through sops? Glancing at the source code it looks like it just shells out to gpg -d
, so this may not be possible right now. It looks like it'd need to explicitly accept a passphrase argument to pass on to GPG. If this isn't possible through sops right now, any ideas for workarounds? Thanks much.
Add an option to reencrypt a file with a new data key, which would allow for key rotation.
test.json - array of hashes (test payload)
[{
"example_key": "exaaasasasasple_value",
"example_array": [
"example_value1",
"example_value2"
],
"example_number": 1234.56789
}]
When using sops for encrypting test.json get the following error.
$ sops test.json
temp file created at /var/folders/k6/660mymls4wbfxxh3j933wp480000gn/T/tmp4bkTf5.json
Traceback (most recent call last):
File "/usr/local/bin/sops", line 11, in <module>
sys.exit(main())
File "/Library/Python/2.7/site-packages/sops/__init__.py", line 292, in main
restore_sops=stash['sops'])
File "/Library/Python/2.7/site-packages/sops/__init__.py", line 411, in load_file_into_tree
tree['sops'] = restore_sops.copy()
TypeError: list indices must be integers, not unicode
sops
should prevent malicious modifications of the content of the file:
I recently upgraded sops and wanted to verify that pip worked properly, as my local python installation has become convoluted.
I found the version is available at the end of the output with --help, but I would like to request the addition of -v or --version to print the version explicitly.
When the --in-place or -i command line flag is passed, sops should encrypt and decrypt in-place, not output the file to stdout.
pynacl provides libsodium implementation in python. We should add support for it and propose it as an alternate encryption method to AES_GCM.
When following the project README.rst
step by step, I got the following output when running sops example.yaml
. Maybe this is expected, but wasn't explicitly stated in the README. I can confirm it successfully opened the file in an editor with the secrets decrypted.
Thanks!
(sops)pmoore@Petes-iMac:~/git/sops master $ sops example.yaml
[warning] skipping kms arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext references a key that either does not exist or you do not have access to.
[warning] skipping kms arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext references a key that either does not exist or you do not have access to.
gpg: encrypted with 1024-bit RSA key, ID 7CD79CC0, created 2015-10-08
"SOPS Functional Tests (https://github.com/mozilla/sops/) <[email protected]>"
temp file created at /var/folders/v9/mll6p_rj5h94dt_m5m8j0f9c0000gn/T/tmpF6qNM1.yaml
file written to example.yaml
I've tried decrypting a file using both sops version 1.10 (latest) and earlier version 1.4, but getting the following traceback. Perhaps recent dependency upgrade broke something?
$ sops -d /tmp/my_config.xml.ENC > config.xml
Traceback (most recent call last):
File "/usr/local/bin//sops", line 9, in
load_entry_point('sops==1.4', 'console_scripts', 'sops')()
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 218, in main
tree = walk_and_decrypt(tree, key, ignoreMac=args.ignore_mac)
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 641, in walk_and_decrypt
branch[k] = decrypt(v, key, aad=caad, stash=nstash, digest=digest)
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 698, in decrypt
default_backend()
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/init.py", line 35, in default_backend
_default_backend = MultiBackend(_available_backends())
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/init.py", line 22, in _available_backends
"cryptography.backends"
AttributeError: 'EntryPoint' object has no attribute 'resolve'
The current implementation of SOPS is functionally based. This is good for programs that rely on transforming data. However, it's beginning to get unwieldy with all of the state passing it is doing (e.g. digest
, iv
, key
, and in #47 unencrypted_suffix
).
I suggest considering a refactor to swap out the core with something more class based so state is always located on self
. I will create a gist to provide examples of what I mean.
no details yet, needs investigation
should add testing in travis
by moving the encrypted/decrypted tempfile to the original path
Via the command line flags --(add|remove)-(kms|pgp).
My $EDITOR variable is currently set to bbedit --wait --resume
. When I try to edit a file I get the following error:
Traceback (most recent call last):
File "/Users/jbuckley/Library/Python/2.7/bin/sops", line 11, in <module>
sys.exit(main())
File "/Users/jbuckley/Library/Python/2.7/lib/python/site-packages/sops/__init__.py", line 298, in main
run_editor(tmppath)
File "/Users/jbuckley/Library/Python/2.7/lib/python/site-packages/sops/__init__.py", line 1282, in run_editor
subprocess.call([editor, path])
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Looking at the documentation for subprocess.call this is occurring because shell=False
by default. I can work around this by creating a shell script that runs the bbedit command with arguments but I wonder if this should be set to shell=True
. It's recommended to not do this with untrusted input so maybe it should actually remain False. Thoughts?
for yaml file foo.yaml
foo:
baz: 'blah'
sops -d foo.yaml --extract '["foo"]["baz"]'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/sops", line 9, in <module>
load_entry_point('sops==1.7', 'console_scripts', 'sops')()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sops/__init__.py", line 241, in main
write_file(tree, path=dest, filetype=otype)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sops/__init__.py", line 1106, in write_file
fd.write(tree.encode('utf-8'))
UnboundLocalError: local variable 'fd' referenced before assignment
This should remove a number of tests for unicode(), str() and bytes().
https://pypi.python.org/pypi/six
Is it possible to add feature where we can only encrypt part of file using _encrypted and rest of the files unencrypted.
same feature _unencrypted
When a file claims to be yaml or json but does not load into a tree as this type, sops
fails to detect it and continues anyway, resulting into a tree failure later on, because the tree is None
.
sops must detect failure to load a yaml or json file and panic accordingly.
@floatingatoll suggested that being able to decrypt a diff would be useful. Something like git diff HEAD~1 | sops -d
.
This isn't trivial, as sops
must be able to parse the diff format and obtain a data key to decrypt the entries. Moreover, if a data key rotation has happened between the diffs, sops
must retrieve the old data key from a previous version of the file.
We could probably recompose each file based on the +
and -
characters, and use that to perform the decryption.
would be great to have ini support - much simpler format
As a newcomer to secret management (used to either an ignored file or committing into memory but in a private repo), I had trouble figuring out how to apply SOPS to my current setup (e.g.would I load into memory at boot? do I store in plaintext on disk?). After enough reasoning, I resolved that:
nodemon
which restarts server upon changes during development, then that can get costly with AWS (could use caching but then we are storing in disk)sops --decrypt
and look at secretsHowever, to prevent all future newcomers from running into the same hurdles, I suggest we add an examples section.
I have thought of 2 configurations that might be nice to showcase:
Catch-all secrets file:
config/secret.enc.yml
) which contains all secrets.enc
suffix (e.g. config/secret.yml
)config/secret.yml
via .gitignore
or similarconfig/secret.yml
and use in respective configurationsHere's an example setup:
https://github.com/twolfson/twolfson.com/blob/3.79.0/config/secret.enc.json
https://github.com/twolfson/twolfson.com/blob/3.79.0/config/static.js#L14
https://github.com/twolfson/twolfson.com/blob/3.79.0/bin/decrypt-config.sh
Secrets in each file:
This setup will likely be more sane once #24 is resolved but each secret would be managed inline in each file. This would be practical if developers want to visually cross-check test vs production values easily.
config.enc
)config.enc
to config
.gitignore
for config
I can reliably reproduce this by creating a JSON kv with an empty value, such as in the example below:
{
"bar": "",
"example_key": "example_value",
"example_array": [
"example_value1",
"example_value2"
],
"example_number": 1234.5678
}
sops -d test.json
prints the following:
{
"bar": "ENC[AES256_GCM,data:,iv:3CNjsdQS/SaFNDup0pbcsfgg41712CPj/Q4b58YT5g0=,tag:Aqg5vJjahkFOjVfjOgWadg==,type:str]",
"example_key": "example_value",
"example_array": [
"example_value1",
"example_value2"
],
"example_number": 1234.5678
}
I browsed __init__.py
but I was unable to find a way to programmatically set the KMS ARN or PGP FP without using an environment variable.
This was the only code snippet I could find that specified the decryption resources, and it appears to only be invoked in main(). I think making direct API calls bypasses this code section entirely.
kms_arns = ""
if 'SOPS_KMS_ARN' in os.environ:
kms_arns = os.environ['SOPS_KMS_ARN']
if args.kmsarn:
kms_arns = args.kmsarn
pgp_fps = ""
if 'SOPS_PGP_FP' in os.environ:
pgp_fps = os.environ['SOPS_PGP_FP']
if args.pgpfp:
pgp_fps = args.pgpfp
Hi. Great stuff here. Putting a PoC together for the team using git as a storage and distribution mechanism. We are currently using https://www.passwordstore.org/ and I'm attracted to sops for its support for AWS KMS.
What are your recommendations for settling merges since they (almost?) always result in a conflict on the mac signature with neither signature going to be the correct one.
I'm comfortable with sops --ignore-mac
and :wq
to re-generate a mac, but this could be a roadblock for adoption since it's additional training to micro-manage the MAC every time there is a merge.
Any recommendations for reducing the scope of this problem? How do you folks handle this issue?
So in our initial toe-dipping investigations we had discovered when running sops on hardware inside our corporate network, it would take > 30 seconds to execute, but from elsewhere it would take < 1 second. It turns out the root cause is due to us not declaring the --no-latest-check
argument when running it, as the method it calls is not aware of common proxy environment variables (https_proxy
and the likes) and the wait is due to proxy timeout, in which case it gives up and just completes the task anyhow.
We could just send a pull request adding proxy support to the check, however I think now is a good time to have a discussion about wether or not the idea of a security product like sops 'phoning home' on every execution by default is a good idea and that it outweighs the risks of out-of-date and potentially vulnerable software being exploited. A very large number of our machines will be automatically executing this code frequently as well as engineers and not necessarily be in an immediate position to upgrade the moment a new release is made available. At the same time we could look at other options - disabling the check everywhere or forking code being the first to mind.
Is this the right default to have?
Following @ametaireau 's comment, we should create a Tree
class, which handles its state internally, so we don't have to expose the tree variable to the user of the object.
tree = Tree() # This loads the tree. In the constructor, do everything you need.
# Then, directly use functions or arguments of the tree. e.g.
tree.key # or tree.get_key() if it takes some time to get the key.
tree.encrypt()
tree.decrypt()
tree.write()
I am very intrigued by this project, and I would like to work it into some of the processes that I oversee.
However, I have some use cases where it would be very convenient to only have some values encrypted, while others remain plaintext, in the same YAML file. For example, I may need to store an encrypted username and password or key for an FTP site, however there may be data in the YAML file that would benefit from remaining unencrypted, such as a custom report date range, or the path to a working directory, etc. If there were a top level "no_encryption" key or something of the sort, I assume this could be accomplished rather trivially.
Is this type of support compatible with the purpose of this project?
Looking forward to your reply.
Thanks,
Legomaniac
We should be able to get KMS ARNs and PGP fingerprints based on the matches on the GPG config file. See the python version for more details
Since we have to allow output formats different from the input ones (i.e., input yaml, output json), it would be easier to assume sops can only encrypt trees, as the Python version does. The Go version currently does not make such assumption and lets Store
s do whatever they want, but by doing this, we'd need each store to create a function to port their data to and from every other store. It's easier to only support trees.
This way, the stores will only have to know how to load the tree, and dump the tree including the sops branch or not. They don't need to take care of encrypting themselves or decrypting.
After a user edits a file, and prior to sops
encrypting it, we should perform a syntax checking step and invite the user to fix any issue there might be with yaml or json syntax.
Currently, sops will just fail if there are syntax errors, and changes will be lost.
When encrypting, we should check each branch to see if it ends with the unencrypted suffix, and if so, leave it unencrypted.
You should be able to override the input and output types via the CLI --input-type and --output-type flags.
Using two different types might be difficult, as you'd have to transfer the data from YAMLStore to JSONStore, for example.
While the benefit of encrypting booleans is clearly close to null, they should be encrypted regardless to follow the same logic as the rest of the file.
When I assume another users privileges I'm unable to run sops -d cat.json
and inspect encrypted data, it looks like sops is unable to get write permissions on /dev/stdout.
I can, however, run sops -d cat.json | cat
to inspect the contents.
Steps to reproduce:
bhourigan@system$ sudo su
root@system# su machineaccount
machineaccount@system$ sops -d cat.json
Traceback (most recent call last):
File "/usr/local/bin/sops", line 9, in <module>
load_entry_point('sops==1.3', 'console_scripts', 'sops')()
File "/usr/local/lib/python2.7/dist-packages/sops/__init__.py", line 226, in main
write_file(tree, path=dest, filetype=otype)
File "/usr/local/lib/python2.7/dist-packages/sops/__init__.py", line 1034, in write_file
fd = open(path, "wb")
IOError: [Errno 13] Permission denied: u'/dev/stdout'
Path permission/ownership chain:
$ ls -ald /dev/stdout
lrwxrwxrwx 1 root root 15 Jan 19 20:10 /dev/stdout -> /proc/self/fd/1
$ ls -ald /proc/self/fd/1
lrwx------ 1 machineaccount machineaccount 64 Jan 22 14:29 /proc/self/fd/1 -> /dev/pts/0
$ ls -ald /dev/pts/0
crw--w---- 1 bhourigan tty 136, 0 Jan 22 14:29 /dev/pts/0
Is this a feature, or a bug?
This could be a known limitation and improper usage on my part, but I ran into an issue while encrypting a json file and redirecting the encrypted version to test.json.enc
.
Creating a new file via sops test.json
works as expected
Encrypting a file in place with sops -i -e test.json
works as expected
Encrypting a file with sops -e test.json > test.json.enc
renders the file unreadable, until test.json.enc is renamed to <something>.json
Steps to reproduce:
bash-3.2$ cat test.json
{
"bar": "baz",
"foo": ""
}
bash-3.2$ sops -e test.json > test.json.enc
please wait while a data encryption key is being generated and stored securely
bash-3.2$ sops -d test.json.enc
bash-3.2$ cat test.json.enc
{
"bar": "ENC[AES256_GCM,data:XBgd,iv:RuJg8xWgkyvsDQb+u8zMzuVsB0OpWyAbsUi9xz8T0Ms=,tag:XTmtCVJ/nzsUr7c7zzbKtA==,type:str]",
"foo": "ENC[AES256_GCM,data:,iv:5HPcYQ9i+ntnEklmLyY7fYAiPDWlGPOyagu238gE7hY=,tag:ZAlB6zZrgz9iXASjrcP5bg==,type:str]",
"sops": {
"lastmodified": "2016-02-03T16:40:59Z",
"attention": "This section contains key material that should only be modified with extra care. See `sops -h`.",
"mac": "ENC[AES256_GCM,data:sPBMhe9F3Bm1kGwO/kQvF295nVHKSaUzq/usH1V3yF3e1G1FTyt2FUx5HuX/nYuu5OqWpWzWoWNDqMYB1Rfyz3ZdWxhph0NT5X7HH7TSSadMtSib5HevgI6BJnUCmlcBuNPX6GWMQHPA3kN/yMGcjJAYD2r6Qd+s/vip9xT0Hxs=,iv:W6dUFubWGsaWLKdfgIWsQKe7heHmT7+Iuzyduu/AUy8=,tag:T9ugdVlJR/22kvAibwR2Qw==,type:str]",
"version": 1.4,
"kms": [
REDACTED
],
"pgp": [
REDACTED
]
}
}
bash-3.2$ sops --help | grep Version
Version 1.4 - See the Readme at github.com/mozilla/sops
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.