aws / aws-encryption-sdk-javascript Goto Github PK
View Code? Open in Web Editor NEWAWS Encryption SDK for Javascript and Node.js
License: Apache License 2.0
AWS Encryption SDK for Javascript and Node.js
License: Apache License 2.0
Related to #44, no human should ever need to write code that creates default CMMs.
I do not remember if this is the case yet, so am making this issue to track this requirement.
I get the following error
Error: unencryptedDataKey has not been set
at Object.needs (/var/task/node_modules/@aws-crypto/material-management/build/main/needs.js:29:15)
at NodeDecryptionMaterial.getUnencryptedDataKey (/var/task/node_modules/@aws-crypto/material-management/build/main/cryptographic_material.js:180:17)
at NodeDefaultCryptographicMaterialsManager.decryptMaterials (/var/task/node_modules/@aws-crypto/material-management-node/build/main/node_cryptographic_materials_manager.js:49:46)
at process._tickCallback (internal/process/next_tick.js:68:7)
while running a lambda function to decrypt DB activity streams.
const aws = require("aws-sdk");
const {
decrypt,
RawAesKeyringNode,
RawAesWrappingSuiteIdentifier,
} = require('@aws-crypto/client-node')
console.log('Loading function');
aws.config.logger = console;
exports.handler = async (event, context) => {
const kms = new aws.KMS({ region: "us-west-2" });
try {
const output = await Promise.all(
event.records.map(async (record) => {
const data = Buffer.from(record.databaseActivityEvents, 'base64');
const key = Buffer.from(record.key, 'base64');
const promise = await kms.decrypt({
CiphertextBlob: key,
EncryptionContext: {
"aws:rds:dbc-id": process.env.cluster_id,
}
}).promise();
console.log(typeof promise.Plaintext, promise.Plaintext);
const wrappingSuite = RawAesWrappingSuiteIdentifier.AES256_GCM_IV12_TAG16_NO_PADDING;
const unencryptedMasterKey = new Uint8Array(promise.Plaintext);
console.log(unencryptedMasterKey.byteLength);
console.log(promise.Plaintext)
const keyring = new RawAesKeyringNode({
keyName: "aes-name",
keyNamespace: "aes-namespace",
wrappingSuite: wrappingSuite,
unencryptedMasterKey: unencryptedMasterKey,
});
const d = await decrypt(keyring, record.databaseActivityEvents, {encoding: 'base64'});
console.log(d);
})
);
console.log(`Processing completed. Successful records ${output.length}.`);
} catch (err) {
console.log(err);
}
};
with Test Data
{
"invocationId": "invocationIdExample",
"deliveryStreamArn": "arn:aws:kinesis:EXAMPLE",
"region": "us-west-2",
"records": [
{
"type": "DatabaseActivityMonitoringRecords",
"version": "1.0",
"databaseActivityEvents": "AYADeNXig1LTUwpx7uXCSRBTBuUAXwABABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREFqSVhETGpKajJ5djh2TFV0RHNuMFM1eDQ5emIvZy9CN1pmUSs4cUtGenJudXJwT3VnSlR0Nlhwc1p4RnZXL0hxUT09AAEAAkJDABtEYXRhS2V5AAAAgAAAAAw70KtkoLqQkIBTM7IAMMOHE9BBr0oWz9kbwm6gadhx4cL0BnBmyxZxwr+dNzYIQN8mF5MwNB2pTwvYc7eJqQIAAAAADAAAEAAAAAAAAAAAAAAAAAAn5FJ3KVLaFhkoQwqhk73u/////wAAAAEAAAAAAAAAAAAAAAEAAAJxBlzFL/29OBU67dwDfo6/iX3XxnwQsGHx65Ts8oNAHrjwZJ9Wl/nnQ2KaY/OdCq0Tv8+u9TUnTlRdWcsXEXZ2PAP9/ejJYsdlIW0kda/jeNfb2PLkj7vzvC7KzYpY8A9BQhc8a8bm5NNeTgh/ARjkUA7pdeG6AA4DXc5w8yl6aN42o+l3c1NNX2DedT+OF9xc6fs+dqkUEzdoHK6mShuaPWdSFwzLKtGfHC5z6bqpe5mI313pccO1WSEzVioFC0g5G7mQ2f2oMuh86BjEAxT9bkU8VmtxVstaDYcXvjvu4S91Lj3UGx1dHnkA3n5P1SSvy99N1xHZxz+sAndAZ9IlcC0+EnBmWppeLwqjRJSm0qxENOkqDF3AFCkiE70HVxmXyLg70+njWXEmHhlyzBCsw2RzeGfiqXuBV12EGWud3kPKHZ3+3UXkkmvHrt/01G8MdCgvhYawsP3K4MqoNt5iZcXtsgCx+6XouIaP+6qIKVFRPVW6T6ue5pEJzYU3AG0tZ8s0/F/I44bUXRKXeEIhfcJGiDdix/fOTVB5iy946RkaCWBxam7AAmC3JjvM5uVe8ZQ5ZjjrPLD+42n1/ny+rRLtY83AfmGCAF8Nv/yj+NQZE7at91gRNYeBjGQk8sX8JchTWwQNYrsgEMWOEZ9B34q6Hqb4im4lm32oJFpwecgTrWdHQ5afSt4A9u4y5BAJXtXqdpBkzP1Ail4Fqit+NYDxdiwuVM37a6baeslaJppzDOH9r40kxiDVDMgFTl/PgXSpL7RbQhWcIA0vcke2DHiO2kBWmHpphK7jLSpprFuVP/WJagSJbJ+ZGvI406UBYacZqWvAyfknVnGttJucETsAZzBlAjEArAzQjkTRcSAUdn0pXvR3D3HhtxGzLvkR48Frso4GzDlXkUIgGHgV7kS0AksB0OdxAjArbCAXaH1CXcMkt1OKlMXZfyeLwJn10fVJ/zxbrxl3b6aaGIR4EI4yTS6T2l8T82o=",
"key": "AQIDAHj5sC4V75fw9OgpNzg8eJz30SjZJKlkaeCghpgU0ZZpcwGyagVf0Vv0OdZEh9ge6wKPAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMIZT3HfWUMjZXZFp0AgEQgDtqf5tHswwRHSQNqlkXMuoVe2N+zfnJVJ0njS2es8vDqGm54lDCbUVMAIkaSZAx62ygv0IFD8UpExo3og=="
}
]
}
I followed https://github.com/awslabs/aws-encryption-sdk-javascript/blob/master/modules/example-node/src/aes_simple.ts and https://docs.amazonaws.cn/en_us/AmazonRDS/latest/AuroraUserGuide/DBActivityStreams.html#DBActivityStreams.CodeExample
Not sure if this is a bug or I did something wrong. Any help would be much appreciated
If the plaintext given to encrypt
is larger than the frame size,
encrypt
is not breaking up the plaintext into frames properly.
When a keyring or sub-keyring fails an operation, what side effect do we want?
Log a record of what happened? Do nothing? Something else?
This affects all keyring implementations.
We need to provide a control on decrypt that will allow the caller to specify a maximum body component size that they are willing to accept.
This value specifies the largest frame size or non-framed body size that they are willing to accept. If in processing the header and/or body header we find that the message contains a frame size or non-framed body larger than the specified size, we will stop and error out.
This is to allow the caller to protect themselves against resource DoS attacks.
@mattsb42-aws
mattsb42-aws 3 days ago
Not sure where the best place for this comment is (probably not this PR), but I am going to assert that returning the trace to the caller as this assembled binary representation is the wrong user experience. If a user has to do post-processing to understand the contents of this structure, that means we did the wrong thing.An alternative might be to return the trace effectively as black box, then provide functions to call to determine whether that trace contains a certain flag.
Writing this down, I do realize that this function already exists for the value as written in the form of the bitwise AND operator.
I think the question we need to ask ourselves is: what is a language-idiomatic way of looking for a value in a container? If it is idiomatic to pass around a binary blob an perform bitwise AND and XOR operations, then ok; if it is not, then we should provide an idiomatic interface to that underlying definition.
@seebees
seebees 6 hours ago AuthorThis conversation should be in the material-management PR, but as you say ;)
I am not adverse to wrapping the raw object with some helper functions. From the spec conversation: I think it would be best to pin down what traces are and how they are used. What I have implemented here is minimally functional because I'm not clear on how people will use them.
In JS binary operations are not insane, but I would argue that they are never intuitive.
@mattsb42-aws
mattsb42-aws 2 minutes agoI'm fine to move this to an issue and treat it as an UX enhancement.
Either:
In the process of requiring encryption context there are still many references to context
. All these words would converge towards encryptionContext
.
It is helpful to have the codebase be consistent and this kind of change can be especially tolerated in this beta period.
All UTF8 encoding/decoding failures should be normalized to our external-facing errors.
It would be great if we could do releases without disabling branch protection.
The blocker for this is whether or not we can push tags with branch protection turned on.
I think we can, because we can make them through the web UI, but I have not tested this.
Change to:
ciphertext
plaintext
// browser-decrypt
{ messageHeader, clearMessage }
// node-decrypt
{ plaintext, messageHeader ?
// node-encrypt
{ ciphertext, messageHeader }
// browser-encrypt
{ messageHeader, cipherMessage }
Ditto on decrypt and parsing.
Make sure that integration-browser can be run locally to test compatibility.
We need to check the other implementations and write down the result, but I am 80% certain that if the plaintext is equal to the frame length, the resulting ciphertext should have a single frame that is a final frame.
If we determine that this is correct, we need to come back to encrypt-browser
and change the frame size to plaintextLength
rather than plaintextLength + 1
.
Hi,
I've been looking into the source code and issues but could not find a way to trace the KMS requests made by client-node
using AWS X-Ray.
Could you please confirm if that's possible?
Thank you.
The four algorithm combinations we support for the raw RSA keyring are:
There is a feature gap between the node and browser keyrings. We need to either close or document this gap.
The fallback section of the readme for the client-browser module lists the features of the SDK that are not supported by certain browsers. However, while it mentions msrcrypto
as an example of a fallback library, it doesn't recommend any fallback strategies or alternatives.
Also, the fallback section points to @aws-crypto/web-crypto-backend, but that file has very little information, and just sends readers back to the readme. Loops like this are very frustrating to users.
NodeCryptographicMaterialsManager
and WebCryptoCryptographicMaterialsManager
are not implementations of the root behavior that all CMMs need to do, they are explicitly implementations of the specific thing called the "default CMM". They should be named and structured accordingly.
After preview.0
, let's see if we can find a checker that will analyze package.json
files to look for issues.
Importing the client-node imports DOM types, this defeats the purpose of separating the browser and node clients.
Before release, or as a fast-follow, we need an example in JavaScript that is analogous to the ones in Java and Python.
https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/sample-cache-example.html
Node11 is not LTS and isn't even supported by AWS Lambda.
IMO, the encryption SDK should support NodeJS10 and maybe NodeJS8.
FWIW, these are the two versions which are currently supported by lambda.
There are a lot of points in the codebase that accept an optional encryption context and default to an empty object if so. We should only do that default value assignment once, at the top-level user interface. Everywhere else, we should require that it is actually provided.
We have a lot of modules in this meta-repo and each module has some important boilerplate including but not limited to things like license, notice, etc.
We should have something in our CI that verifies that these are all consistent across all modules and the top-level repo.
A lot of the copyright notices just state 2018. We need to update those to the correct statement.
Browsers do not support 192-bit AES keys at this time.
Just in case this ever changes, we should add a test to our browser test suite to verify this and error if this ever changes.
Node.js v12.9 now supports oaepHash
https://nodejs.org/api/crypto.html#crypto_crypto_publicencrypt_key_buffer
Both of these need to be preconditions in serialize
at a minimum
@mattsb42-aws
mattsb42-aws 3 days agoThat doesn't mean they'll never change. What happens if/when the SDK types change and we don't?
If having a common dependency is intractable, can we put something in our CI to test and make sure that the types are continuing to match?
@seebees
seebees 6 hours ago AuthorIf the types change and we don't nothing will happen to the run time. (In TypeScript all type information is removed at compile time).
The worst that will happen is that people who are doing development on new KMS Keyrings will not have as pleasant an experience until the types are updated.
However. This problem is not unique to me. Anyone who wants to consume the v3 sdk in this way will have a similar problem and if/as this happens it will be the JS SDK team that will have the burden to fix it.
Finally, breaking up the SDK in this way is explicitly done to avoid common dependancies. So while I could just include them, both the JS SDK team and myself have gone to great lengths to avoid this. I think it would be better to drive to a world where we don't need to.
(A CI style solution is in principle possible, but annoying as I changed the types so that they are independent of other things)
@mattsb42-aws
mattsb42-aws 2 minutes agoThe worst that will happen is that people who are doing development on new KMS Keyrings will not have as pleasant an experience until the types are updated.
My concern is more that I'm not seeing any way that we would know if the types are updated.
That said, this is also something that I am fine moving into an enhancement feature.
The current behavior, as least for the webcrypto backend[1], is to only attempt a KDF if suite.kdf === 'HKDF'
and suite.kdfHash
is set. Otherwise, it treats this as a non-KDF suite.
The correct behavior is:
if (!kdf && !kdfHash) {
// This is a non-KDF suite.
} else if (kdf === 'HKDF' && kdfHash) {
// This is a valid HKDF suite. Do that.
} else {
// This is an invalid or unsupported suite. Throw an appropriate error.
}
To avoid issues with accidentally creating invalid/incomplete package artifacts, let's add a CI stage that spins up a Verdaccio server, publishes all packages to that server, then goes into a temp directory, installs everything from the Verdaccio server, and runs all integration tests.
AFAIK, the sdkv2 + v3 compatibility in https://github.com/awslabs/aws-encryption-sdk-javascript/blob/master/modules/kms-keyring/src/kms_types.ts#L28-L34 is not working as expected. It requires the user have both SDKV2 and V3 installed.
Importing the client-node imports the the sdk v3 kms node and kms browser clients. This defeats point of separating the browser and node clients
I should support both versions. And I can use import types (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-9.html) to simplify the dependancies.
version 0.1.0-preview.1
From cursory investigation of Node Crypo module native code implementation, Node makes use of smart pointers to zero memory (for symmetric keys) or call OpenSSL helpers (pkey_free()) to clear private key memory contents when keys are no longer needed.
We use instanceOf to insure some security properties of Cryptographic Materials, and Encrypted Data Keys. However, under some conditions, versions of the dependent packages may result in copies of these files being installed (local node_modules folders) for the packages that have differing versions.
This means that these instanceOf checks may fail. Because while the code is the same, the actual instances are not.
What is the best direction?
How pervasive can this be?
Currently, the frame length in encrypt-browser
is determined by the plaintext size. We should support arbitrary frame lengths.
Not a launch blocker.
For the other Encryption SDK's to use for compatibility testing.
DER encoding is always signed.
Browsers use 'raw' encoding.
Where the r and s values are just concatenated.
If the first bit of r or s is 1,
an additional 0 byte should be added.
The current implementation of raw2der
does not do this.
Currently if no configured keyring is found to unwrap any of the encrypted data keys,
the CMM will throw an unencryptedDataKey has not been set
error.
While this error is accurate, it is unhelpful.
Another HMAC-based Key Derivation Function for node.js.
Link to HKDF or KDF in Wikipedia
The function is very simple, but having a controlled, reviewed, and blessed version is valuable.
What does it mean for a function to be controlled? Also substitute a synonym (approved?) for blessed.
As a crypto primitive
Is this function a cryptographic primitive?
this is helpful to have, but the nodejs crypto module only exposes what openssl supports. Since it is so simple and derivable, there are no plans to include this in core.
If we're not using it, why do we have it in a module?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.