adhorn/aws-lambda-chaos-injection

Chaos Injection library for AWS Lambda

aws-lambda-chaos-injection's Issues

Inject faults for lambda/apigateway

how can i use the example.py to inject faults to lambda/apigateway service

inject faults for aws service api calls

sketch of a config (faults, delays) and implementation
https://gist.github.com/kapilt/eb8f9b82473d7fa8295fbf1d10c11c11

ideally we should be able to control which services, rates, and delay ranges. errors are a little harder to configure, since error codes may need to match up to individual api calls, although there's a few generic ones.

Fix order of events

Executing the lambda handler in all cases and only over-writing the error code if a valid parameter result is received means that the function code will be invoked and will falsely report an error instead of not being invoked and reporting a fake error.

aws-lambda-chaos-injection/chaos_lambda.py

Lines 343 to 346 in a6d10af

	if _fault_type == "status_code":
	result = func(*args, **kwargs)
	if isinstance(_chaos_conf.get("error_code"), int):
	_error_code = _chaos_conf.get("error_code")

The simple solution would be to fix the order of operations. The structurally better solution would be to rebuild the whole library with a hooks pattern where the customer could define a list of faults/frequencies to inject before and after function execution.

Support for local configuration file

Using SSM is great, but maybe someone would prefer not having additional latency just to read the configuration.

Supporting a local configuration file or even just reading the configuration from an environment variable could be a useful alternative.

Dynamic exception and message for @inject_exception

I'd expect to be able to configure something like this:

@inject_exception(TypeError)
def lambda_handler_with_type_error(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_exception(ValueError)
def lambda_handler_with_value_error(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_exception(ValueError, "Custom message here")
def lambda_handler_with_custom_message(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

Handle AccessDenied

When the Lambda function doesn't have enough permissions to read the SSM parameter, you get an AccessDeniedException ("botocore.exceptions.ClientError: An error occurred (AccessDeniedException)"). We could handle that here.

This way we could show customers a proper error such as "Your Lambda function is missing the required IAM permissions to read the chaos configuration from SSM" - and maybe even provide them with the correct JSON document, as documented here.

Dynamic delay for @inject_delay

I'd expect to be able to configure something like this:

@inject_delay(400)
def lambda_handler_with_short_delay(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_delay(2000)
def lambda_handler_with_long_delay(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

Improve tests

A few considerations about tests:

1. I think we shouldn't test for printed output in our tests, as it doesn't allow you to easily change and/or add new logs.

2. also, we could use a library such as moto to avoid actually API calls during unit tests, so anybody can run unit tests without configuring AWS credentials. Additionally, we can always run the same tests as "integration tests" on a real AWS account.

3. we could speed up all the tests related to delays by patching time.sleep. This is not super critical now, but it'd allow us to add more tests and test much longer intervals without waiting minutes to unit tests to complete.

For example, this:

@ignore_warnings
def test_get_delay(self):
    with patch('sys.stdout', new=StringIO()) as fakeOutput:
        response = handler('foo', 'bar')
        assert (
            'Injecting 400 of delay with a rate of 1' in fakeOutput.getvalue().strip()
        )
    self.assertEqual(str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}")

could simply become this:

def test_get_delay(self):
    with patch('time.sleep', return_value=None) as patched_time_sleep:
        response = handler('foo', 'bar')
        patched_time_sleep. assert_called()
    self.assertEqual(str(response), "{'statusCode': 200, 'body': 'Hello from Lambda!'}")

And we could mock SSM like this:

from moto import mock_ssm

@mock_ssm
class TestExceptionMethods(unittest.TestCase):
    ...

Document required permissions for SSM

It's not clear whether & which IAM permissions are needed to read the configuration from SSM.

Dynamic status code for @inject_statuscode

I'd expect to be able to configure something like this:

@inject_statuscode(301)
def lambda_handler_with_301(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }

@inject_statuscode(400)
def lambda_handler_with_300(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from Lambda!'
    }