rpgreen/apilogs

Easy logging and debugging for Amazon API Gateway and AWS Lambda Serverless APIs

apilogs

apilogs is a fork of the excellent awslogs project with specific customizations suited to querying and streaming logs for Serverless APIs using Amazon API Gateway and AWS Lambda.

Simply provide an API Gateway API ID and Stage name and apilogs will automatically aggregate log events from all log streams for your API Gateway API as well as all Lambda function log streams attached to your API.

Installation/Running -------$ pip install apilogs

i.e. stream logs for your Serverless API:

$ apilogs get --api-id xyz123 --stage prod --watch

Grep for errors one hour ago using credentials from AWS CLI profile "myprofile":

$ apilogs get --api-id xyz123 --stage test2 --profile myprofile --aws-region us-east-1 --start='2h ago' --end='1h ago' | grep "ERROR"

Features

• Aggregate logs from across streams.
  • Aggregate all streams in a group.
  • Aggregate streams matching a regular expression.
• Colored output.
  • Highlight found text in logs
• List existing groups
  • $ apilogs groups
• List existing streams
  • $ apilogs streams /var/log/syslog
• Watch logs as they are created
  • $ apilogs get /var/log/syslog ALL --watch
• Human-friendly time filtering:
  • --start='23/1/2015 14:23'
  • --start='2h ago'
  • --start='2d ago'
  • --start='2w ago'
  • --start='2d ago' --end='1h ago'
• Retrieve event metadata:
  • --timestamp Prints the creation timestamp of each event.
  • --ingestion-time Prints the ingestion time of each event.

Options

• apilogs groups: List existing groups
• apilogs streams GROUP: List existing streams withing GROUP
• apilogs get GROUP [STREAM_EXPRESSION]: Get logs matching STREAM_EXPRESSION in GROUP.
  • Expressions can be regular expressions or the wildcard ALL if you want any and don't want to type .*.

Note: You need to provide to all these options a valid AWS region using --aws-region or AWS_REGION env variable.

Time options

While querying for logs you can filter events by --start -s and --end -e date.

• By minute:
  • --start='2m' Events generated two minutes ago.
  • --start='1 minute' Events generated one minute ago.
  • --start='5 minutes' Events generated five minutes ago.
• By hours:
  • --start='2h' Events generated two hours ago.
  • --start='1 hour' Events generated one hour ago.
  • --start='5 hours' Events generated five hours ago.
• By days:
  • --start='2d' Events generated two days ago.
  • --start='1 day' Events generated one day ago.
  • --start='5 days' Events generated five days ago.
• By weeks:
  • --start='2w' Events generated two week ago.
  • --start='1 week' Events generated one weeks ago.
  • --start='5 weeks' Events generated five week ago.

• Using specific dates:

  • --start='23/1/2015 12:00' Events generated after midday on the 23th of January 2015.
  • --start='1/1/2015' Events generated after midnight on the 1st of January 2015.
  • --start='Sat Oct 11 17:13:46 UTC 2003' You can use detailed dates too.

  Note, for time parsing awslogs uses dateutil.

• All previous examples are applicable for --end -e too.

Filter options

You can use --filter-pattern if you want to only retrieve logs which match one CloudWatch Logs Filter pattern. This is helpful if you know precisely what you are looking for, and don't want to download the entire stream.

For example, if you only want to download only the report events from a Lambda stream you can run:

$ apilogs get my_lambda_group --filter-pattern="[r=REPORT,...]"

Full documentation of how to write patterns: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html

Contribute

• Fork the repository on GitHub.
• Write a test which shows that the bug was fixed or that the feature works as expected.
  • Use tox command to run all the tests in all locally available python version.
• Send a pull request and bug the maintainer until it gets merged and published. :).

For more instructions see TESTING.rst.

Helpful Links

• http://aws.amazon.com/cloudwatch/
• http://boto.readthedocs.org/en/latest/ref/logs.html
• http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/cloudwatch_limits.html

How to provide AWS credentials to apilogs

Although, the most straightforward thing to do might be use --aws-access-key-id and --aws-secret-access-key or --profile, this will eventually become a pain in the ass.

• If you only have one AWS account, my personal recommendation would be to configure aws-cli. apilogs will use those credentials if available.
• If you have multiple AWS accounts or you don't want to setup aws-cli, I would recommend you to use envdir in order to make AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY available to apilogs.

apilogs's Issues

Error when using the --help or --version switches

I'm unable to load the help menu because awslogs is still referenced in the _version.py file.
#2

SyntaxError: invalid syntax

Hi,

I installed apilogs today with pip install apilogs. The install was successful. But when I invoke the command, I get below error:

apilogs groups --profile TestMgmtAdmin
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/bin/apilogs", line 7, in
from apilogs.bin import main
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/apilogs/init.py", line 2, in
from .core import AWSLogs # noqa
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/apilogs/core.py", line 168
print "Error fetching logs for Lambda function {0} with group {1}. This function may need to be invoked.".format(fxn, lambda_group, e)
^
SyntaxError: invalid syntax

I am getting the same error for any of the accounts I have set up profiles for. i.e. dev, test and prod.

PS: The awslogs command installs and runs successfully.

Error filtering log events if Lambda function is aliased in APIGW

Hi,

If the Lambda integration in APIGW points to an aliased Lambda function, the call to filter_log_events fails because the regex doesn't match. Might need to strip the the alias before calling filter_log_events.

C:\Users\me\Downloads\apilogs (master)
λ python -m apilogs.bin get --api-id xyz123 --stage live --watch --profile dev
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 923, in _bootstrap_inner
    self.run()
  File "C:\Users\me\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 871, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\me\Downloads\apilogs\apilogs\core.py", line 229, in generator
    response = self.client.filter_log_events(**kwargs)
  File "C:\Users\me\AppData\Local\Programs\Python\Python35-32\lib\site-packages\botocore\client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\me\AppData\Local\Programs\Python\Python35-32\lib\site-packages\botocore\client.py", line 407, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidParameterException) when calling the FilterLogEvents operation: 1 validation error detected: Value '/aws/lambda/IDM_ChangeEmailAddress:live' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+

Collect events from all associated lambdas

Hello! 👋

I first wanted to say thanks for creating this tool. It promises to make our team's serverless development and support processes significantly simpler. I appreciate the time you've spent here!

When we were kicking the tires yesterday, we noticed that when there are multiple lambdas associated to a gateway, apilogs was only reporting the events from the first lambda, instead of the whole collection.

I was able to fix the problem locally and expect to have PR on the way in the next day or so, but I wanted to get this recorded for your visibility.

Omitted optional args cause error

I'm trying to get the logs from a single lambda function, but I'm having trouble.

My aws-cli is configured with keys. Running apilogs get /aws/lambda/cinemaMaster spits out an error message complaining about missing optional args.

Traceback (most recent call last):
  File "/usr/local/bin/apilogs", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/apilogs/bin.py", line 149, in main
    options.log_group_name = "API-Gateway-Execution-Logs_" + options.api_id + "/" + options.stage
TypeError: cannot concatenate 'str' and 'NoneType' objects