gorillastack/athena-cloudtrail-partitioner

Automate the daily partitioning of your CloudTrail bucket in Athena

athena-cloudtrail-partitioner's Issues

Lambda times out when before getting all partitions in CloudTrail bucket

The code doesn't work on a CloudTrail bucket with lots of data - for example a CloudTrail bucket with a years data from 100+ accounts and all regions.

Before the call in line 17 in the handler.js is finished, the lambda function reaches its execution time limit:
to const partitionTree = await getAllParitions(bucket, path);

Also, please note that you have a minor typo in getAllParitions method - it should probably be getAllPartitions, but since the method is also spelled the same way in s3.js it doesn't really matter.

What does matter is, that it can take more than 15 minutes to enumerate a CloudTrail with lots of data. Is there a way you can store the enumeration data in DynamoDB as well, so multiple runs of the Lambda could allow it to pick up where it left?

Lambda permissions policy missing

Hi,

I set up the Athena CloudTrail Partitioner and a bit after I noticed FailedInvocations in the CloudWatch Events metrics for the scheduled rule. Per https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_Troubleshooting.html#LAMfunctionNotInvoked it seems that there was a permissions policy missing on the Lambda function. I added the following CloudFormation resource to the stack and all appears to be well now.

## Permissions for CloudWatch Events to invoke the Lambda function
 PartitionLambdaPolicy:
   Type: AWS::Lambda::Permission
   Properties:
     Action: lambda:InvokeFunction
     FunctionName: !Ref PartitionLambda
     Principal: events.amazonaws.com
     SourceArn: !GetAtt LambdaSchedule.Arn

Throwing this out there in the hope that it might help anyone else.

Ryan

Deploy fails outside of ap-southeast-2

It seems that it can't be deployed in regions other than ap-southeast-2 because the code S3 bucket is in that region. I'm going to try and copy the code to a bucket in us-east-1 and use the parameters to override this, but at the very least that process should be documented.