skydive-project / skydive-flow-exporter Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
The idea is to provide VPC Logs (https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) Analytics capabilities by:
The motivation for generating VPC Flow Logs compliant records is that this enables us to make use of the available analytical tools which has emerged around the AWS standard, tools such as:
The MVP would include v2 format (most commonly used currently) stored in CSV encoding to S3. (using empty values for fields we can't easy populate). The desired output of the MVP would be integration with AWS analytics tools: https://aws.amazon.com/blogs/mt/analyzing-vpc-flow-logs-got-easier-with-support-for-s3-as-a-destination/
There are three version: v1, v2 and v3. It appears v2 is still in wide use; while v3 is the future version - so plan is to support both v2 and v3 with the default being v2.
Template:
<version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport> <dstport> <protocol> <packets> <bytes> <start> <end> <action> <log-status>
Accept record:
2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
No data:
2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA
Skipped data:
2 123456789010 eni-11111111aaaaaaaaa - - - - - - - 1431280876 1431280934 - SKIPDATA
An example of row:
2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html: "The maximum file size for a log file is 75 MB. If the log file reaches the file size limit within the 5-minute period, the flow log stops adding flow log records to it. Then it publishes the flow log to the Amazon S3 bucket, and creates a new log file."
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html: "Log files are saved to the specified Amazon S3 bucket using a folder structure that is determined by the flow log's ID, Region, and the date on which they are created. The bucket folder structure uses the following format."
Directory format:
bucket_ARN/optional_folder/AWSLogs/aws_account_id/vpcflowlogs/region/year/month/day/log_file_name.log.gz
Filename format:
aws_account_id_vpcflowlogs_region_flow_log_id_timestamp_hash.log.gz
Timestamp format:
YYYYMMDDTHHmmZ
First setup folder/bucket as detailed here https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html
Next run Skydive+Exporter and get logs to be reported to AWS S3.
Next check you can use Athena to query flow logs as detailed here: https://docs.aws.amazon.com/athena/latest/ug/vpc-flow-logs.html
Some of the VPC Flow logs record fields require special deeper integration (also requiring skydive core modifications)
But for the MVP we may only provide rudimentary support, or leave fields empty/default value.
The code has references to "github.com/skydive-project/skydive/contrib/exporters/core" in the import sections in numerous files under secadvisor/mod. When changing these to "github.com/skydive-project/skydive-flow-exporter/core" we get errors that various dependencies cannot be found in the expected places.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.