A Python package which wanders across your AWS account and records your resources in a variety of Storage Connectors
Home Page: http://cloudwanderer.io
License: MIT License
Now that secondary attributes are first-class citizens of our boto3 objects it no longer makes sense to discover them separately, in fact we're spending more time filtering them out!
Remove all methods CloudWanderer beginning with write_secondary_attribute and bring their functionality into the methods that begin write_resource.
To ensure feature parity with AWS Config Advanced Query we need to support: CodePipeline Pipeline
ERROR:cloudwanderer.utils:An error occurred (ValidationException) when calling the Query operation: Filter Expression can only contain non-primary key attributes: Primary key attribute: _resource_type_index
Traceback (most recent call last):
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/utils.py", line 21, in exception_logging_wrapper
method(**kwargs)
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/cloud_wanderer.py", line 106, in write_resources_in_region
self.write_resources_of_service_in_region(
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/cloud_wanderer.py", line 141, in write_resources_of_service_in_region
self.write_resources_of_type_in_region(
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/cloud_wanderer.py", line 175, in write_resources_of_type_in_region
self._clean_resources_in_region(service_name, resource_type, region_name, urns)
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/cloud_wanderer.py", line 201, in _clean_resources_in_region
storage_connector.delete_resource_of_type_in_account_region(
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/storage_connectors/dynamodb.py", line 276, in delete_resource_of_type_in_account_region
for resource in resource_records:
File "/Users/sammartin/git/CloudWanderer/cloudwanderer/storage_connectors/dynamodb.py", line 239, in read_resources
result = self.dynamodb_table.query(**query_args)
File "/Users/sammartin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/sammartin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "/Users/sammartin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/sammartin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Filter Expression can only contain non-primary key attributes: Primary key attribute: _resource_type_index
To ensure feature parity with AWS Config Advanced Query we need to support: AutoScaling ScheduledAction
To ensure feature parity with AWS Config Advanced Query we need to support: CodeBuild Project
Some resource attributes (eg vpc dns names enabled) are not returned by the main festive resource request and require additional api calls. These attributes should be populated into storage and returned when a urn is requested.
To ensure feature parity with AWS Config Advanced Query we need to support: ApiGatewayV2 Stage
Currently S3 buckets are enumerated and their regions queried in every region but only written in the region to which they pertain.
This will represent a significant overhead in accounts with lots of buckets that regularly query all regions.
To ensure feature parity with AWS Config Advanced Query we need to support: AutoScaling ScalingPolicy
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 VPCEndpointService
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 VPNGateway
Some services have identically named resources (e.g. cloudformation stacks and opsworks stacks).
With the current exclude_resources it's impossible to exclude cloudformation stacks without also excluding opsworks stacks (or vice versa).
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 CustomerGateway
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 EgressOnlyInternetGateway
Sub resources, especially those which do not have their own ARNs (e.g. inline policies), are currently quite difficult to discover as they require manually assembling the custom ARN to pull them out (or querying all subresources of that type and figuring out which ones belong to the resource you're interested in).
Populating an attribute into storage which lists the subresources' URNs will be easily maintainable and make resource discovery much easier.
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 RegisteredHAInstance
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 VPCEndpoint
Currently the objects sent by CloudWanderer to the StorageConnectors are the raw boto3 resources.
This means that resource attributes have to be written separately, and that it's infeasible to yield the resources from CloudWanderer methods as we're writing them (which might be a useful feature).
Instead we should require our CloudInterface classes to pass CloudWandererResources back when called, which can then be passed on to the storage connector as a standard. This will increase standardisation and allow us to shield CloudWanderer and the StorageConnectors from any changes in the boto3 resources' metadata.
Have a method per individual type of query is very cumbersome.
Much easier to refactor to allow optional arguments on a single read_resource method and handle the complexity without the user having to know about it.
We are a gnats hair from a standardised interface for the cloud providers.
Move any AWS specific methods into the boto3 interface and create an abstract class for the cloud provider.
e.g.
resource.get_secondary_attribute(name="vpc_enable_dns_support")
Tests are taking an excessively long time to run, especially when testing anything to do with managed policies.
See if there is a way to optimise this.
Currently only the parent resource gets deleted.
This will probably need the implementation of #59 in order to resolve.
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 VPNConnection
This will make it easier to pull the remaining resource data from resources taken from partial indexes
To ensure feature parity with AWS Config Advanced Query we need to support: ApiGatewayV2 Api
To ensure feature parity with AWS Config Advanced Query we need to support: CloudFront Distribution
To ensure feature parity with AWS Config Advanced Query we need to support: ACM Certificate
Currently any resources with multiple identifiers (e.g. iam role inline policies) have their identifiers concatenated on to the end of the URN. This needs formalising and testing to ensure it doesn't happen anywhere we don't want it to (e.g. where a single identifier is sufficient to uniquely identify the resource).
Resources that no longer exist should be cleaned up on writing of that resource in that region.
To ensure feature parity with AWS Config Advanced Query we need to support: AutoScaling AutoScalingGroup
In order to enable event based workflows (e.g. updating a resource in storage from a CloudWatch Event trigger) we need to be able to trigger the writing of a resource from its URN.
currently logs to root
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 FlowLog
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 EIP
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 Host
Some boto3 subresources match the cloudwanderer definition of a subresource (i.e. that it has no unique identity of its own and is only queryable from its parent.)
An example of this is ec2:route which only exists as part of a route table.
Update CloudWanderer to support these resources.
To ensure feature parity with AWS Config Advanced Query we need to support: AutoScaling LaunchConfiguration
To ensure feature parity with AWS Config Advanced Query we need to support: ApiGateway Stage
To ensure feature parity with AWS Config Advanced Query we need to support: XRay EncryptionConfig
To ensure feature parity with AWS Config Advanced Query we need to support: CloudTrail Trail
At the moment attempting to write a subresource will result in an error. In order to fully support event driven workflows we must also be able to write subresources.
To ensure feature parity with AWS Config Advanced Query we need to support: EC2 NatGateway
Be explicit about what region is being queried and exclude global resources not in that region.
To ensure feature parity with AWS Config Advanced Query we need to support: CloudFront StreamingDistribution
Allow multiple storage connectors so that we can write to multiple storage destinations at once (e.g. DynamoDB and S3)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.