jckuester/awsweeper

A tool for cleaning your AWS account

awsweeper's Issues

Filter by relative date

I would like to filter by a relative date instead of an absolute one. An example might be:

aws_instance:
  - created:
      before: 5d

This would delete all resources that are older than 5 days at the time of running awsweeper. We could use the standard Go terms like 5d, 2h, 2m etc.

Unable to delete Cloudformation stacks

I've this config:

aws_cloudformation_stack:
  - id: awsweeper-del-me

running it I get these logs:

awsweeper --region eu-west-1 --parallel 1 --debug --force config.yaml

   • downloaded and installed provider                  name=aws version=2.59.0
   • configured provider                                name=aws version=2.59.0
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
<snipping>
      • failed to import resource; trying to read resource without import error=Throttling: Rate exceeded
        status code: 400, request id: 18399ea7-5742-409a-9e62-14cf0a14cb51 id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: dca88124-cc18-4375-8aaf-39bcd0bbd6dd
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: cc2e00e1-1554-4bd3-9ca9-f235fd738e37
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: f78f05c8-a354-4465-92d1-5e1099bae9b5
<snipping>
      • failed to get tags        error=state is nil: <nil> id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
<snipping>
        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
                Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • unable to delete resource                          error=resource state is nil; need to call update first resource_id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
   • TOTAL NUMBER OF DELETED RESOURCES: 0

and I can still see the cfn stack in the aws console :(

I've tried it with version 0.7 and it seems to work:

 awsweeper --region eu-west-1 --debug --force config.yaml        
   • downloaded and installed provider                  name=aws version=2.43.0
   • configured provider                                name=aws version=2.43.0
   • using region: eu-west-1  
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)

        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted>/:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
        Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • worker starts deleting resource                    id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack worker_id=5
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
   • TOTAL NUMBER OF DELETED RESOURCES: 1

Filter resources when a specific tag is not present altogether

As far I understand, we can filter resources based on a specific tag's value. For example, the following is going to select all ecs clusters based on a tag named environment whose value is not play

aws_ecs_cluster:
  - tags:
      environment: NOT(play)

Is there any way we can filter resources, if a specific tag is not present altogether. So based on previous example, I want to delete all ecs_clusters where the tag environment is not present.

Thanks.

No dates for Cloudformation stacks

my config:

aws_cloudformation_stack:
  - id: NOT(mystackname1|mystackname2)
    created:
      after: 2020-01-01

my command: ./awsweeper --profile my-sandbox --region eu-west-1 --dry-run --output string ./config.yml

gives an empty result (so nothing to be deleted)

INFO[0000] using region: eu-west-1                      
INFO[0001] This is a test run, nothing will be deleted! 

If I check the json by removing the date filter I will see an object like this one:

[
  {
    "Type": "aws_cloudformation_stack",
    "ID": "arn:aws:cloudformation:eu-west-1:*****:stack/*****/****-****-****-***-****",
    "Tags": {
      "***": "****"
    },
    "Created": null,
    "Attrs": null
  },
... <redacted>

You can see here that the date field is null

Update 1: To be noted that i'm using version v0.4.1 because of #81
Update 2: I've cloned the project and compiled it, same result

Shrink the size of the binary

Using UPX you can shrink the size of the binaries from ~100mb to ~25mb. The settings I used were upx -9 <BINARY>.

$ upx -9 awsweeper
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX git-716d20  Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 27th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
 118215644 ->  25976848   21.97%   macho/amd64   awsweeper

Packed 1 file.

aws_ebs_snapshot doesn't support import

delete aws_ecs_snapshot failed with the following error:

error=failed to import resource: resource aws_ebs_snapshot doesn't support import

install.sh

Hi,

some reason I'm not able to install on my Mac os is there any help or guide,

Thanks you in advance.

$ curl -sSfL https://raw.githubusercontent.com/cloudetc/awsweeper/master/install.sh | sh -s v0.5.0
cloudetc/awsweeper info checking GitHub for tag 'v0.5.0'
cloudetc/awsweeper info found version: 0.5.0 for v0.5.0/darwin/amd64
localuser@xxxxxxxxx ~/allstate_Devops/awsweeper (master)
$ awsweeper --help
bash: awsweeper: command not found

Filters for all resource types

It would be great if I could set a config for all supported resource types. An example config might be:

"*":
  - tags:
    environment: test

or

all:
  - tags:
    environment: test

Release 0.5 has as assets terradozer

Weired stuff

SIGSEGV on MacOS

Didn't investigate much, but I figured it's useful to leave this stacktrace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x26db73b]

goroutine 177 [running]:
github.com/cloudetc/awsweeper/vendor/github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsKmsAliasRead(0xc420134000, 0x300f2a0, 0xc4200cca00, 0xc420134000, 0x0)
	/home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_kms_alias.go:122 +0x35b
github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc4203d4540, 0xc42056a000, 0x300f2a0, 0xc4200cca00, 0xc42024d408, 0x1, 0x3286980)
	/home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:354 +0x167
github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc4204e6460, 0xc42070c2d0, 0xc42056a000, 0xa, 0xc4205aa088, 0x0)
	/home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
github.com/cloudetc/awsweeper/command.(*Wipe).wipe.func1(0xc42063d800, 0xc420920480, 0xc42070c2d0, 0xc4206d9660, 0xc4204cec50)
	/home/janck/go/src/github.com/cloudetc/awsweeper/command/wipe.go:126 +0x26c
created by github.com/cloudetc/awsweeper/command.(*Wipe).wipe
	/home/janck/go/src/github.com/cloudetc/awsweeper/command/wipe.go:100 +0x210

panic: interface conversion: interface {} is *string, not *time.Time

Hello,

I'm running awsweeper (macOS Mojave 10.14.1) with a config.yml with inside :

aws_ami:

i got an error :

---

panic: interface conversion: interface {} is *string, not *time.Time

goroutine 1 [running]:
github.com/cloudetc/awsweeper/resource.DeletableResources(0xc000618958, 0x7, 0x32e4280, 0xc0005bf100, 0xc0005bf100, 0x0, 0x0, 0xc00071ae80, 0xc000a5d400)
	/home/janck/git/github.com/cloudetc/awsweeper/resource/resource.go:37 +0x71a
github.com/cloudetc/awsweeper/command.(*Wipe).Run(0xc000a892c0, 0xc00028c2d0, 0x1, 0x1, 0xc000a89230)
	/home/janck/git/github.com/cloudetc/awsweeper/command/wipe.go:65 +0x1ea
github.com/mitchellh/cli.(*CLI).Run(0xc00028e8c0, 0xc000a89200, 0x3d13023, 0x4)
	/home/janck/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:255 +0x207
github.com/cloudetc/awsweeper/command.WrappedMain(0xc00003a0b8)
	/home/janck/git/github.com/cloudetc/awsweeper/command/wrapped_main.go:92 +0x753
main.main()
	/home/janck/git/github.com/cloudetc/awsweeper/main.go:14 +0x22

other supported resources works fine.

[0.5.0] Regex filters for aws_iam_policy → id

We have some corporate governance stuff applied over my company's 100+ AWS accounts. We developers use Terraform, the corporate governance resources are created using a variety of other tools.

I'm trying to clean-up all IAM policies that are created by developers, and not managed by these governance tools. But it would appear that regex support is not working correctly against aws_iam_policies → id field.

I've tried:

aws_iam_policy:
  - id: NOT(CloudTrail_CloudWatchLogs_Policy)
  - id: NOT(DatapipeOpsAccessPolicy)
  - id: NOT(Kentik)
  - id: NOT(turbot)

…as well as…

aws_iam_policy:
  - id: NOT(.*CloudTrail_CloudWatchLogs_Policy.*)
  - id: NOT(.*DatapipeOpsAccessPolicy.*)
  - id: NOT(.*Kentik.*)
  - id: NOT(.*turbot.*)

Am I applying the values correctly?

Add Check for AWS Account ID Before Running

A common scenario in AWS deployments is to have a production account and a test account. To ensure that the deployment is only run against the test account, it would be really helpful if the awsweeper could check the account id passed in on the command line or environment variable and then verify that the credentials its using are for that account. If they don't match, then fail before executing.

If this feature sounds worthwhile, I'd be happy to submit a PR for it.

Support for Kinesis resources

Requesting addition of AWS Kinesis Data Streams as time permits.

Remove null fields with `null` value from JSON and YAML output

This is what it currently looks like:

  {
    "Type": "aws_iam_role",
    "ID": "AWSServiceRoleForAutoScaling",
    "Tags": null,
    "Created": null,
    "Attrs": null
  },

- type: aws_iam_role
  id: AWSServiceRoleForAutoScaling
  tags: {}
  created: null
  attrs: {}

Delete Lambda functions

It would be neat if the tool supported deleting aws_lambda_functions.

aws_kms_alias invalid memory address

I'm using MacOS and I'm testing the tool with all.yml, I think there is a bug with the option: aws_kms_alias. If I comment this option #aws_kms_alias works well.

Regards!

---
Type: aws_kms_alias
Found: 14
    Id: alias/aws/acm
    Id: alias/aws/connect
    Id: alias/aws/dynamodb
    Id: alias/aws/ebs
    Id: alias/aws/elasticfilesystem
    Id: alias/aws/es
    Id: alias/aws/glue
    Id: alias/aws/kinesisvideo
    Id: alias/aws/rds
    Id: alias/aws/lambda
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x26db73b]
goroutine 619 [running]:
github.com/cloudetc/awsweeper/vendor/github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsKmsAliasRead(0xc420298700, 0x300f2a0, 0xc4200b4a00, 0xc420298700, 0x0)
    /home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_kms_alias.go:122 +0x35b
github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42038ff10, 0xc420502640, 0x300f2a0, 0xc4200b4a00, 0xc420462390, 0x1, 0x3286980)
    /home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:354 +0x167
github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc42044b5e0, 0xc420502460, 0xc420502640, 0xa, 0xc420734ce8, 0x0)
    /home/janck/go/src/github.com/cloudetc/awsweeper/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
github.com/cloudetc/awsweeper/command.(*Wipe).wipe.func1(0xc4205e34a0, 0xc42052acf0, 0xc420502460, 0xc4208876e0, 0xc420510cc0)
    /home/janck/go/src/github.com/cloudetc/awsweeper/command/wipe.go:126 +0x26c
created by github.com/cloudetc/awsweeper/command.(*Wipe).wipe
    /home/janck/go/src/github.com/cloudetc/awsweeper/command/wipe.go:100 +0x210

Multi filter option

Hi,
Using awsls I can retreive some IAM Roles, for example :

• AWSCodePipelineServiceRole-eu-west-1-code-pipe
• AWSReservedSSO_AWSManager_43EDDe333zzerqszee23
• AWSServiceRoleForSSO
• codebuild-as-code-build-service-role
• MyCustomAccountAccessRole

My goal, delete all IAM Roles except :

• those who start with "AWSReserved*"
• those who start with "AWSServiceRole*"
• the dedicated "MyCustomAccountAccessRole"

So awsweeper should delete :

• AWSCodePipelineServiceRole-eu-west-1-code-pipe
• codebuild-as-code-build-service-role

I try to build a filter.yml file to do this... but without success. How can I build a multi filtering on one attribute like "id" here.
Any help please ?

Thanks.

Sweep all regions

It would be nice if I could specify all available regions to be swept. An example might be:

$ awsweep --region all config.yml

Pop up `panic: can't use ElementIterator on null value` error

Hi @jckuester , recently I got the following error while cleanup ecs resource. the weird thing is current region doesn't have any ecs cluster, but it still happens each time.

	Type: aws_ecs_cluster
	Found: 1

		Id:		arn:aws:ecs:ap-southeast-2:xxxxxxxxxxxx:cluster/default

	---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
panic: can't use ElementIterator on null value

goroutine 194 [running]:
github.com/zclconf/go-cty/cty.Value.ElementIterator(0x5f98800, 0xc00048b450, 0x0, 0x0, 0x7, 0x0)
	/home/jan/go/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1038 +0x11a
github.com/zclconf/go-cty/cty.Value.AsValueMap(0x5f98800, 0xc00048b450, 0x0, 0x0, 0x1)
	/home/jan/go/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1243 +0xfe
github.com/jckuester/terradozer/pkg/provider.enableForceDestroyAttributes(0x5f98800, 0xc00048b450, 0x0, 0x0, 0x203000, 0x203000, 0x203000, 0xf4a6622f9a8800fb)
	/home/jan/go/pkg/mod/github.com/jckuester/[email protected]/pkg/provider/provider.go:236 +0xf7
github.com/jckuester/terradozer/pkg/provider.TerraformProvider.DestroyResource.func1(0x4c0da0)
	/home/jan/go/pkg/mod/github.com/jckuester/[email protected]/pkg/provider/provider.go:200 +0x86
github.com/hashicorp/terraform/helper/resource.Retry.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/jan/go/pkg/mod/github.com/hashicorp/[email protected]/helper/resource/wait.go:22 +0x5e
github.com/hashicorp/terraform/helper/resource.(*StateChangeConf).WaitForState.func1(0xc00926e720, 0xc0001d4540, 0xc000850600, 0xc007e240c0, 0xc0082242d0, 0xc0082242c8)
	/home/jan/go/pkg/mod/github.com/hashicorp/[email protected]/helper/resource/state.go:103 +0x298
created by github.com/hashicorp/terraform/helper/resource.(*StateChangeConf).WaitForState
	/home/jan/go/pkg/mod/github.com/hashicorp/[email protected]/helper/resource/state.go:80 +0x1bf

The awsweeper policy is following with latest version v.0.10.1

aws_ecs_cluster:
  - tags:
      NOT(Owner): .*

Could you help to take a look?

Provide functionality deletion by ARN

It would be very useful if you can provide possibility delete any resource by ARN

Filter based on Creation Date

Hi There,

Awesome project, it would be fantastic if Creation Date filter would exist for all type of objects.

Cheers,
smarunich

How to add new resources

Great tool, by far the best out the several I've used.

I'd like to add some more resources, starting with aws_athena_named_query but possibly more in the future.

Is this guide still relevant, #21? It seems like you're working on a way to auto-generate the code. Would it be better to wait?

Thanks,

Support for CloudTrail resources

Requesting addition of AWS CloudTrail trails as time permits.

aws_iam_user doesn't work via new feature #101 in v0.8.0

Hi @jckuester , just let you know there may be a bug happened for aws_iam_user of #101
I confirmed all the iam users in my aws account have already tagged the Owner and the awsweeper policy with following:

aws_iam_user:
  - tags:
      NOT(Owner): .*

The result after applying this policy shows up it will filler out entire iam users which actually shouldn't be filler out.

For the other resources applying the same #101 pattern, they look good for now.

Feel free to let me know if anything miss-config from my side.

Feature Request: Delete by last use time

The fear I have with deleting by creation time is that I have old resources that are still being used. This is meaningful for things like CloudWatch Log Groups and IAM Access Keys.

Some of these are easy (e.g., IAM Access Keys have a last-used property) and some are less so (e.g., CloudWatch Log Groups last use time would be defined by the latest event in each Log Stream).

The addition of this functionality would make this tool incredibly useful to me.

awsweeper running in Lambda issue

Hi @jckuester, I just have a specific scenario
I want to run the awsweeper in AWS Lambda.
Things are quite good if I just deploy each Lambda with awsweeper in different region
And awsweeper is working as expected in each Lambda of regions

But for cost reason, setup 4 NAT GATEWAY for Lambda use in each region is quite expensive as awsweeper need to download something when running the CMD in Lambda

So I just try to setup 1 Lambda in a region basically running 4 CMD cleanup for save cost purpose with following:

./bin/awsweeper --region ap-northeast-1 --dry-run ./config/aws_resource.yml
./bin/awsweeper --region ap-northeast-2 --dry-run ./config/aws_resource.yml
./bin/awsweeper --region ap-southeast-1 --dry-run ./config/aws_resource.yml
./bin/awsweeper --region ap-southeast-2 --dry-run ./config/aws_resource.yml

But I got the issue in CloudWatch Loggroup which implies CMD not running correctly.

• downloaded and installed provider name=aws version=2.59.0
--
  | 2020-06-03T20:00:07.051+08:00 | • configured provider name=aws version=2.59.0
  | 2020-06-03T20:00:07.095+08:00 | • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
  | 2020-06-03T20:00:09.369+08:00 | panic: can't use ElementIterator on null value
  | 2020-06-03T20:00:09.369+08:00 | goroutine 1 [running]:
  | 2020-06-03T20:00:09.369+08:00 | github.com/zclconf/go-cty/cty.Value.ElementIterator(0x6378ba0, 0xc000506410, 0x0, 0x0, 0x10, 0x2)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/go/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1038 +0x101
  | 2020-06-03T20:00:09.369+08:00 | github.com/zclconf/go-cty/cty.Value.AsValueMap(0x6378ba0, 0xc000506410, 0x0, 0x0, 0x1)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/go/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:1243 +0xe5
  | 2020-06-03T20:00:09.369+08:00 | github.com/cloudetc/awsweeper/pkg/resource.GetTags(0xc0008f3770, 0xb, 0xb, 0x0)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/git/github.com/cloudetc/awsweeper/pkg/resource/select.go:54 +0xe7
  | 2020-06-03T20:00:09.369+08:00 | github.com/cloudetc/awsweeper/pkg/resource.Filter.Apply(0xc0004633e0, 0xc000ba2ea0, 0x3, 0x4, 0xc000ba2ea0, 0x3, 0x4)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/git/github.com/cloudetc/awsweeper/pkg/resource/select.go:15 +0x10a
  | 2020-06-03T20:00:09.369+08:00 | github.com/cloudetc/awsweeper/pkg/resource.List(0xc000528170, 0xc000bc0000, 0xc00068e400, 0xc000440b20, 0x5ab8ee6, 0x6, 0xc00032a390, 0x0, 0x0)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/git/github.com/cloudetc/awsweeper/pkg/resource/list.go:56 +0x61f
  | 2020-06-03T20:00:09.369+08:00 | main.mainExitCode(0x0)
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/git/github.com/cloudetc/awsweeper/main.go:157 +0xd77
  | 2020-06-03T20:00:09.369+08:00 | main.main()
  | 2020-06-03T20:00:09.369+08:00 | /home/jan/git/github.com/cloudetc/awsweeper/main.go:30 +0x22
  | 2020-06-03T20:00:09.373+08:00 | [ERROR] CalledProcessError: Command '['./bin/awsweeper', '--region', 'ap-northeast-1', '--dry-run', './config/aws_resource.yml']' returned non-zero exit status 2. Traceback (most recent call last):   File "/var/task/src/awsweeper.py", line 17, in lambda_handler     output2 = subprocess.check_output(CMD2, universal_newlines=True)   File "/var/lang/lib/python3.8/subprocess.py", line 411, in check_output     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,   File "/var/lang/lib/python3.8/subprocess.py", line 512, in run     raise CalledProcessError(retcode, process.args,
  | 2020-06-03T20:00:09.374+08:00 | END RequestId: ae1b42fc-50be-4e77-8b27-ceb92a1664a1

The above 4 CMD is running good in my local laptop.
So just want to know if something happen with the code that is not compatible for executing CMD with --region ap-northeast-1 and this specific region is not what Lambda deployed in.

i.e. I run ./bin/awsweeper --region ap-northeast-1 --dry-run ./config/aws_resource.yml in one Lambda and the Lambda is deployed in ap-northeast-2

The Lambda role is quite loose as I give Lambda most power to running anything, so I think this error is not related to permission issue.

Support for Lambda resources

Requesting addition of AWS Lambda functions as time permits.

Support for EKS resources

Hi,

since you're rely on terraform resources, can you add aws_eks_cluster support?

Thanks!

Support for GuardDuty resources

Requesting addition of AWS GuardDuty as time permits.

hangs running in ec2 with admin perms

Was testing this running in an ec2 instance with admin iam role and it hangs:

./awsweeper --region us-east-1 --dry-run dev.yml 
INFO[0000] using region: us-east-1

Haven't had time to dig into why it would be permanently hanging here. Initial permissions tested were:

"ec2:Delete*",
"ec2:Describe*",
"ec2:Disassociate*",
"ec2:TerminateInstances"

Changed that to an admin role and still hangs. Any ideas?

Support for SNS resources

Requesting addition of AWS SNS Topics and Subscriptions as time permits.

Support for Workspaces and Directory Service resources

Requesting addition of AWS Directory Service and AWS Workspaces as time permits.

Route53 Hosted Zone not working in GovCloud

When executing awsweeper 0.4.1 on macos 10.15.3 against either us-gov-west-1 or us-gov-east-1, AWS GovCloud regions, the route53 hosted zones appear in --dry-run mode but are not deleted when the parameter is removed. Command was invoked with full administrator privileges with no policies prohibiting hosted zone deletion.

How to contribute support for more resources to this project

The following explanation hopefully makes it easy contributing the resources you need to this project. As an example, we go through adding the resource aws_vpc (which is already supported).

1) Add Terraform type ID of the new resource type

Fine the terraform name of the AWS resource type to support (here is a list of all existing resource types) and add it to resource/supported.go.

	Vpc                 TerraformResourceType = "aws_vpc"

2) Add function to list resources

Add here a function that lists all resources of the new resource type. For this you need to find the method of the AWS go API to list all VPCs (you find it by looking in the go doc of the AWS API)

func (a *AWS) vpcs() (interface{}, error) {
	output, err := a.DescribeVpcs(&ec2.DescribeVpcsInput{})
	if err != nil {
		return nil, err
	}
	return output.Vpcs, nil
}

Supported resources need to be added to resource/supported.go.

3) Add delete ID

The field name of the ID used by Terraform to delete the resource. For this we actually have to look in the delete method of the AWS provider for the VPC

var (
	deleteIDs = map[TerraformResourceType]string{
                ...
		Vpc:                 "VpcId",
                ...
        }
)

4) Add custom filter method

In most cases can be left as filterGeneric. Some resources require custom filter methods.

5) Integration tests

Solid testing is necessary, since we don't want this tool to delete resources unexpectedly :-) Integration tests for each resource have its own file; the ones for aws_vpc are in test/vpc_test.go.

The test for each resource is structured as follows:

1. Create two different resources of aws_vpc (one with ID/tag foo, another with ID/tag bar)
2. Create a yaml filter for AWSweeper to delete resource with ID/tag foo
3. Check if resource with ID/tag foo is gone
4. Check if resource with ID/tag bar hasn't been deleted

enhancement: use terraform data lookup to enumerate resources

Suggestion to not use aws sdk describe-* API to enumerate instances, instead use output of terraform data lookups. This makes more sense we use terraform to delete resources so why not offload discovery too.

`
data "aws_instances" "ec2" {
instance_tags {
Name = "*"
}
instance_state_names = ["running", "stopped"]
}

output "ec2" {
value = "${data.aws_instances.ec2.ids}"
}
`

Code structure

Hey,
There are few parts in the code structure that can be improved.
For instance, wipe functionality can not be imported and get executed by another project because it's highly coupled to CLI args and printing the output directly to stdout.

By decoupling wiping functionality and keep command/* only to convert cli args to input params to existing functionalities the project can be used in both way, either from cli or by code.

Another thing that I think can be improved is logging.
I can see currently the project is using all log, logrus and cli.UI to output messages and logs.
This can be unified as well.

Let me know if you agree and need help to improve this. :)
Would be happy to help if I can.

Regards

Is --region option really working?

Hello folks,

I just discovered awseeper and I'm wondering about the usage of --region option.

With no AWS related environment variables, I have something strange:

$ env | grep AWS
$ awsweeper --region eu-west-3 --dry-run --output yaml awsweeper.yml  | cksum
INFO[0001] This is a test run, nothing will be deleted! 
3240463751 18140
awsweeper --region eu-west-1 --dry-run --output yaml awsweeper.yml  | cksum
INFO[0001] This is a test run, nothing will be deleted! 
3240463751 18140

Both call list me the same resources, but if I set AWS_DEFAULT_REGION:

export AWS_DEFAULT_REGION=eu-west-3
awsweeper --dry-run --output yaml awsweeper.yml  | cksum          
INFO[0001] This is a test run, nothing will be deleted! 
3240463751 18140
export AWS_DEFAULT_REGION=eu-west-1
awsweeper --dry-run --output yaml awsweeper.yml  | cksum
INFO[0001] This is a test run, nothing will be deleted! 
3032715009 2173

My awsweeper.yml is just a list of resources without any filters, like:

head awsweeper.yml 
---
aws_instance:
aws_autoscaling_group:
aws_ebs_snapshot:
aws_ebs_volume:

Thanks for feedbacks!

S3 Buckets not working in GovCloud

When executing awsweeper 0.4.1 on macos 10.15.3 against either us-gov-west-1 or us-gov-east-1, AWS GovCloud regions, the s3 buckets appear in --dry-run mode but are not deleted when the parameter is removed. Command was invoked with full administrator privileges with no policies prohibiting bucket deletion.

error: go get -u -v github.com/cloudetc/awsweeper

Hi - I am trying to get the project and try it out, but this error, please advise, thank you.

`github.com/aws/aws-sdk-go/service/ec2/ec2iface
github.com/cloudetc/awsweeper/resource
github.com/cloudetc/awsweeper/command

github.com/cloudetc/awsweeper/command

go/src/github.com/cloudetc/awsweeper/command/wrapped_main.go:137:27: cannot use conf (type *"github.com/hashicorp/terraform/terraform".ResourceConfig) as type *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".ResourceConfig in argument to p.Validate
go/src/github.com/cloudetc/awsweeper/command/wrapped_main.go:146:23: cannot use conf (type *"github.com/hashicorp/terraform/terraform".ResourceConfig) as type *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".ResourceConfig in argument to p.Configure
go/src/github.com/cloudetc/awsweeper/command/wrapped_main.go:151:9: cannot use &p (type *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".ResourceProvider) as type *"github.com/hashicorp/terraform/terraform".ResourceProvider in return argument:
*"github.com/hashicorp/terraform/terraform".ResourceProvider is pointer to interface, not interface
`

Support for ACM resources

Requesting addition of AWS Certificate Manager (ACM) as time permits.

[0.5.0] Complex tag keys

We have some corporate governance stuff applied over my company's 100+ AWS accounts. We developers use Terraform, the corporate governance resources are created using CloudFormation.

I'm trying to nuke all security groups created by developers, while not touching the ones generated by our overlords.

aws_security_group:
  - tags
      "aws:cloudformation:logical-id": NOT(.*)

tags is supposed to be a map, and since the key has colons in it, I believe I'm supposed to wrap it in quotation marks so the parser doesn't freak out. But when I do this (0.5.0 release), I get the following error message:

failed to unmarshal config: config.yml error=yaml: line 67: mapping values are not allowed in this context

Is there a schema I can look at, are the docs incorrect, or is is because of my weird YAML key? This issue seems a little different from #45, but maybe I'm wrong and they're related?

Support for ALB and NLB resources

Requesting addition of load balancer (ALB) and network load balancer (NLB) support within the AWS EC2 service as only elastic load balancers are supported.

Support for RDS resources

Requesting addition of AWS Relational Database (RDS) as time permits. Initially focusing on the deletion of the database instances themselves.

Homebrew

FYI I submitted a pr to get awsweeper into Homebrew: Homebrew/homebrew-core#33984

Support for CloudWatch resources

Requesting addition of AWS CloudWatch Logs, Alarms, and Events as time permits.

[MacOs] Region Override

I have a number of users defined in my aws credentials file that I would like to run awsweeper against, but in different regions. I have observed the following behaviour (all done using the --dry-run option)

1. When my credentials do not contain a region (which is what I want), I get the following response:

err: Not a valid region:

That is what I expected. However, when supplying the --region override for any region in which I have resources (along with the relevant --profile), it returns no resources. Running a describe on the resource with the same --profile and --region using the aws cli, I see the resources I expect.

2. When my credentials do contain a region, everything works fine.

3. When my credentials do contain a region, and I try to override it with the --region flag, it still just runs against the region defined in the credentials. So the override does not seem to work.

The behaviour I expected was that if I had credentials defined with or without a region, using the --region override would cause awsweeper to honour this override.

IAM role filter id ignored?

Hi! I'm using awsweeper v0.3.2 (binary release off github) with the following config file:

aws_ami:
aws_autoscaling_group:
aws_cloudformation_stack:
aws_ebs_snapshot:
aws_ebs_volume:
aws_efs_file_system:
aws_eip:
aws_elb:
aws_iam_group:
aws_iam_instance_profile:
aws_iam_policy:
aws_iam_role:
    - id: OrganizationAccountAccessRole
aws_iam_user:
aws_instance:
aws_internet_gateway:
aws_key_pair:
#aws_kms_alias:
#aws_kms_key:
aws_launch_configuration:
aws_nat_gateway:
aws_network_acl:
aws_network_interface:
aws_route53_zone:
aws_route_table:
aws_s3_bucket:
aws_security_group:
aws_subnet:
aws_vpc:
aws_vpc_endpoint:

The role with id OrganizationAccountAccessRole is deleted anyway. --dry-run shows OrganizationAccountAccessRole as the id of the resource, so it's not like the problem is it's secretly using an ARN as the id or anything.

Sweep all resources not working

When leveraging the entire all.yml services list and using 0.4.1 on macos 10.15.3 against AWS GovCloud regions, us-gov-west-1 and us-gov-east-1 where resource counts are higher than 800 per service awsweeper hangs and often needs to have the config file only include a subset or each service individually.

Had to break out ebs, eip, and security groups out to individual executions. Also, it appears as though vpc peering and public ip associations make it difficult to easily delete vpc's.

Tags filter not working

For me, the tags filter is not working... I have multiple resources that have the tag environment: test. However, when I run the following config:

aws_ami:
  - tags:
      environment: test
aws_autoscaling_group:
  - tags:
      environment: test
aws_cloudformation_stack:
  - tags:
      environment: test
aws_ebs_snapshot:
  - tags:
      environment: test
aws_ebs_volume:
  - tags:
      environment: test
aws_efs_file_system:
  - tags:
      environment: test
aws_eip:
  - tags:
      environment: test
aws_elb:
  - tags:
      environment: test
aws_iam_group:
  - tags:
      environment: test
aws_iam_instance_profile:
  - tags:
      environment: test
aws_iam_policy:
  - tags:
      environment: test
aws_iam_role:
  - tags:
      environment: test
aws_iam_user:
  - tags:
      environment: test
aws_instance:
  - tags:
      environment: test
aws_internet_gateway:
  - tags:
      environment: test
aws_key_pair:
  - tags:
      environment: test
aws_kms_alias:
  - tags:
      environment: test
aws_kms_key:
  - tags:
      environment: test
aws_launch_configuration:
  - tags:
      environment: test
aws_nat_gateway:
  - tags:
      environment: test
aws_network_acl:
  - tags:
      environment: test
aws_network_interface:
  - tags:
      environment: test
aws_route53_zone:
  - tags:
      environment: test
aws_route_table:
  - tags:
      environment: test
aws_s3_bucket:
  - tags:
      environment: test
aws_security_group:
  - tags:
      environment: test
aws_subnet:
  - tags:
      environment: test
aws_vpc:
  - tags:
      environment: test
aws_vpc_endpoint:
  - tags:
      environment: test

To me it looks like awsweeper doesn't support filtering by tags for all the AWS resources that officially support tags (e.g. aws_iam_role) in Terraform. Is that possible?