netflix/consoleme

A Central Control Plane for AWS Permissions and Access

ConsoleMe

Check out our quick start guide , documentation , feature videos , ReInvent Talk, and Blog Post .

Overview

ConsoleMe is a web service that makes AWS IAM permissions and credential management easier for end-users and cloud administrators.

ConsoleMe provides numerous ways to log in to the AWS Console.

An IAM Self-Service Wizard lets users request IAM permissions in plain English. Cross-account resource policies will be automatically generated, and can be applied with a single click for certain resource types.

Weep (ConsoleMe’s CLI) supports 5 different ways of serving AWS credentials locally.

Cloud administrators can create/clone IAM roles and natively manage IAM roles, users, inline/managed policies, S3 Buckets, SQS queues, and SNS topics across hundreds of accounts in a single interface.

Users can access most of your cloud resources in the AWS Console with a single click. Cloud administrators can configure ConsoleMe to authenticate users through ALB Authentication, OIDC/OAuth2, or SAML.

… And more. Check out our docs to get started.

Project resources

• Discord
• Docs
• Weep (our CLI)
• Source Code
• Issue tracker
• Blog Post
• ReInvent Talk
• Anonymous Feedback Form

Third Party Mentions

• Achieving least-privilege at FollowAnalytics with Repokid, Aardvark and ConsoleMe
• Netflix’s ConsoleMe local installation on Linux machine
• Improving database security at FollowAnalytics with AWS IAM database authentication and ConsoleMe
• Awesome IAM Policy Tools
• Netflix on AWS Case Study
• Netflix Open Sources ConsoleMe to Manage Permissions and Access on AWS

Companies that use ConsoleMe (alphabetically sorted)

• AB180
• Calm
• FollowAnalytics
• myKaarma
• National Center for Biotechnology Information
• Feel free to submit a PR or let us know in an Issue if you'd like to add your company to this list.

consoleme's Issues

Allow ConsoleMe to sync accounts from multiple AWS organizations

ConsoleMe currently only allows syncing from one AWS Organization. Allow syncing across multiple AWS organizations.

Unable to open the page..........!

Hi,

Someone help me, I'm getting the following error.

ERROR: Service 'consoleme' depends on service 'consoleme-redis' which is undefined.

Backend Refactor Notes

The backend refactor for ConsoleMe will be tracked in this project board. This issue will be used for initial exploration and discussion around the plan for the refactor.

Module structure

Current state

consoleme
├── __init__.py
├── __main__.py
├── celery
│   ├── __init__.py
│   └── celery_tasks.py
├── config
│   ├── __init__.py
│   └── config.py
├── exceptions
│   ├── __init__.py
│   └── exceptions.py
├── handlers
│   ├── __init__.py
│   ├── audit.py
│   ├── autologin.py
│   ├── base.py
│   ├── credentials.py
│   ├── dynamic_config.py
│   ├── errors.py
│   ├── groups.py
│   ├── headers.py
│   ├── health.py
│   ├── index.py
│   ├── policies.py
│   ├── request.py
│   ├── roles.py
│   ├── saml.py
│   ├── swagger.py
│   └── users.py
├── lib
│   ├── __init__.py
│   ├── alb_auth.py
│   ├── auth.py
│   ├── aws.py
│   ├── cookie-py
│   ├── crypto.py
│   ├── duo.py
│   ├── dynamo.py
│   ├── elasticsearch.py
│   ├── generic.py
│   ├── google.py
│   ├── groups.py
│   ├── handler_utils.py
│   ├── json_encoder.py
│   ├── jwt.py
│   ├── oauth2.py
│   ├── plugins.py
│   ├── policies.py
│   ├── redis.py
│   ├── requests.py
│   ├── s3_helpers.py
│   ├── saml.py
│   ├── ses.py
│   ├── singleton.py
│   └── timeout.py
├── routes.py
└── templates
    ├── __init__.py
    └── [snip]

Future state

consoleme
├── __init__.py
├── __main__.py
├── celery
│   ├── __init__.py
│   └── celery_tasks.py
├── config
│   ├── __init__.py
│   └── config.py
├── exceptions
│   ├── __init__.py
│   └── exceptions.py
├── handlers
│   ├── __init__.py
│   ├── v1
│   │   ├── __init__.py
|   │   ├── audit.py
|   │   ├── autologin.py
|   │   ├── base.py
|   │   ├── credentials.py
|   │   ├── dynamic_config.py
|   │   ├── errors.py
|   │   ├── groups.py
|   │   ├── headers.py
|   │   ├── health.py
|   │   ├── index.py
|   │   ├── policies.py
|   │   ├── request.py
|   │   ├── roles.py
|   │   ├── saml.py
│   │   └── swagger.py
│   └── v2
│       ├── __init__.py
│       ├── [handler_name_1].py
│       ├── [...]
│       └── [handler_name_n].py
├── lib
│   ├── __init__.py
│   ├── alb_auth.py
│   ├── auth.py
│   ├── aws.py
│   ├── cookie-py
│   ├── crypto.py
│   ├── duo.py
│   ├── dynamo.py
│   ├── elasticsearch.py
│   ├── generic.py
│   ├── google.py
│   ├── groups.py
│   ├── handler_utils.py
│   ├── json_encoder.py
│   ├── jwt.py
│   ├── oauth2.py
│   ├── plugins.py
│   ├── policies.py
│   ├── redis.py
│   ├── requests.py
│   ├── s3_helpers.py
│   ├── saml.py
│   ├── ses.py
│   ├── singleton.py
│   └── timeout.py
├── routes.py
└── templates
    ├── __init__.py
    └── [snip]

Tasks

Checkboxes indicate issues have been created, not that the work has been completed.

• restructure handlers/ to have nested dirs - #58
  • handlers/v1/
    • move existing handlers here
  • handlers/v2/
  • update routes.py to point to v1 handlers
• swagger spec
  • enumerate endpoints - #60
  • enumerate methods for each endpoint - #61
  • work with hee won to iterate on API contract for each endpoint/method - #62
• restructure lib/?
  • might just want to add new methods into existing lib modules. gotta noodle on this.
• determine abstractions for data access. this could help us separate out celery tasks into separate services/apis

Feature request: Make index page and enabled/disabled pages configurable

@castrapel mentioned that the index page can be disabled and have it point to another endpoint by modifying the internal routes here

He also mentioned that for modifying the header page, you are working on refactoring that into react. More notes From our chat for your reference:

What you should be able to do for the internal routes is: (r"/?", PolicyViewHandler). Your "/" route should take precedence over the OSS routes here: https://github.com/Netflix-Skunkworks/consoleme/blob/master/consoleme/routes.py

Direct integration with LDAP/AD to retrieve user groups

Purpose: Enable ConsoleMe to retrieve a user's groups from LDAP / Active Directory

Context: ConsoleMe currently expects your SSO identity provider to pass in a user's groups as a claim.

We want to support retrieving groups directly from LDAP/AD. We don't currently have any integrations with LDAP/AD.

I'm not getting the list of *ROLE*

I'm able to see the policies and existing roles in the roles and policies tab.
But I'm unable to see anything in AWS CONSOLE ROLE.

Help Needed

Allow exporting entire IAM/SQS/SNS/S3 resource to terraform in policy editor

ConsoleMe's policy editor currently supports IAM roles, SQS queues, SNS topics, and S3 buckets. The policy editor should allow users to export their resources into Terraform. The UI should provide this option if export_to_terraform_enabled is True.

Unable to create trust policy for central account - Prohibited field "Principal"

Describe the bug
When creating a trust policy for my central account role based on these instructions, I'm unable to create a trust policy for my role. When I attempt to add the policy to my newly created role, Amazon reports the following error:

This policy contains the following error: Has prohibited field Principal For more information about the IAM policy grammar, see AWS IAM Policies.

To Reproduce
Steps to reproduce the behavior:

1. Create a AWS IAM role without a policy.
2. Once the AWS IAM role is created, attach a new policy to it with the following JSON:

{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      }
    },
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "INSERT_THE_ROLE_ARN_YOU_JUST_CREATED_HERE"
      }
    }
  ],
  "Version": "2012-10-17"
}

3. When saving the trust policy you will see the Amazon error.

Expected behavior
I should see the policy created.

Screenshots
Here is a screenshot of the error:

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

Additional context

There are two possibilities here: either I don't understand fully AWS IAM roles and policies, or the sample trust policy in your instructions is not correct. I am leaning towards this issue being the former case but I'm going to go ahead and post this issue anyways just in case it's not.

Create empty Swagger spec for new API endpoints

Create a Swagger YAML with necessary metadata but no endpoints.

Reporting Security Warnings

Describe the bug
N/A

To Reproduce
N/A

Expected behavior
N/A

Screenshots
N/A

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

Additional context
I apologize if this is the wrong place to put this. During my security research I've discovered a security vulnerability dud. What I mean by "dud" is that at this time there is no security vulnerability (it's more of a security warning) however if/when the feature is completed there will be a security vulnerability (assuming no changes are made on the vulnerable code). I'm erring on the side of reporting this as an issue on this Github repo. However, before I do so, I want to know, would you rather have me submit this through Bugcrowd? Thanks for your time and help guys!

Allow displaying Service Control Policies that affect a role

Service Control Policies can be used in environments using AWS Organizations. This policy defines the upper boundary of your resources. In the IAM role policy editor, we should show the Service Control Policies affecting that role. This should be configurable in three ways:

1. Do not show SCPs affecting the role
2. Show SCPs affecting the role to all users
3. Show SCPs affecting the role to ConsoleMe administrators

Enumerate API endpoints in Swagger

Work with @saillinux to determine the endpoints needed for the v2 API and document them in the Swagger document created from #59.

ConsoleMe fails to start

Just cloned the project, configured aws credentials, and ran docker-compose -f docker-compose.yaml -f docker-compose-dependencies.yaml up -d but it fails to start. Am I missing a step?

The error message from the end of the console output is:

Setting up libxmlsec1-nss:amd64 (1.2.27-2) ...
Setting up libxmlsec1-gnutls:amd64 (1.2.27-2) ...
Setting up libxmlsec1-dev (1.2.27-2) ...
Processing triggers for libc-bin (2.28-10) ...
ERROR: Service 'consoleme-celery' failed to build : open /var/lib/docker/image/overlay2/layerdb/tmp/write-set-361021071/diff: read-only file system

And Docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 217
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.121-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 5
 Total Memory: 4.825GiB
 Name: docker-desktop
 ID: ZIZE:GGNU:3PTZ:KDOM:XY3H:3BFT:IQUR:MDPD:ICBW:NHWZ:N3WQ:JWXI
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 40
  Goroutines: 45
  System Time: 2020-12-12T05:35:09.2905047Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false

Update policy editor to support editing IAM managed policies

At Netflix, we use mostly inline policies to grant permissions to IAM roles. We want permissions granular to a role because it's easier to remove a role's unused permissions with Repokid. Some organizations use managed policies exclusively. ConsoleMe's policy editor should be updated to support (admin) creating, viewing, (admin) editing, and requesting changes to managed policies.

Refactor Policy Request Review Page in React

Paginator is missing for 'resources' search under Policies

In “Policies” table view, when viewing a paginated page other than the first page, if the search is used, no results appear. This seems to be a function of the results also being paginated (if there are enough results to be viewed on the page you were on, it appears to work), but the pagination widget is not visible and does not make it obvious there are no results or which page of the results you are on.

Works for Tech:

Does not work for Resources:

Docs: How to add / update Python / Javascript requirements

Update the docs on how to install specific dependencies, update specific dependencies, or update all dependencies for both JS and Python dependencies.

consoleme-celery yarn error during docker-compose installation

Describe the bug
When installing consoleme locally with docker and running the following command:

docker-compose -f docker-compose.yaml -f docker-compose-dependencies.yaml up -d

I receive the following error:

Step 20/22 : RUN yarn --cwd ui build:prod
 ---> Running in 049ad792edf1
yarn run v1.22.10
$ yarn run clean:prod && yarn build && yarn run copy:prod
$ rm -rf build && rm -rf ../consoleme/templates
$ react-app-rewired build
Creating an optimized production build...
events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at ChildProcess.target._send (internal/child_process.js:832:20)
    at ChildProcess.target.send (internal/child_process.js:703:19)
    at ChildProcessWorker.send (/apps/consoleme/ui/node_modules/jest-worker/build/workers/ChildProcessWorker.js:299:17)
    at WorkerPool.send (/apps/consoleme/ui/node_modules/jest-worker/build/WorkerPool.js:32:34)
    at Farm._process (/apps/consoleme/ui/node_modules/jest-worker/build/Farm.js:147:10)
    at Farm._enqueue (/apps/consoleme/ui/node_modules/jest-worker/build/Farm.js:176:10)
    at Farm._push (/apps/consoleme/ui/node_modules/jest-worker/build/Farm.js:183:12)
    at /apps/consoleme/ui/node_modules/jest-worker/build/Farm.js:106:14
    at new Promise (<anonymous>)
    at Farm.doWork (/apps/consoleme/ui/node_modules/jest-worker/build/Farm.js:69:21)
    at JestWorker._callFunctionWithArgs (/apps/consoleme/ui/node_modules/jest-worker/build/index.js:182:23)
    at /apps/consoleme/ui/node_modules/terser-webpack-plugin/dist/index.js:304:45
Emitted 'error' event on ChildProcess instance at:
    at internal/child_process.js:836:39
    at processTicksAndRejections (internal/process/task_queues.js:75:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: Service 'consoleme-celery' failed to build : The command '/bin/sh -c yarn --cwd ui build:prod' returned a non-zero code: 1

To Reproduce
Steps to reproduce the behavior:

1. Clone the repo
2. Cd into the project
3. Run the following command: docker-compose -f docker-compose.yaml -f docker-compose-dependencies.yaml up -d
4. See error

Expected behavior
Consoleme should be properly installed.

Screenshots
N/A

Desktop (please complete the following information):

• OS: MacOS Catalina 10.15.7
• Docker Desktop version 3.0.4

Smartphone (please complete the following information):
N/A

Additional context
None.

Replace ACE Editor in Self-Service-Step 3 (JSON Preview mode) with Monaco

We're still using ACE Editor in Self Service Step 3. We should replace this with our standard Monaco Policy Editor and remove all ACE dependencies

Allow requesting addition/removal of IAM Permission boundaries to IAM roles

Permission boundaries are similar to IAM managed policies, except they don't grant additional permissions to a role. Rather, they specify the maximum permissions a role can have.

A role may have a maximum of one permission boundary. Administrators should be able to add/remove a permission boundary via the role's policy editor page. End-users should be able to request the addition or removal of a permssion boundary.

It might be worth referencing the Managed Policy UI component, as the permission boundary component might look similar.

Feature request: Open Source for Recent Permission Errors from CloudTrail

As discussed previously, the "Recent permission errors" add on is not part of the open source yet. Would be very helpful to have that included.

Create Login Page Mockup (init_ui branch)

We still need to tidy up the documentation but here's a TLDR: Use the init_ui branch.

1. Run yarn setup (yarn start in the consoleme-ui directory)
   https://github.com/Netflix-Skunkworks/consoleme/blob/init_ui/consoleme-ui/README.md

2. For the backend setup, you should be able to run our OIDC demo:

docker-compose -f docker-compose-dependencies.yaml -f docker-compose-oidc.yaml up

And access the main site at http://localhost:8081 ( username and password is consoleme_user / consoleme_user or [email protected]/consoleme_user )

Create Terraform / Packer deployment scripts

Optimize Docker Image size

The docker image is unnecessarily large. There are optimizations we can do to reduce it, ie: The ones described here.

Feature Request: Kubernetes Deployment via Helm

It would be great for Kubernetes users if consoleme provides Helm Chart as a deployment strategy. I have a somewhat working version of consoleme Helm Chart. I can contribute my Helm Chart if it's ok.

Documentation

We need to make a docs page

Failing to start ConsoleMe

While working on #260, I'm trying to get the server to work. Towards the end it dies. I do have AWS Config running. I'm not clear on what I'm doing wrong.

From the end of cloud-init-output.log:

...
+ systemctl start celery
+ systemctl enable celery
Created symlink from /etc/systemd/system/multi-user.target.wants/celery.service to /etc/systemd/system/celery.service.
+ systemctl enable consoleme
Created symlink from /etc/systemd/system/multi-user.target.wants/consoleme.service to /etc/systemd/system/consoleme.service.
+ systemctl start consoleme
+ python3.8 /apps/consoleme/scripts/initialize_redis_oss.py
{"asctime": "2020-12-10T20:06:03Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "celery_tasks.py", "funcName": "cache_cloud_account_mapping", "levelname": "DEBUG", "lineno": 1298, "module": "celery_tasks", "threadName": "MainThread", "message": "Successfully cached cloud account mapping", "function": "consoleme.celery.celery_tasks.cache_cloud_account_mapping", "num_accounts": 1, "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:03Z+0000"}
{"asctime": "2020-12-10T20:06:04Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "celery_tasks.py", "funcName": "cache_s3_buckets_for_account", "levelname": "DEBUG", "lineno": 1001, "module": "celery_tasks", "threadName": "MainThread", "message": null, "function": "consoleme.celery.celery_tasks.cache_s3_buckets_for_account", "account_id": "154724799477", "number_s3_buckets": 4, "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:04Z+0000"}
{"asctime": "2020-12-10T20:06:05Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "celery_tasks.py", "funcName": "cache_sns_topics_for_account", "levelname": "DEBUG", "lineno": 963, "module": "celery_tasks", "threadName": "MainThread", "message": null, "function": "consoleme.celery.celery_tasks.cache_sns_topics_for_account", "account_id": "154724799477", "number_sns_topics": 0, "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:05Z+0000"}
{"asctime": "2020-12-10T20:06:06Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "celery_tasks.py", "funcName": "cache_sqs_queues_for_account", "levelname": "DEBUG", "lineno": 923, "module": "celery_tasks", "threadName": "MainThread", "message": null, "function": "consoleme.celery.celery_tasks.cache_sqs_queues_for_account", "account_id": "154724799477", "number_sqs_queues": 0, "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:06Z+0000"}
{"asctime": "2020-12-10T20:06:07Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "celery_tasks.py", "funcName": "cache_managed_policies_for_account", "levelname": "DEBUG", "lineno": 808, "module": "celery_tasks", "threadName": "MainThread", "message": null, "function": "consoleme.celery.celery_tasks.cache_managed_policies_for_account", "account_id": "154724799477", "number_managed_policies": 765, "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:07Z+0000"}
{"asctime": "2020-12-10T20:06:07Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "exceptions.py", "funcName": "__init__", "levelname": "ERROR", "lineno": 14, "module": "exceptions", "threadName": "ThreadPoolExecutor-10_0", "message": "Unable to retrieve expected data.", "eventTime": "2020-12-10T12:06:02.941114-08:00", "hostname": "ip-10-1-1-11", "timestamp": "2020-12-10T20:06:07Z+0000"}
Traceback (most recent call last):
  File "/apps/consoleme/scripts/initialize_redis_oss.py", line 59, in <module>
    celery.cache_resources_from_aws_config_for_account(account_id)
  File "/apps/consoleme/env/lib64/python3.8/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/apps/consoleme/env/lib64/python3.8/site-packages/celery/app/task.py", line 393, in __call__
    return self.run(*args, **kwargs)
  File "/apps/consoleme/env/lib64/python3.8/site-packages/celery/app/autoretry.py", line 50, in run
    raise task.retry(exc=exc, **retry_kwargs)
  File "/apps/consoleme/env/lib64/python3.8/site-packages/celery/app/task.py", line 690, in retry
    raise_with_context(exc or Retry('Task can be retried', None))
  File "/apps/consoleme/env/lib64/python3.8/site-packages/celery/app/autoretry.py", line 35, in run
    return task._orig_run(*args, **kwargs)
  File "/apps/consoleme/consoleme/celery/celery_tasks.py", line 1125, in cache_resources_from_aws_config_for_account
    redis_result_set = async_to_sync(retrieve_json_data_from_redis_or_s3)(
  File "/apps/consoleme/env/lib64/python3.8/site-packages/asgiref/sync.py", line 139, in __call__
    return call_result.result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/apps/consoleme/env/lib64/python3.8/site-packages/asgiref/sync.py", line 204, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/apps/consoleme/consoleme/lib/cache.py", line 158, in retrieve_json_data_from_redis_or_s3
    raise DataNotRetrievable("Unable to retrieve expected data.")
consoleme.exceptions.exceptions.DataNotRetrievable: Unable to retrieve expected data.
+ echo 'Running custom userdata script'
Running custom userdata script
Cloud-init v. 19.3-3.amzn2 finished at Thu, 10 Dec 2020 20:06:08 +0000. Datasource DataSourceEc2.  Up 438.67 seconds

Make Celery Tasks and Worker OSS compatible

ConsoleMe failed to start

Describe the bug
I've installed Consoleme through docker and ran docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml up -d in the install directory. And I can see docker containers running

CONTAINER ID   IMAGE                       COMMAND                  CREATED        STATUS          PORTS                                            NAMES
0e3475a59f34   consoleme/consoleme         "bash -c ' python sc…"   47 hours ago   Up 11 seconds   0.0.0.0:3000->3000/tcp, 0.0.0.0:8081->8081/tcp   consoleme_consoleme_1
d6a18f78293b   consoleme/consoleme         "bash -c ' python sc…"   47 hours ago   Up 11 seconds   8081/tcp                                         consoleme_consoleme-celery_1
f514ab72cc3d   cnadiminti/dynamodb-local   "java -jar DynamoDBL…"   47 hours ago   Up 45 hours     8000/tcp, 0.0.0.0:8005->8005/tcp                 consoleme-dynamodb
cf5703c5d435   redis:alpine                "docker-entrypoint.s…"   47 hours ago   Up 13 seconds   0.0.0.0:6379->6379/tcp                           consoleme-redis

But I can't load http://localhost:3000 page in browser. Here are the logs from consoleme_consoleme_1 container

2021-03-12 07:09:49,382 - WARNING - root - [retrieve_or_decode_configuration.py:56 - <module>() ] - Neither the CONSOLEME_CONFIG_B64 or CONSOLEME_CONFIG_S3 environment variables are defined. This is required to retrieve ConsoleMe's configuration via S3, or decode it from Base64. Exiting
Requirement already satisfied: argh in /usr/local/lib/python3.8/site-packages (0.26.2)
Requirement already satisfied: watchdog in /usr/local/lib/python3.8/site-packages (2.0.2)
Obtaining file:///apps/consoleme/default_plugins
Installing collected packages: consoleme-default-plugins
  Attempting uninstall: consoleme-default-plugins
    Found existing installation: consoleme-default-plugins 0.1
    Uninstalling consoleme-default-plugins-0.1:
      Successfully uninstalled consoleme-default-plugins-0.1
  Running setup.py develop for consoleme-default-plugins
Successfully installed consoleme-default-plugins
Traceback (most recent call last):
  File "/usr/local/bin/watchmedo", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/watchdog/watchmedo.py", line 581, in main
    parser.dispatch()
  File "/usr/local/lib/python3.8/site-packages/argh/helpers.py", line 55, in dispatch
    return dispatch(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/argh/dispatching.py", line 174, in dispatch
    for line in lines:
  File "/usr/local/lib/python3.8/site-packages/argh/dispatching.py", line 277, in _execute_command
    for line in result:
  File "/usr/local/lib/python3.8/site-packages/argh/dispatching.py", line 231, in _call
    result = function(namespace_obj)
  File "/usr/local/lib/python3.8/site-packages/watchdog/watchmedo.py", line 555, in auto_restart
    observe_with(observer, handler, args.directories, args.recursive)
  File "/usr/local/lib/python3.8/site-packages/watchdog/watchmedo.py", line 111, in observe_with
    observer.start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/api.py", line 256, in start
    emitter.start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/utils/__init__.py", line 93, in start
    self.on_thread_start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify.py", line 118, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_buffer.py", line 35, in __init__
    self._inotify = Inotify(path, recursive)
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_c.py", line 187, in __init__
    Inotify._raise_error()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_c.py", line 437, in _raise_error
    raise OSError(err, os.strerror(err))
OSError: [Errno 38] Function not implemented
yarn run v1.22.10

To Reproduce
Steps to reproduce the behavior:

• Checkout ConsoleMe git repo https://github.com/Netflix/consoleme
• cd into consoleme directory
• Run docker-compose -f docker-compose-dockerhub.yaml -f docker-compose-dependencies.yaml up -d
• Check the logs of consoleme_consoleme-celery_1 container to see the errors

Expected behavior
I should be able to access ConsoleMe web interface

Desktop (please complete the following information):

• OS: MacBook Pro (13-inch, M1, 2020) - Bigslur-v11.0.1 - AppleM1
• Browser - Brave
• Docker version 20.10.1, build 831ebeae96
• docker-compose version 1.27.4, build 40524192

Enumerate endpoint methods in Swagger

Building on the outcome of #60, determine what HTTP methods need to be supported for each v2 API endpoint.

Update policy editor to support IAM users

Netflix runs almost exclusively with IAM roles. We discourage the use of IAM users. However, in reality, many of us are forced to use IAM users due to vendor tooling that requires it. We should update the policy editor to support IAM users similar to how it supports IAM roles.

Dead Code Documentation

Is your feature request related to a problem? Please describe.
I'm finding dead code in this application. I'm not entirely sure why this code exists. I have two guesses:

• Not all of Consoleme code is posted in this repo.
• The dead code was used at one point but is now removed.

Describe the solution you'd like
Can there be some explanation in the documentation that explains why dead code exists, assuming it's because there is code not pushed to this repo.

Describe alternatives you've considered
The dead code could be removed but that would be a lot of work. This assumes that the code is in fact dead.

Additional context
Since I'm a security researcher I'm having a hard time determining whether or not I should submit bugs for "dead" code (if a vulnerability is found). There are requirements that I have to meet when I submit security bugs, for example a proof of concept, and when I'm not sure if this dead code is used by Netflix I won't know what to do. Some additional guidance on this would be fantastic.

Move existing handlers to subdirectory

• Move the existing HTTP handler modules to consoleme/handlers/v1/
• Update consoleme/routes.py to point to the new handlers.
• Create consoleme/handlers/v2/__init__.py. This module will be the home of the new API handlers.

Feature request: "Export to Terraform" option in Self service permissions after policy creation

It would be really helpful to have a Terraform formatted version of the self service permissions.

Example output format:

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:Describe*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy

Feature request: Resource ARN fields for Self service permissions

I feel like it is very easy for users to specify * in the resource ARN field. this is when you go to "Roles and Policies" -> "self Service Permissions" -> Select a role -> In step 2, "Provide Permission details", under "Select Desired Permissions", select "Other." You will notice that the "Resource ARN or wildcard" field is wildcard by default.

Of course, we should provide the user with the ability to do * anyway, but if we can guide them to use resource constraints, even when the resource is not filled in via typeahead + AWS Config, that would be very valuable.

Ideally, this would allow the user to select valid resource ARN types available via that service (check out this query from Policy Sentry), grab the raw ARN format, and then allow the user to fill out the fields that they need to fill out in order to create a valid ARN for that resource type.

A couple of security issues found in "central-account" Terraform

I ran Cloudrail against the central-account, and it found a couple of interesting things:

Rule: Ensure no used security groups allow ingress from 0.0.0.0/0 or ::/0 to port 22 (SSH)
 - 1 Resources Exposed:
-----------------------------------------------
   - Exposed Resource: [module.server.aws_instance.this[0]] (.terraform/modules/server/main.tf:5)
     Violating Resource: [aws_security_group.external]  (security_group.tf:1)

     Evidence:
         Internet
             | Subnet module.network.aws_subnet.public[0] has Internet Gateway
             | Instance module.server.aws_instance.this[0] is on module.network.aws_subnet.public[0]
             | Subnet routes traffic from instance to Internet Gateway
             | Subnet uses NACL nacl-pseudo-7b63af1c-0465-416e-8fc5-67230d1c9c17 which allows port 22
             | Instance uses Security Group ['aws_security_group.external']
             | Security Group allows port 22
         Instance

-----------------------------------------------
Rule: Ensure IMDSv2 is used and IMDSv1 is disabled
 - 1 Resources Exposed:
-----------------------------------------------
   - Exposed Resource: [module.server.aws_instance.this[0]] (.terraform/modules/server/main.tf:5)
     Violating Resource: [module.server.aws_instance.this[0]]  (.terraform/modules/server/main.tf:5)

     Evidence:
             | The EC2 module.server.aws_instance.this[0] is allowing IMDSv1

-----------------------------------------------
Rule: Ensure VPC Endpoint for S3 is enabled in all route tables in use in a VPC
 - 1 Resources Exposed:
-----------------------------------------------
   - Exposed Resource: [module.network.aws_vpc.this[0]] (.terraform/modules/network/main.tf:24)
     Violating Resource: [module.network.aws_route_table.public[0]]  (.terraform/modules/network/main.tf:101)

     Evidence:
             | The VPC module.network.aws_vpc.this[0] has a S3 Endpoint Gateway, but module.network.aws_subnet.public[0] uses module.network.aws_route_table.public[0], which does not have a route to the endpoint gateway

-----------------------------------------------
Rule: Ensure VPC Endpoint for DYNAMODB is enabled in all route tables in use in a VPC
 - 1 Resources Exposed:
-----------------------------------------------
   - Exposed Resource: [module.network.aws_vpc.this[0]] (.terraform/modules/network/main.tf:24)
     Violating Resource: [module.network.aws_route_table.public[0]]  (.terraform/modules/network/main.tf:101)

     Evidence:
             | The VPC module.network.aws_vpc.this[0] has a DYNAMODB Endpoint Gateway, but module.network.aws_subnet.public[0] uses module.network.aws_route_table.public[0], which does not have a route to the endpoint gateway

• The first one about the 0.0.0.0: it's true that @kmcquade and @castrapel mentioned in the examples NOT to use 0.0.0.0, but maybe we should have the example set to the private subnet by default, to avoid mistakes?
• The second one is a limitation of the module used to create the server, I opened a ticket for it. However, having a server that is publicly accessible without IMDSv2 can cause problems (especially with a web server).
• The following have to do with how the VPC module is being used. You're asking for S3 and DynamoDB VPC endpoints, but not actually using them.

I understand this is an example for people to adopt and adapt, but I think it may be a good idea to update the template. Happy to do it if you give the thumbs up.

Make unit tests open-source compatible

Many of the unit tests included in this repo require the usage of internal Netflix plugins. We should make the unit tests completely compatible with what is available in open source

Direct integration with Google to retrieve user groups

Purpose: Enable ConsoleMe to retrieve a user's Google groups from Google directly.

Context: ConsoleMe currently expects your SSO identity provider to pass in a user's groups as a claim.

We want to support retrieving groups directly from Google (AD/LDAP as well, but those are different issues).

Our existing Google API functionality is here: https://github.com/Netflix/consoleme/blob/master/consoleme/lib/google.py . This would serve as a good starting point.

jwt.decode and AWS ALB authentication (through Cognito)

In my current setup, I am using the AWS ALB + Cognito User Pool authentication.

When doing so, the JWT decoding fails at

consoleme/consoleme/lib/alb_auth.py

Line 54 in 069a1b7

access_token_jwt = jwt.decode(encoded_claims_jwt, pub_key, verify=False)

Simplified logs from journalctl -u consoleme:

[...] "name": "consoleme", "processName": "MainProcess", "filename":"alb_auth.py", "funcName": "authenticate_user_by_alb_auth", "levelname": "ERROR", "lineno": 68, "module": "alb_auth", "threadName": "MainThread", "message": "Unable to derive user's groups from access_token. This is expected for some identity providers.", "function": "consoleme.lib.alb_auth.authenticate_user_by_alb_auth", "error": "It is required that you pass in a value for the \"algorithms\" argument when calling decode().", "user": "USER" [...]

Changing the code to:

access_token_jwt = jwt.decode(encoded_claims_jwt, pub_key, algorithms=["RS256"], options={"verify_signature": False})

fixes this error: the jwt can be decoded. The jwt.decode signature changed in 2.0.0.

But once that change is made, I get a failure to verify and the groups failed to be read from token:

Apr 06 02:15:14 ip-10-1-1-11 env[9767]: Traceback (most recent call last):
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: File "/apps/consoleme/env/lib64/python3.8/site-packages/tornado/web.py", line 1683, in _execute
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: result = await result
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: File "/apps/consoleme/consoleme/handlers/auth.py", line 11, in prepare
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: await super(AuthHandler, self).prepare()
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: File "/apps/consoleme/consoleme/handlers/base.py", line 170, in prepare
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: return await self.authorization_flow()
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: File "/apps/consoleme/consoleme/handlers/base.py", line 345, in authorization_flow
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: res = await authenticate_user_by_alb_auth(self)
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: File "/apps/consoleme/consoleme/lib/alb_auth.py", line 64, in authenticate_user_by_alb_auth
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: access_token_jwt["aud"],
Apr 06 02:15:14 ip-10-1-1-11 env[9767]: KeyError: 'aud'

There's indeed no aud, nor cid key in my access_token_jwt.

The workaround is to comment out the validate_token but I'd rather not do that long term. (Same with verify_signature: False). My JWT experience isn't great - is Cognito out of spec here, or is this Okta specific JWT code? The validate_token function is imported from okta_jwt.jwt.

Provide PyCharm configuration

SES Email Feature for Consoleme Documentation

Is your feature request related to a problem? Please describe.
I'm wanting to know more about Consoleme's email feature. I've browsed through your documentation and don't see any mention of it. However, when I do searches in your codebase I see usage of Amazon's SES service, which handles emails. I'm rather confused, does Consoleme support email notifications?

Describe the solution you'd like
Can we make it more prominent in the documentation the email features that are part of Consoleme, if any exist?

Describe alternatives you've considered
Besides documentation, it might help to make a brief mention of it in the list of features, assuming of course the email feature works.

Additional context
N/A

unable to create redis container.

environment:

Ubuntu 20.04

While running this command: python scripts/initialize_redis_oss.py

Error Path : consoleme\consoleme\celery\celery_tasks.py

Create issues for tracking API contract

Once endpoints and methods have been determined (#60 and #61), create an issue to establish the initial draft API contract for each endpoint.

Join Slack Channel

Hey @adamgfest , are you able to join this Slack channel? (redacted)

Cross-Account Assume Role policy generation

When a policy request is created for sts:AssumeRole on a role that is owned by the organization, ConsoleMe should generate the cross-account Assume Role Trust policy update automatically.

SQS/SNS Subscription

ConsoleMe should have a standalone UI component for subscribing an SQS queue to an SNS topic across accounts. It should be created in such a way that we can add future support in our Policy Review framework, and allow users to request it.

Create a ConsoleMe policy that is readonly

The current policy under spoke-accounts gives ConsoleMe both the read AND write access that it needs. However, if someone wants to deploy ConsoleMe in read-only, they'd want to use a different policy. Especially alarming is "iam:*"...

The documentation refers to this, but it may be a good idea to:

1. Explain why each permission is needed (per ConsoleMe feature).
2. Generate a version that is read-only, with an explanation of what features wouldn't be available.

Support S3 bucket creation/cloning

Create a feature similar to the IAM role creation/cloning feature. It should allow the creation/cloning of S3 buckets.

Logout Option

As a user, it would be useful to have a Logout button. User groups are sent during the initial login/JWT, so I need to logout and log back in if my group membership has changed.

A logout button next to, or drop down, from the right-hand "User Avatar/Username" would be a good location.

Work around is to create a new private/container browser (or manually delete cookies).

Implement Parliament linting in ConsoleMe's policy editor

Duo's Parliament linter can help identify bad IAM policy patterns. We should integrate this into ConsoleMe and provide hints in Monaco editor to inform end-users about these issues.

Here's a demo with ACE Editor, which ironically, we were using before we started using Monaco: https://parliament.summitroute.com/