Comments (12)
Reopening this issue in light of @Clete2 's comment.
Current code in master
branch:
Current algorithm is:
- Pull and collate a list of Lambda functions from each region
- For each possible "destination" Node:
- Verify that it's an IAM Role that can be assumed by
lambda.amazonaws.com
(otherwise continue) - For each possible "source" Node:
- Check and store info on if the source node can pass the destination node to Lambda (simulation)
- Check if the source node can create a Lambda function: if so, and can pass the role, create an Edge (simulation)
- Create a copy of the list of Lambda functions and append data on if the source node can edit the function's code or function's configuration (two simulations)
- For each item in the list-copy:
- Check if the function's execution role is the destination role (otherwise continue)
- If the source node can edit the function's code (per earlier check) then create an Edge and break
- For each item in the list-copy:
- If the source node can edit the function's code, edit the function's configuration, and pass the destination role, then create an Edge and break
- Verify that it's an IAM Role that can be assumed by
The time-consuming stuff is probably the simulations. The number of those we currently do (worst-case) is:
LR = number of roles that Lambda can assume
LF = number of Lambda functions
N = number of IAM Users and Roles in the account
LR * (2N + (2N * LF))
Meaning an account with 100 users and roles, 8 of which are assumable roles for Lambda, and 1000 Lambda functions will result in ~1.6M simulation calls. If our simulator can process 2 simulations/sec (need to benchmark to see what this actually looks like, I could see the ReadOnlyAccess
policy being slow to process for PMapper) then that'd be 9.26 days to finish.
A couple things I'll look at:
- Adding a check to see if "source" can call the edit/reconfigure Lambda operations for any function (
*
) to short-circuit the2N * LF
checks - Skipping edit/reconfigure edges if the caller can do the "create a function and pass the role" approach
- Benchmarking simulation rates across different actions/resources/policies
- Adding an LRU cache at the policy-layer (i.e. effect/action/resource/conditions -> policy has matching statement) to reduce the impact of larger policies during simulation
from pmapper.
All done!
The png
command failed because dot
was not on the path.
The graphml
one worked and I'm playing around with it in Gephi
. Thanks very much for your support.
from pmapper.
Hi @beacomni ,
A couple questions to start:
- Is this the
master
branch,v1.1.0-dev
branch, or PyPi installation code? - What scale of Lambda functions and Lambda Execution Roles (IAM Roles that trust
lambda.amazonaws.com
are in the account? A power of 10 (1, 10, 100, 1000, etc.) is fine.
I suspect, based on the last line of output, that it's taking a while to chew through the Lambda functions in the account. I have a few suggestions in the meantime:
- Install and try the code from
v1.1.0-dev
if you aren't already. - Run with the
--debug
parameter to get debug output (you'll likely see a flurry of output during this stage). - Create a script that uses the
principalmapper
library to gather the graph together, but exclude thelambda
service.
from pmapper.
Hi @beacomni : wanted to check in. Do you have any follow-up questions? Were the previous suggestions helpful?
from pmapper.
I'm having the same issue, that it hangs on generating edges for Lambda:
2021-08-24 11:37:36+0000 | Found new edge: role/<snip> can update the trust document to access role/<snip>
2021-08-24 11:37:36+0000 | Pulling data on Lambda functions
2021-08-24 11:37:51+0000 | Generating Edges based on Lambda data.
It's been on that line since 8/24. The date is now 8/30. We have a big account with 1780 Lambdas currently. On a smaller account this step took a while but we extrapolated that analyzing this many Lambdas should have only taken a few hours. Yet, a single CPU has been pegged for 6 days.
We are on version 1.1.3 from pip. Adding debug doesn't give us any additional information.
I am running pmapper without Lambda now, but that removes a large benefit of pmapper. @ncc-erik-steringer - Do you have any advice?
from pmapper.
@ncc-erik-steringer thanks for the comment. I'll adapt your math to my account.
LR * (2N + (2N * LF))
1780 * (2 * 2872 + (2 * 2872 * 1780)) <-- I assume the # of assumable Lambda roles ~= the number of Lambdas. We generally develop one role per Lambda
1780 * (5744 + (5744 * 1780))
1780 * (5744 + 10224320)
1780 * 10230064
18,209,513,920
18,209,513,920 / 2 simulations per second = 9,104,756,960 seconds
9,104,756,960 / 60 = 151,745,949.33333334 minutes
151,745,949.33333334 / 60 = 2,529,099.1555555556 hours
2,529,099.1555555556 / 24 = 105,379.1314814815 days
105,379.1314814815 / 365.2425 = 288.5182624735 years
Yes, I know our account is way too big.
One thing I am worried about is that my access token expires every hour, and I am worried once it's done calculating Lambda edges, and we're all 6 feet under, it'll move on to the next step and try to call AWS APIs and fail due to an expired token.
from pmapper.
Just pushed 725c05d to the v1.1.4-dev
branch with some of the work to help cut down on Lambda authorization simulation calls. I suspect it isn't going to help your case out very much but it's worth a try in the meantime.
Looking at the caching options, it turns out it'll take some breaking changes to add some LRU caches like I wanted. Maybe for v2.0.0 some day.
For IAM Roles, you may be able to configure a profile with the AWS CLI using credential_process
(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html) that might refresh? Otherwise you may need to write some code and use the botocore library's RefreshableCredentials class https://github.com/boto/botocore/blob/4927d3e3baa6a00226a1f017b638807beeb613a0/botocore/credentials.py#L368 .
from pmapper.
Thanks for the new code. I have pulled it down and started running it on a c5.large instance. I'll leave this running for a week or so and let you know if it finishes or if I kill it.
For passwords I broke down and created a read-only user with static creds.
from pmapper.
@ncc-erik-steringer it finished! On Sept 17th.
Since 9/17 I have been running the visualize command and it's still chugging.
from pmapper.
So roughly 16 days to complete? That's actually faster than I anticipated even with the changes. Thank you for being able to put PMapper through the stress test.
For visualization, PMapper basically sends the data to graphviz and waits for it to do the job. However, you may be able to output to a different filetype (.graphml
) and use another renderer that works faster (Gephi or Cytoscape). That's controlled with the --filetype
param.
from pmapper.
Thanks for the tip. I just started the following command:
pmapper --account <myacct> visualize --filetype graphml --only-privesc
The png
command is still running since the 17th and I did not kill it. Will update once I make some progress.
from pmapper.
Hey @ncc-erik-steringer,
- Skipping edit/reconfigure edges if the caller can do the "create a function and pass the role" approach
I'm having trouble understanding why have we skipped this edge creation which can indicate another possible privilege escalation path. I get we are trying to reduce time complexity but skipping checks will reduce the detection of the attack surface, don't you think?
Or am I missing something here, will you please care to explain?
from pmapper.
Related Issues (20)
- PMapper 1.1.5 builds edges that include role/AWSServiceRoleForSupport when performing authorization checks HOT 10
- Terraform Plans HOT 2
- Graph Deletion HOT 1
- Local user who can assume an admin role not in graph HOT 6
- Stuck at Generating Edges based on lambda data HOT 1
- MFA requirements in roles can lead to misleading results
- can_privesc() method only returns one edge_list ?
- Traceback when doing connected query for role that does not exist
- FileNotFoundError in graph_cli
- Exception When Policy is Only Used as Permission Boundary HOT 1
- Permission boundaries not considered when querying
- Python 3.10 fails to run HOT 1
- Does not run in 3.11 due to mapping import error HOT 1
- iam:ListAccessKeys denied exception in gathering.py
- Stack trace on incorrect PMAPPER_STORAGE environment variable
- Stack trace on missing credentials
- Crash while scanning principals that use deprecated permission policies HOT 3
- Performance issues scanning large accounts HOT 8
- AWS Policy with minimal permissions
- Collections Module issue in Python 3.10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pmapper.