circleci-public / circleci-env-inspector Goto Github PK
View Code? Open in Web Editor NEWA NodeJS tool for discovering all your secrets on CircleCI
License: MIT License
A NodeJS tool for discovering all your secrets on CircleCI
License: MIT License
This isn't a big deal, but on some Enterprise Linux distributions, podman is supported over docker. I replaced 'docker' with 'podman' in the run.sh script and it works fine. A nice-to-have would be some form a basic distro or binary check. Thanks! ...or even a simple note in the README.md.
Feature Request
It would be helpful to see the redacted value of environment variables inside a context. Currently I'm only getting the variable name, timestamps, and context id.
Current output.
{
"org-name": {
"contexts": [
{
"name": "context-name",
"id": "xxxxxxxxx-yyyyyyyyyyyy-zzzzzzz",
"url": "<redacted-url>",
"variables": [
{
"variable": "VARIABLE_NAME",
"updated_at": "2020-10-07T22:15:38.317Z",
"context_id": "xxxxxxxxx-yyyyyyyyyyyy-zzzzzzz",
"created_at": "2020-10-07T22:15:38.317Z"
}
]
}
]
}
}
Desired output. Notice the new value
field.
{
"org-name": {
"contexts": [
{
"name": "context-name",
"id": "xxxxxxxxx-yyyyyyyyyyyy-zzzzzzz",
"url": "<redacted-url>",
"variables": [
{
"variable": "VARIABLE_NAME",
"value": "xxxxxxABCDE",
"updated_at": "2020-10-07T22:15:38.317Z",
"context_id": "xxxxxxxxx-yyyyyyyyyyyy-zzzzzzz",
"created_at": "2020-10-07T22:15:38.317Z"
}
]
}
]
}
}
We submitted a ticket as we were trying to pull this data by hand and then found this tool. Unfortunately, it seems to be returning empty arrays for some project's environment variables when in fact, in the UI, there are environment variables attached. I don't really have much information to share as the tool doesn't return any errors. We have in excess of 600 projects with a subset actually being built. We are using Github as our SCM.
Customers may have added AWS credentials through CircleCI's legacy AWS integration.
This can be confirmed when we inspect the .aws
block when hitting the v1.1 API endpoint for project settings as follows:
$ curl -H "Circle-Token: $CIRCLE_TOKEN" https://circleci.com/api/v1.1/project/<vcs>/<org>/<project>/settings | jq ".aws."
{
"keypair" : {
"access_key_id" : "xxxxABCD",
"secret_access_key" : "xxxxdefg"
}
}
To delete these AWS credentials, customers can follow this Support Article:
https://support.circleci.com/hc/en-us/articles/11990015505947-How-to-remove-legacy-AWS-integration-secrets
Since the January 4th security incident, some customers may have also received emails from AWS (titled [Action Required] CircleCI Security Alert to Rotate Access Keys
) from January 7th onwards.
These AWS access keys mentioned in the emails can refer to the credentials added via the legacy AWS integration in CircleCI as well.
Would it be possible for us to update this tool to:
I understand we only look for project environment variables at the project level, currently:
CircleCI-Env-Inspector/src/utils/circleci.ts
Lines 133 to 149 in 027538c
Once I run the "run.sh", prompt for Circle-Ci token is shown, then Github token. I don't have Github integration with Circle-ci. With or without supplying the Github token, it shows my org info and asks to confirm the Org. Then it fails saying "bad credentials". If it is a bad credentials, how is it getting my Org info?
It seems that variables in repoData are sometimes undefined
CircleCI-Env-Inspector/index.js
Lines 136 to 149 in f234e25
file:///project/index.js:150
USER_DATA.projects = repoData.filter((repo) => repo.variables.length > 0);
^
TypeError: Cannot read properties of undefined (reading 'length')
at file:///project/index.js:150:63
at Array.filter (<anonymous>)
at file:///project/index.js:150:31
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Node.js v19.3.0
This is just a "generic help" issue. I'm sort of a part-time admin and I'm happy if I can figure out how to get CircleCI things to work. So the whole "redo quite a lot of what you did over the last few years" is making me scared.
So, I read the threads and I get this script and I download it. And then it asks for a CircleCI API Token which I think I know how to make. Then it wants a GitHub API Token. I mean, I know how to generate a personal token, but what permissions do I need to give it? GitHub has like 50 options!
Are there more complete instructions than just "Follow the prompts?"
Problem: Currently one has to run the script multiple times for each organization/user one has access to. In order to get all the contexts IIUC (the project list seems always the same). Each time it overwrites the previous circleci-data.json
file if one isn't careful.
Request: It could just automatically iterate over all orgs and print one complete json.
There's an error said: TypeError: collaboratorList.map is not a function
🤔
using f234e25
[+] Building 44.0s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 131B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/node:latest 3.2s
=> [1/4] FROM docker.io/library/node@sha256:d5222e1ebd7dd7e9683f47a8861 36.6s
=> => resolve docker.io/library/node@sha256:d5222e1ebd7dd7e9683f47a8861a 0.0s
=> => sha256:d5222e1ebd7dd7e9683f47a8861a4711cb4407a4830 1.21kB / 1.21kB 0.0s
=> => sha256:cdd495c146f343f30e3c9426bd7464d980b7ce4ecc2 2.21kB / 2.21kB 0.0s
=> => sha256:c3e6129b48b69d14c5e7a5605e2b94003fb71aac 53.68MB / 53.68MB 17.5s
=> => sha256:99dfcef0545e5bca269c65c0208385c886321a1e0 10.87MB / 10.87MB 6.8s
=> => sha256:4f35ae1752089da53145990b6db29ec09bcea4decfb 7.53kB / 7.53kB 0.0s
=> => sha256:ef0470f9572d03767b054118fe19e60a9c7e2510168 5.15MB / 5.15MB 2.0s
=> => sha256:940d63caabf45c64562daab3c243c8724dd47bf45 54.68MB / 54.68MB 8.4s
=> => sha256:0f50438bf9d47df748db3fa6d353315cfa3b13 189.80MB / 189.80MB 32.4s
=> => sha256:165e17284706d6e36604eac744ed334496c9a112029 4.21kB / 4.21kB 8.9s
=> => sha256:eda669170e836103532e615512b54a61c243150a 45.88MB / 45.88MB 18.1s
=> => sha256:fa3488cfb3dfbe2adf2082aea18a81d227d5d40209 2.28MB / 2.28MB 18.6s
=> => extracting sha256:c3e6129b48b69d14c5e7a5605e2b94003fb71aac82eac46b 1.0s
=> => sha256:4d1b81970d4011bae7b100df525b963b2fcd3292301c1e 451B / 451B 18.6s
=> => extracting sha256:ef0470f9572d03767b054118fe19e60a9c7e2510168b3a08 0.1s
=> => extracting sha256:99dfcef0545e5bca269c65c0208385c886321a1e02129772 0.1s
=> => extracting sha256:940d63caabf45c64562daab3c243c8724dd47bf45054d902 1.1s
=> => extracting sha256:0f50438bf9d47df748db3fa6d353315cfa3b13c62806c353 2.7s
=> => extracting sha256:165e17284706d6e36604eac744ed334496c9a112029c9bed 0.0s
=> => extracting sha256:eda669170e836103532e615512b54a61c243150a0583d112 0.9s
=> => extracting sha256:fa3488cfb3dfbe2adf2082aea18a81d227d5d40209f63ee5 0.1s
=> => extracting sha256:4d1b81970d4011bae7b100df525b963b2fcd3292301c1eee 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 222.80kB 0.0s
=> [2/4] WORKDIR /project 0.5s
=> [3/4] COPY . . 0.0s
=> [4/4] RUN npm install 3.3s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:f0dc5774ce47be0c356860146dd6b0f66522a7d28a706 0.0s
=> => naming to docker.io/library/cci-env-inspector 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
? Enter your CircleCI API token [hidden]
? Enter your GitHub API token [hidden]
file:///project/index.js:50
choices: await collaboratorList.map((collaboration) => collaboration.name),
^
TypeError: collaboratorList.map is not a function
at file:///project/index.js:50:37
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Node.js v19.3.0
As the title says, it would be great to have the creation date for environment variables in projects.
This is available for contexts already while projects only show the variable name and last characters.
CircleCI has famously never shown the value of secrets in the UI. Will this tool be able to show the values? In the README, it shows the value as "xxxx". Could the README be updated to explain if the value would be hidden or not?
Within our account we have a large number of projects. When the code tries to fetch the project variables, it falls over trying to get every project in parallel as there are too many ports open.
To work around the issue I replaced the code here with a for ... of - https://github.com/CircleCI-Public/CircleCI-Env-Inspector/blob/main/src/index.mts#L106-L123
When running the script:
? Enter your CircleCI API token [hidden]
? Enter your GitHub API token [hidden]
? Select an account ...orgnameredacted...
? Is this an Organization (Not a User)? Yes
Seeing the following error:
file:///project/node_modules/node-fetch/src/body.js:234
const error_ = error instanceof FetchBaseError ? error : new FetchError(`Invalid response body while trying to fetch ${data.url}: ${error.message}`, 'system', error);
^
FetchError: Invalid response body while trying to fetch https://api.github.com/orgs/...redacted.../repos?per_page=100&page=2: Premature close
at consumeBody (file:///project/node_modules/node-fetch/src/body.js:234:60)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Response.text (file:///project/node_modules/node-fetch/src/body.js:158:18)
at async Response.json (file:///project/node_modules/node-fetch/src/body.js:148:16)
at async getGitHubRepos (file:///project/index.js:116:19)
at async file:///project/index.js:134:18 {
type: 'system',
errno: 'ERR_STREAM_PREMATURE_CLOSE',
code: 'ERR_STREAM_PREMATURE_CLOSE',
erroredSysCall: undefined
}
Node.js v19.3.0
When navigating directly to the url mentioned in the above error, I am getting the following response:
[
]
More of a question than an issue.
I can see the inspector is fetching the checkout SSH key(s?) through CircleCI's api call /api/v2/project/{project-slug}/checkout-key
, but I can't see the Additional SSH Keys being fetched, or an API method for it.
Is it possible to return them? The question arises from the need of potentially rotate it!
Thanks
It would be a lot easier to use if instead of having to clone this repo and build the docker image, if the image would be published and the instructions were simply a command like
docker run -it some-circleci-env-inspector-image > circleci-data.json
I'm trying to fetch payload for our organization, and apparently running into a rate limit.
Input output:
? Enter your CircleCI API token [hidden]
? Select a VCS GitHub
? Enter your GitHub API token [hidden]
? Select an account REDACTED
? Is this an Organization (Not a User)? Yes
Error output:
undefined:1
Rate limited.
^
SyntaxError: Unexpected token 'R', "Rate limited." is not valid JSON
at JSON.parse (<anonymous>)
at Response.json (file:///project/node_modules/node-fetch/src/body.js:149:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async fetchWithToken (file:///project/utils.js:45:24)
at async file:///project/index.js:122:40
at async Promise.all (index 137)
at async file:///project/index.js:119:18
Node.js v19.3.0
The project was running fine last week.
Since PR#24 getCircleCIRepos()
is being called in an infinite loop by getPaginatedData()
on the first of 5 accounts/organisations I work on.
In case it's relevant, I'm not an administrator of that organisation.
Right now I cannot find the documentation for CircleCI's /api/private but I'll try and debug the issue myself.
I've added a single debug line:
diff --git a/src/utils/utils.mts b/src/utils/utils.mts
index 740f027..9e48b5e 100644
--- a/src/utils/utils.mts
+++ b/src/utils/utils.mts
@@ -54,6 +54,7 @@ export async function getPaginatedData<T>(
identifier,
pageToken
);
+ console.log(`responseBody.items.length: ${responseBody.items.length}, pageToken: ****${pageToken.slice(-4)}`);
if (response.ok && responseBody.items.length > 0)
items.push(...responseBody.items);
pageToken = responseBody.next_page_token;
The output is:
Running Docker image...
? Enter your CircleCI API token [hidden]
Found 5 accounts.
Fetching data for <withheld> from GitHub... 1/5
Fetching Contexts...
responseBody.items.length: 1, pageToken: ****
responseBody.items.length: 0, pageToken: ****
Fetching Projects...
responseBody.items.length: 49, pageToken: ****
responseBody.items.length: 49, pageToken: ****yMtw
responseBody.items.length: 50, pageToken: ****SpBA
responseBody.items.length: 49, pageToken: ****NGIA
responseBody.items.length: 49, pageToken: ****eYBw
responseBody.items.length: 49, pageToken: ****vswA
responseBody.items.length: 49, pageToken: ****1stw
responseBody.items.length: 50, pageToken: ****QcTw
responseBody.items.length: 49, pageToken: ****U-3A
responseBody.items.length: 49, pageToken: ****o22Q
responseBody.items.length: 49, pageToken: ****FwfI
responseBody.items.length: 49, pageToken: ****p4Qg
responseBody.items.length: 50, pageToken: ****ojsI
responseBody.items.length: 49, pageToken: ****bprA
responseBody.items.length: 49, pageToken: ****yAkA
responseBody.items.length: 49, pageToken: ****ih4Y
responseBody.items.length: 49, pageToken: ****QrGA
responseBody.items.length: 50, pageToken: ****b6LQ
responseBody.items.length: 49, pageToken: ****YreQ
Output ends with:
]
},
... 58 more items
]
}
I have a large number of repositories and have several stale contexts from old projects that were not named well by predecessors. It would be very useful to have some . . . context. . . when troubleshooting contexts. ie What project(s) user(s) read a particular value most recently?
https://github.com/CircleCI-Public/CircleCI-Env-Inspector/blob/main/index.js#L127
repo.variables?.length > 0)
still returns an error when repo.variables
doesn't exist.
Fetching Contexts...
Fetching Projects...
Fetching Project Variables...
file:///project/node_modules/node-fetch/src/index.js:108
reject(new FetchError(`request to ${request.url} failed, reason: ${error.message}`, 'system', error));
^
FetchError: request to https://circleci.com/api/v2/project/gh/{redacted}/envvar?page-token= failed, reason: getaddrinfo EAI_AGAIN circleci.com
Any idea why this is happening? And can we exclude repos that cause problems?
Error below occurs while running run.sh
. A number of account succeed before this error occurs. Is there some way to exclude the account causing the error? Thanks.
...
Fetching data for percero from GitHub... 8/10
Fetching Contexts...
Fetching Projects...
Fetching Project Variables...
file:///project/node_modules/node-fetch/src/index.js:108
reject(new FetchError(`request to ${request.url} failed, reason: ${error.message}`, 'system', error));
^
FetchError: request to https://circleci.com/api/v2/project/gh/percero/com.percero.platform.sdk.as--as-percero/checkout-key failed, reason: getaddrinfo EAI_AGAIN circleci.com
at ClientRequest.<anonymous> (file:///project/node_modules/node-fetch/src/index.js:108:11)
at ClientRequest.emit (node:events:513:28)
at TLSSocket.socketErrorListener (node:_http_client:496:9)
at TLSSocket.emit (node:events:513:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'EAI_AGAIN',
code: 'EAI_AGAIN',
erroredSysCall: 'getaddrinfo'
}
Node.js v18.13.0
Extracting data from container...
Error: No such container:path: circleci-env-inspector:/project/circleci-data.json
Cleaning up...
Done.
First, thank you for this tool @KyleTryon, it's great! 👍
With recent code changes, I am kicked by CircleCI-Env-Inspector
with "No user name returned":
$ ./run.sh
Building Docker image...
Sending build context to Docker daemon 1.575MB
Step 1/6 : FROM node:18
---> b68a472583ef
Step 2/6 : WORKDIR /project
---> Using cache
---> 75c4382e6ff4
Step 3/6 : COPY . .
---> 117e651dd0a3
Step 4/6 : RUN npm install && npm run build
---> Running in 62841b1f4c2f
added 559 packages, and audited 790 packages in 7s
93 packages are looking for funding
run `npm fund` for details
1 moderate severity vulnerability
To address all issues, run:
npm audit fix
Run `npm audit` for details.
npm notice
npm notice New major version of npm available! 8.19.3 -> 9.4.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.4.1>
npm notice Run `npm install -g [email protected]` to update!
npm notice
> [email protected] build
> rimraf ./dist && tsup src/index.ts --format esm
CLI Building entry: src/index.ts
CLI Using tsconfig: tsconfig.json
CLI tsup v6.5.0
CLI Using tsup config: /project/tsup.config.ts
CLI Target: es2022
CLI Cleaning output folder
ESM Build start
ESM dist/index.js 10.78 KB
ESM dist/index.js.map 24.30 KB
ESM ⚡️ Build success in 11ms
Removing intermediate container 62841b1f4c2f
---> 10dd24e44b40
Step 5/6 : ENTRYPOINT ["node"]
---> Running in e160b991ae17
Removing intermediate container e160b991ae17
---> cd824b927cf1
Step 6/6 : CMD ["./dist/index.js"]
---> Running in 69d0d6fe6915
Removing intermediate container 69d0d6fe6915
---> fd68ac65d18a
Successfully built fd68ac65d18a
Successfully tagged cci-env-inspector:latest
Running Docker image...
? Enter your CircleCI API token [hidden]
No user name returned
Extracting data from container...
Cleaning up...
Done.
I can get it working just by resetting to older version:
$ git reset --hard 027538c
HEAD is now at 027538c feat: introduce ts-jest and test example (#48)
Then run with same token:
$ ./run.sh
Building Docker image...
Sending build context to Docker daemon 1.561MB
Step 1/6 : FROM node:18
---> b68a472583ef
Step 2/6 : WORKDIR /project
---> Using cache
---> 75c4382e6ff4
Step 3/6 : COPY . .
---> f10a212604e2
Step 4/6 : RUN npm install && npm run build
---> Running in c17948d150f0
added 551 packages, and audited 782 packages in 7s
92 packages are looking for funding
run `npm fund` for details
1 moderate severity vulnerability
To address all issues, run:
npm audit fix
Run `npm audit` for details.
npm notice
npm notice New major version of npm available! 8.19.3 -> 9.4.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.4.1>
npm notice Run `npm install -g [email protected]` to update!
npm notice
> [email protected] build
> rimraf ./dist && tsup src/index.ts --format esm
CLI Building entry: src/index.ts
CLI Using tsconfig: tsconfig.json
CLI tsup v6.5.0
CLI Using tsup config: /project/tsup.config.ts
CLI Target: es2022
CLI Cleaning output folder
ESM Build start
ESM dist/index.js 7.54 KB
ESM dist/index.js.map 16.69 KB
ESM ⚡️ Build success in 11ms
Removing intermediate container c17948d150f0
---> 58190ecbdc31
Step 5/6 : ENTRYPOINT ["node"]
---> Running in de8715c08d38
Removing intermediate container de8715c08d38
---> e071a97ef5de
Step 6/6 : CMD ["./dist/index.js"]
---> Running in fda04859e004
Removing intermediate container fda04859e004
---> 066f9185e145
Successfully built 066f9185e145
Successfully tagged cci-env-inspector:latest
Running Docker image...
? Enter your CircleCI API token [hidden]
Found 6 accounts.
Fetching data for **** from GitHub... 1/6
Fetching Contexts...
Fetching Projects...
Fetching Project Variables...
etc...
Any idea what happens? 🤔
Thank you in advance
When I run this (after generating a new API token), I am getting a dump of all the contexts for the organisation as expected, but the projects array is empty whereas I expected it to list at least the 8 organisation projects followed by the API token holder.
Getting Contexts Data...
Getting Projects Data...
Getting Projects Variables...
Log created at circleci-data.json
Extracting data from container...
Cleaning up...
Done.
Output:
{
...
"projects": [],
"unavailable": []
}
I have created API tokens in a project/repo with both admin level privileges and readonly, all producing this output upon running the docker image.
Running Docker image...
? Enter your CircleCI API token [hidden]
Error: Couldn't fetch accounts. Please open an issue. [
Response {
size: 0,
[Symbol(Body internals)]: {
body: [PassThrough],
stream: [PassThrough],
boundary: null,
disturbed: true,
error: null
},
[Symbol(Response internals)]: {
type: 'default',
url: 'https://circleci.com/api/v2/me/collaborations',
status: 401,
statusText: 'Unauthorized',
headers: [Object],
counter: 0,
highWaterMark: 16384
}
}
]
Extracting data from container...
Error: No such container:path: circleci-env-inspector:/project/circleci-data.json
Cleaning up...
Done.
I didn't realize that this existed, so I started circleci-audit
today. It uses oclif
and is MIT licensed, in case you want to contribute / extend it instead of, or in addition to this tool: https://github.com/blimmer/circleci-audit/
Hi there,
I ran this earlier and noticed that it is not including all projects (and their respective project level variables). I'm unable to determine why this is. I initially suspected it was because we disconnected from of the projects in Circle CI from GitHub as we are moving to GitHub Actions but haven't found a pattern to confirm this. I'm not sure if the issue I'm facing is specific to the configuration of our account or if others may be facing the same issue without noticing.
Not sure if its related or not: CircleCI-Archived/api-preview-docs#97 (comment)
Getting this error after inputs
undefined:1
Rate limited.
^
SyntaxError: Unexpected token 'R', "Rate limited." is not valid JSON
at JSON.parse (<anonymous>)
at Response.json (file:///project/node_modules/node-fetch/src/body.js:149:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async file:///project/index.js:139:21
at async Promise.all (index 252)
at async file:///project/index.js:136:18
Node.js v19.3.0
When running this against an organization, this produced a node error.
...
? Enter your CircleCI API token [hidden]
? Enter your GitHub API token [hidden]
? Select an account fooOrganization
? Is this an Organization (Not a User)? Yes
file:///project/index.js:149
USER_DATA.projects = repoData.filter((repo) => repo.variables.length > 0);
^
TypeError: Cannot read properties of undefined (reading 'length')
at file:///project/index.js:149:63
at Array.filter (<anonymous>)
at file:///project/index.js:149:31
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Node.js v19.3.0
Hi, I'm receiving a permission denied error with below output:
$ docker run -it --rm cci-env-inspector
? Enter your CircleCI API token [hidden]
? Select a VCS GitHub
? Enter your GitHub API token [hidden]
? Select an account [hidden]
? Is this an Organization (Not a User)? Yes
Failed to get project variables with the following error:
[ { message: 'Permission denied' } ]
For CircleCI API token - i've created a personal API token which has read + write scope.
For Github token - i've created a classic Personal access token which has repo
and workflow
scope.
The script was able to list all the organisations correctly, but it errors immediately after that...
Some users can use the same account name for bitbucket and github, however, the account id that is used is based off the first account name match, which would result in the wrong id being used access repositories
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.