snyk / snyk-api-import Goto Github PK

⚡️Snyk API powered import tool to help you automate & monitor a large scale import into Snyk organizations. Designed for onboarding with a built in queue & retries 📈

Home Page: https://snyk.io

License: Other

JavaScript 0.07% TypeScript 99.93%

snyk-apis snyk snyk-tooling

snyk-api-import's Introduction

This repository is in maintenance mode, no new features are being developed. Bug & security fixes will continue to be delivered. Open source contributions are welcome for small features & fixes (no breaking changes)

Snyk helps you find, fix and monitor for known vulnerabilities in your dependencies, both on an ad hoc basis and as part of your CI (Build) system.

snyk-api-import

Snyk API project importer. This script is intended to help import projects into Snyk with a controlled pace utilizing available Snyk APIs.

What does it offer?

rate limiting handling - the script will pace requests to avoid rate limiting from Github/Gitlab/Bitbucket etc and to provide a stable import.
queue - requests to Snyk are queued to reduce failures.
retries - the script will kick off an import in batches, wait for completion and then keep going. Any failed requests will be retried before they are considered a failure and logged.

If you need to adjust concurrency you can stop the script, change the concurrency variable and start again. It will skip previous repos/targets that have been requested for import.

Installation

snyk-api-import CLI can be installed through multiple channels.

Standalone executables (macOS, Linux, Windows)

Use GitHub Releases to download a standalone executable of Snyk CLI for your platform.

More installation methods

Install with npm or Yarn

Snyk snyk-api-import CLI is available as an npm package. If you have Node.js installed locally, you can install it by running:

npm install snyk-api-import@latest -g

or if you are using Yarn:

yarn global add snyk-api-import

Usage

By default the import command will run if no command specified.

import - kick off a an API powered import of repos/targets into existing Snyk orgs defined in import configuration file. 100% support available for all project types supported via Import API.
help - show help & all available commands and their options
orgs:data - util generate data required to create Orgs via API.
orgs:create - util to create the Orgs in Snyk based on data file generated with orgs:data command.
import:data - util to generate data required to kick off an import.
list:imported - util to generate data to help skip previously imported targets during import.

The logs can be explored using Bunyan CLI

FAQ

Error: ENFILE: file table overflow, open or Error: EMFILE, too many open files

If you see these errors then you may need to bump ulimit to allow more open file operations. In order to keep the operations more performant tool logs as soon as it is convenient rather than wait until very end of a loop and log a huge data structure. This means depending on number of concurrent imports set the tool may exceed the system default ulimit.

Some of these resources may help you bump the ulimit:

ERROR: HttpError: request to https://github.private.com failed, reason: self signed certificate in certificate chain

If your Github / Gitlab / Bitbucket / Azure is using a self signed certificate, you can configure snyk-api-import to use this certificate when calling the HTTPS APIs.

export NODE_EXTRA_CA_CERTS=./path-to-ca

Does this work with brokered integrations?

Yes. because we reuse the existing integration with your SCM (git) repository to perform the imports, the brokered connection will be used when configured.

What is supported for import command?

snyk-api-import supports 100% of the same integration types and project sources as the Import API documentation. If an example is not in the docs for your use case please see the API documentation

snyk-api-import's People

Contributors

Stargazers

Watchers

snyk-api-import's Issues

assets for latest release is empty[🐛]

We have a Snyk provided snyk mapper container that includes a script to install_snyk_tools.sh. It uses https://api.github.com/repos/snyk-tech-services/snyk-api-import/releases/latest to retrieve a URL that is used for a sha256 comparison before moving snyk-api-import-linux to the container's /usr/local/bin/snyk-api-import, then chmod +x it. Because the assets[] section for the latest release is empty, it can not determine the URL to retrieve snyk-api-import-linux.sha256 for that comparison.

Debug log

N/A

Screenshots

N/A

[🐛] Error in v1.70.2 - TypeError: Converting circular structure to JSON

v1.70.2

DEBUG=snyk* snyk-api-import-macos list:imported --integrationType=gitlab --groupId=${SNYK_GROUP_ID}

Expected behaviour

Generate a list of the previously imported repos

Actual behaviour

Error (debug log below)

Steps to reproduce

Perform the steps in: https://github.com/snyk-tech-services/snyk-api-import/blob/master/docs/mirror-gitlab.md#re-importing-new-repos--orgs-only-while-mirroring

Debug log

Generate the previously imported log to skip all previously imported repos...
  snyk:generate-data-script ℹ️  Options: {"_":["list:imported"],"integrationType":"gitlab","integration-type":"gitlab","groupId":"XXX","group-id":"XXX","$0":"snyk-api-import-macos"} +0ms
  snyk:api-group Fetching page: 1 +0ms
  snyk:generate-data-script Failed to list all imported targets in Snyk.
  snyk:generate-data-script TypeError: Converting circular structure to JSON +368ms
ERROR! Failed to list imported targets in Snyk. Try running with `DEBUG=snyk* <command> for more info`.
ERROR: Converting circular structure to JSON
Importing repos as snyk projects...

[🙏] Documentation for integrating with Azure-Repos

Describe the user need
Provide a reference or documentation on how to utilize this tool with Azure Repos.

Describe expected behaviour

Provide a method to generate the json file that is supported via

snyk-api-import import:data

It appears that mirror from Azure Repos is supported, but I do not see a format or a method to generate the data to the import the project to Snyk.
Additional context

The goal and intent is mirror projects from Azure Repos and import those into Snyk.

[🙏] Support for build.gradle.kts

Describe the user need
I want Snyk to be able to import git repos that use build.gradle.kts instead of only supporting build.gradle.

Describe expected behaviour

When importing from a git repository that uses build.gradle.kts, Snyk should be able to automatically detect and analyze dependencies the same way it would for a project using build.gradle.

Additional context

From https://docs.snyk.io/getting-started/supported-languages-and-frameworks/java-and-kotlin

Kotlin: build.gradle.kts files are not currently supported in Git.

This appears to be a known limitation, but I'm not sure if this is on the roadmap or being tracked anywhere else. If so, please point me in that direction.

DEBUG=snyk* fails to provide more error info in case of generate-data-script failure[🐛]

node -v:
npm -v:
OS:
docker container running in azure kubernetes - python:3.10-alpine
(I have run the same shell commands with the same variables in my local terminal on my m1 macbook macos 13.4.1 (with the macos binary), and it works as expected)
Command run:
"DEBUG=snyk* snyk-api-import-alpine import:data --orgsData=/home/app/snyk-created-orgs.json --source=github-enterprise". Variables SNYK_TOKEN, GITHUB_TOKEN, SNYK_ORG_ID, SNYK_LOG_PATH, and SNYK_IMPORT_PATH are set.

Expected behaviour

First. I really appreciate this tool and I think it very thorough and useful.
I would expect the error output to indicate the cause of the generate-data-script failure. I have ensured that the snyk-created-orgs.json is present, all env variables are set, and my container has read and write access to the directory. I would expect the list-repos-script to fetch 3 repos from the api (confirmed using the same .json in my local terminal). I am using the same snyk-created-orgs.json that works as expected running in my local terminal. *Note: I am building the docker image with linux amd64 emulation to ensure it works in aks

Actual behaviour

The import:data command fails to generate data. Using DEBUG does not indicate a cause.
2023-07-14T15:43:11.822Z snyk:generate-targets-data Processing
No sourceUrl provided for Github Enterprise source, defaulting to https://api.github.com
2023-07-14T15:43:11.829Z snyk:list-repos-script Fetching all repos data for org:
2023-07-14T15:43:11.829Z snyk:list-repos-script Fetching page: 1
2023-07-14T15:43:12.062Z snyk:list-repos-script Received # repos from API for page 1: 0
2023-07-14T15:43:12.062Z snyk:generate-data-script Failed to create organizations.
ERROR! Failed to generate data. Try running with DEBUG=snyk* <command> for more info.
ERROR: Error: No targets could be generated. Check the error output & try again.
ERROR! Failed to generate data. Try running with DEBUG=snyk* <command> for more info.
ERROR:root:The file at /home/app/github-enterprise-import-targets.json does not exist.

Steps to reproduce

I have successfully run these exact commands in my local (with the appropriate binary). I am trying to containerize them so that I may run them in a kubernetes cronjob and keep everything in sync automatically. Here is the dockerfile and python script for shell commands.

My dockerfile:

FROM python:3.10-alpine

Install curl

RUN apk add --no-cache curl

Download and install the snyk-api-import tool

RUN curl -L https://github.com/snyk-tech-services/snyk-api-import/releases/download/v2.19.5/snyk-api-import-alpine -o /usr/local/bin/snyk-api-import-alpine &&
chmod +x /usr/local/bin/snyk-api-import-alpine

Create working folder

RUN mkdir -p /home/app && chmod 777 /home/app

Set working folder

WORKDIR /home/app

Copy the python script to the Docker image

COPY snyk-api-import.py .

Run the command on container startup

CMD ["python", "/home/app/snyk-api-import.py"]

My python script:

import subprocess
import os
import logging
import shutil

Setup logging

logging.basicConfig(level=logging.INFO)

Copy json file to root directory

source_path = "/config/snyk-created-orgs.json"
destination_path = "/home/app/snyk-created-orgs.json"
shutil.copy(source_path, destination_path)

Load the secrets and Snyk organization ID

github_token = os.getenv('GITHUB_TOKEN')
snyk_token = os.getenv('SNYK_TOKEN')
snyk_org_id = os.getenv('SNYK_ORG_ID')

if github_token is None or snyk_token is None or snyk_org_id is None:
logging.error("Environment variables are not set correctly. Please check your GITHUB_TOKEN, SNYK_TOKEN, and SNYK_ORG_ID.")
exit(1)

Set environment variables

os.environ['GITHUB_TOKEN'] = github_token
os.environ['SNYK_TOKEN'] = snyk_token
os.environ['SNYK_ORG_ID'] = snyk_org_id
os.environ['SNYK_LOG_PATH'] = "/home/app"

Check if the file exists

file_path = "/home/app/snyk-created-orgs.json"
if not os.path.exists(file_path):
logging.error(f"The file at {file_path} does not exist.")
exit(1)
if os.path.exists(file_path):
logging.info(f"{file_path} is present")

Import:data command

import_data_command = "DEBUG=snyk* snyk-api-import-alpine import:data --orgsData=/home/app/snyk-created-orgs.json --source=github-enterprise"
subprocess.run(import_data_command, shell=True)

os.environ['SNYK_IMPORT_PATH'] = "/home/app/github-enterprise-import-targets.json"

Check for presence of github-enterprise-import-targets.json

targets_json_path = "/home/app/github-enterprise-import-targets.json"
if not os.path.exists(targets_json_path):
logging.error(f"The file at {targets_json_path} does not exist.")
exit(1)
if os.path.exists(targets_json_path):
logging.info(f"{targets_json_path} is present")

Import command

import_command = "snyk-api-import-alpine import"
subprocess.run(import_command, shell=True)

Sync command

sync_command = f"snyk-api-import-alpine sync --orgPublicId={snyk_org_id} --source=github-enterprise"
subprocess.run(sync_command, shell=True)

Debug log

INFO:root:/home/app/snyk-created-orgs.json is present
2023-07-14T15:43:11.813Z snyk:generate-data-script ℹ️ Options: {"_":["import:data"],"orgsData":"/home/app/snyk-created-orgs.json","orgs-data":"/home/app/snyk-created-orgs.json","source":"github-enterprise","$0":"/usr/local/bin/snyk-api-import-alpine"}
2023-07-14T15:43:11.822Z snyk:generate-targets-data Processing
No sourceUrl provided for Github Enterprise source, defaulting to https://api.github.com
2023-07-14T15:43:11.829Z snyk:list-repos-script Fetching all repos data for org:
2023-07-14T15:43:11.829Z snyk:list-repos-script Fetching page: 1
2023-07-14T15:43:12.062Z snyk:list-repos-script Received # repos from API for page 1: 0
2023-07-14T15:43:12.062Z snyk:generate-data-script Failed to create organizations.
ERROR! Failed to generate data. Try running with DEBUG=snyk* <command> for more info.
ERROR: Error: No targets could be generated. Check the error output & try again.
ERROR! Failed to generate data. Try running with DEBUG=snyk* <command> for more info.
ERROR: Error: No targets could be generated. Check the error output & try again.
ERROR:root:The file at /home/app/github-enterprise-import-targets.json does not exist.

Screenshots

If applicable, add screenshots to help explain your problem.

[🐛] Gitlab Groups -> Snyk Orgs fails

Expected behaviour

Orgs created fit into Snyk Platform limitations and requirements.

Actual behaviour

Orgs are not created because the 60 character limit is blocking the functionality.

Steps to reproduce

Use Gitlab SCM
Create Groups with nested SubGroups
Continue until the 60 character threshold is passed
Run the tool.

Debug log

If applicable, please add DEBUG=*snyk* <command here> before your command and include the output here **ensuring to remove any sensitive/personal details or tokens.

Screenshots

The automated release is failing 🚨

🚨 The automated release from the `master` branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

No npm token specified.

An npm token must be created and set in the NPM_TOKEN environment variable on your CI environment.

Please make sure to create an npm token and to set it in the NPM_TOKEN environment variable on your CI environment. The token must allow to publish to the registry https://registry.npmjs.org/.

Good luck with your project ✨

Your semantic-release bot 📦🚀

The automated release is failing 🚨

🚨 The automated release from the `master` branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can fix this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

If you are not sure how to resolve this, here are some links that can help you:

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

Cannot push to the Git repository.

semantic-release cannot push the version tag to the branch master on the remote Git repository with URL https://github.com/snyk-tech-services/snyk-api-import.

This can be caused by:

a misconfiguration of the repositoryUrl option
the repository being unavailable
or missing push permission for the user configured via the Git credentials on your CI environment

Good luck with your project ✨

Your semantic-release bot 📦🚀

[🙏] Optionally allow Snyk project deletion instead of deactivation

Describe the user need
When running snyk-api-import, archived/renamed/deleted packages from the SCM provider are deactivated in Snyk. While this makes sense as a the default behavior, having a --delete argument to delete the projects instead of deactivating them would be extremely helpful.

Describe expected behaviour
Adding an optional --delete argument to snyk-api-import sync should delete projects (and targets, if applicable) instead of deactivate them.

Additional context
We are currently running a secondary custom automation script to delete the deactivated projects. Having the functionality built-in would be much more efficient.

[🐛] Rename files with colons : in them

node -v: v16.13.1
npm -v: 8.5.1
OS: Microsoft Windows 10 Version 21H2
Command run: git clone https://github.com/snyk-tech-services/snyk-api-import.git

Expected behaviour

Repository should be usable on Windows platforms.

Actual behaviour

Repository cannot be cloned in a Windows operating system. Colon is a reserved file system character (e.g. C:) and results in an error when the repository is cloned via git clone. This has the effect of making it difficult, if not impossible, to use snyk-api-import from source.

Steps to reproduce

On a Windows machine, run git clone https://github.com/snyk-tech-services/snyk-api-import.git

Debug log

Cloning into 'snyk-api-import'...
remote: Enumerating objects: 3527, done.
remote: Counting objects: 100% (883/883), done.
remote: Compressing objects: 100% (316/316), done.
remote: Total 3527 (delta 689), reused 655 (delta 552), pack-reused 2644
Receiving objects: 100% (3527/3527), 925.47 KiB | 2.30 MiB/s, done.
Resolving deltas: 100% (2258/2258), done.
error: invalid path 'src/cmds/import:data.ts'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

[🙏] Add a switch/method to enable a custom mapping instead of Github Org == Snyk Org

Describe the user need
Some customers only have 1 Github Org with many repos within this Org and do access control by using Github Teams to control access to repos within this single org. Currently, snyk-api-import will automatically map a Github org to a Snyk Org but any customer that wants to create a Snyk Org per Team (or Business Unit) they can do so, but during the import:data phase, they will need to manually map the OrgID and IntegrationID underneath each repo within json generated by snyk-api-import which could be time consuming. The ask here is if the customer has some mapping of Snyk org to Repo (coudl they use some type of config file/mapping file) to automatically assign the repos the right OrgID and IntegrationID

If you have an idea how you would like this to behave please share in as much detail as possible.

Potentially if a customer had a list of Team:Repo as a csv, if we can somehow make a utility to first map the Team:Repo to Orgslug:Repo (As Team would be the orgName) -> OrgID:Repo (Where we convert the OrgSlug into OrgID) -> OrgID + IntegrationID:Repo (Where based on the OrgID we can also pull the IntegrationID) -> somehow append this OrgID + IntegrationID into the json generated by snyk-api-import then customers would have a nice final json of Repo + OrgID + IntegrationID that is generated nicely without manual json manipulation

Add any other context or screenshots about the feature request here.

[🐛]Compatibility of Axios with Proxy

axios/axios#3384

Based on the above github issue, it looks like it may make sense to add axios support for proxy (which snyk-api-import) uses for customers that need to use this tool behind a proxy.

[🐛] When importing projets via bitbucket cloud

When importing via bitbucket cloud, this works if you mirror the github example json. The time it fails, is if you have "fork":<true/false> as you may see if you use the snyk-api-import to clone github orgs and then just modify the json by changing the integration ID to switch it from Github to Bitbucket cloud integration ID.

As soon as I removed the "fork":<true/false> the import worked as expected. (and the error complaining about invalid paramater: name went away)

,"errorData":{"errorMessage":"400 - {"ok":false,"code":400,"message":"Invalid parameter format.","name":"invalidParamData"}","name":"Unkno
wn","code":400,"requestId":"7d11d88d-4e25-4b94-8e65-d3688e86220c"},"msg":"Failed to import target","time":"2022-01-18T16:47:30.621Z","v":0}

This was the import error log I got and below is the json I used.

{
"targets":
[
{
"target":{
"fork": false,
"name": "",
"owner": "",
"branch": ""
},
"integrationId": "",
"orgId": ""
}
]
}

[🙏] add a -v switch for snyk-api-import to show version of tool

Describe the user need
I want Snyk to be able to tell me the version of snyk-api-import tool I'm using without having to compare against hash/checksum :)

Describe expected behaviour

snyk-api-import -v switch would ideally return the version of the tool being used

Additional context

Just something that would be nice to have :)

Convert Epoch time in Output to datetime string

Currently the debug output shows time in Epoch format see below:

snyk:import-projects-script Loaded 7665 targets to import 1592806773188 +0ms

Might be more useful to present times in a datetime string rather than a unix timestamp

[🐛] Add Token Warning to README

node -v:
npm -v:
OS: OSX
Command run: python sny_scm_refresh.py --sca on --container on --iac on --code on

Expected behaviour

The script imports new projects from repositories in GitHub

Actual behaviour

The script returns 401 HTTP Responses when reaching out to Snyk to request locationUrl's/make import requests. I can only confirm this for targets in GitHub, as I don't have any targets in any other repository services.

Steps to reproduce

Use a Snyk token from a provisioned service account in Group Overview Page -> Cog Wheel in Top Right -> Service Accounts with admin privileges
Run the import script with import-targets.json referencing GitHub repositories.

It does run correctly with an API Token, similar to the scm-refresh-script. It may be worth putting a warning like they do in the installation section:

Make sure to use a user API Token that has access to the Snyk Orgs you need to process with the script. A service account will not work for GitHub, which is the only SCM currently supported at this

Debug log

snyk ERROR: ApiAuthenticationError: Error: Request failed with status code 401
    at Object.makeSnykRequest (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/request.ts:77:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.RequestsManager._makeRequest [as next] (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/requestManager.ts:100:24) {
  data: {
    code: 401,
    message: 'Invalid credentials',
    errorRef: 'd4f08495-63a9-4d1a-a6a1-ba78ad4698b6'
  }
} +0ms
  snyk ERROR: ApiAuthenticationError: Error: Request failed with status code 401
    at Object.makeSnykRequest (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/request.ts:77:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.RequestsManager._makeRequest [as next] (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/requestManager.ts:100:24) {
  data: {
    code: 401,
    message: 'Invalid credentials',
    errorRef: '4e48c722-46bf-42e0-aa48-dbd265c10982'
  }
} +8ms
  snyk ERROR: ApiAuthenticationError: Error: Request failed with status code 401
    at Object.makeSnykRequest (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/request.ts:77:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.RequestsManager._makeRequest [as next] (/Users/christiantonnesen/.asdf/installs/nodejs/16.14.0/.npm/lib/node_modules/snyk-api-import/node_modules/snyk-request-manager/src/lib/request/requestManager.ts:100:24) {
  data: {
    code: 401,
    message: 'Invalid credentials',
    errorRef: 'd4b7ddad-1591-4e36-b08e-11df80b4bff5'
  }
}

Import targets for the ACR [Feature Request]

Describe the user need

I want Snyk to supply a target JSON list for the ACR or allow CSV Import instead JSON

As of now, I am using the following bash script to generate the list of images from the ACR. This generate the CSV

registry_name='sec'
destination='snyk-auto-import/output.csv'
az acr login --name $registry_name
touch $destination

repos="$(az acr repository list -n $registry_name --output tsv)"

for i in $repos; do
    images="$(az acr repository show-tags -n $registry_name --repository $i --output tsv --orderby time_desc)"
    for j in $images; do
        echo $i":"$j >> $destination;
    done;
done;

cat output.csv

Describe expected behaviour

It would be nice if snyk can supply the target json similar SCM target - https://github.com/snyk-tech-services/snyk-api-import/blob/master/docs/import-data.md or allow the user to supply the CSV

Additional context

This feature can be available in the Integration UI, rather than running the additional script

Provide regular batch/repo status when polling

Currently once the tool issues an import of a batch of repos, it displays the following until the imports are finished

  snyk:poll-import Will re-check import task in "30000 ms" +30s
  snyk:poll-import Polling locationUrl=https://snyk.io/api/v1/org/########/integrations/########/import/######## +0ms
  snyk:poll-import Will re-check import task in "30000 ms" +312ms
  snyk:poll-import Polling locationUrl=https://snyk.io/api/v1/org/########/integrations/########/import/######## +0ms
  snyk:poll-import Import task status is "pending" +795ms
  snyk:poll-import Import task status is "pending" +486ms
  snyk:poll-import Will re-check import task in "30000 ms" +30s
  snyk:poll-import Polling locationUrl=https://snyk.io/api/v1/org/########/integrations/########/import/######## +0ms
  snyk:poll-import Will re-check import task in "30000 ms" +480ms
  snyk:poll-import Polling locationUrl=https://snyk.io/api/v1/org/########/integrations/########/import/######## +0ms
  snyk:poll-import Import task status is "pending" +282ms
  snyk:poll-import Import task status is "pending" +649ms

The polling can last quite a long time. It would be useful to provide a periodic reminder of what batch or current org/repo names are being processed while the polling is going on.

Create Snyk Organizations with top level GitLab groups[🙏]

Describe the user need
With the orgs:data creation with GitLab, it will create a Snyk organization for the groups and their subgroups. The customer would like a flag that you can specify a flag that will tell orgs:data to create Snyk organizations with subgroups or without subgroups.

Describe expected behavior

The plan would be a flag like include_subgroups for GitLab only. When this flag is false it will remove sub-groups from the groups collected when orgs:data is run. Snyk-api-import will only create orgs for the top-level groups.

Additional context

I believe this would need to be specified in this file on line 23 and having an if block when a flag is specified too, exclude sub-groups with top_level_only flag in this doc.

Create logs directory if it doesn't exist

I ran the tool with the environment variable set as such:
export SNYK_LOG_PATH=${SRC_DIR}/logs/

In my source directory, the logs directory doesn't exist. I then ran:

DEBUG=snyk* ${SRC_DIR}/snyk-api-import

The tool proceeded to run. I think the expect behaviour when the directory doesn't exist should be one of:

create the directory (preferred)
fail and warn

If no scannable content found in default branch, additional branches specified in targets.json are not attempted[🐛]

./run_targets_emu.out:2024-05-28 19:58:00 | INFO | tasks.targets:generate_targets_for_repository:88 - [probono-microsite] Import file specified branches, using those..

import json
f = open('targets.json')
targets=json.load(f)
for t in targets['targets']:
... if t['target']['name'] == 'probono-microsite':
... print(t)
... print()
...
{'orgId': '90cde499-8dbe-482e-aba2-ed76b1426d15', 'integrationId': '140f3713-7c89-405e-b347-16d469fff5c6', 'target': {'fork': False, 'owner': 'massmutual-git', 'name': 'probono-microsite', 'branch': 'develop'}}

{'orgId': '90cde499-8dbe-482e-aba2-ed76b1426d15', 'integrationId': '140f3713-7c89-405e-b347-16d469fff5c6', 'target': {'fork': False, 'owner': 'massmutual-git', 'name': 'probono-microsite', 'branch': 'qa'}}

Expected behaviour

Despite no scannable content in default (main) branch (yet), import should continue with non-default branches that do have scannable content at this point.

Actual behaviour

Import sees no scannable content in default branch and does not attempt the non-default target repositories.

Steps to reproduce

Create a repository with a default branch that holds no scannable content
Create a new branch that does have scannable content
Execute import against these targets
Observe that only the default branch is attempted

Debug log

If applicable, please add DEBUG=*snyk* <command here> before your command and include the output here **ensuring to remove any sensitive/personal details or tokens.

Screenshots

If applicable, add screenshots to help explain your problem.

Have sourceOrg as a configurable switch/flag[🙏]

Describe the user need
Right now you can specify sourceOrgID in the org json to ensure that all subsequently created orgs will have the same config as the source org. This is great, except its manual (have to manually modify the org json and add sourceOrg).
Describe expected behaviour

Ideally by having a switch like --sourceOrgID=xxxx you can set a single sourceOrgID across all orgs in the json you are trying to upload to snyk.

Additional context

Add any other context or screenshots about the feature request here.

[🐛] Sync function does not delete shallow clones

Expected behaviour

Sync function should delete the shallow clonses from the /tmp directory when finished analyzing.

Actual behaviour

snyk:clone-and-analyze Failed to delete /tmp/snyk-clone- Error was Error: ENOTEMPTY: directory not empty, rmdir '/tmp/snyk-clone-'

Steps to reproduce

/usr/local/bin/snyk-api-import sync --orgPublicId= --source=github

Debug log

See above in Actual behaviour

[🐛] Hitting wrong Github API endpoint to enumerate repositories

Node version v17.7.1
Amazon Linux 2 OS

$ snyk-api-import import:data --orgsData=/home/ec2-user/snyk-orgs.json --source=github-enterprise --integrationType=github-enterprise --sourceUrl=https://api.github.com

Expected behaviour

Return list of repositories under organisation.

Actual behaviour

HTTP 404 error as the tool is trying to go to https://api.github.com/api/v3/orgs/<org_name>/repos?per_page=100&page=1 whereas it should be https://api.github.com/orgs/<org_name>/repos?per_page=100&page=1

Steps to reproduce

Run the above command against any Github Enterprise organisation.

Debug log

snyk-api-import import:data --orgsData=/home/ec2-user/snyk-orgs.json --source=github-enterprise --integrationType=github-enterprise --sourceUrl=https://api.github.com
  snyk:generate-data-script ℹ️  Options: {"_":["import:data"],"orgsData":"/home/ec2-user/snyk-orgs.json","orgs-data":"/home/ec2-user/snyk-orgs.json","source":"github-enterprise","integrationType":"github-enterprise","integration-type":"github-enterprise","sourceUrl":"https://api.github.com","source-url":"https://api.github.com","$0":"snyk-api-import"} +0ms
  snyk:generate-targets-data Processing XXXX +0ms
  snyk:list-repos-script Fetching all repos data for org: XXXX +0ms
  snyk:list-repos-script Fetching page: 1 +1ms
  snyk:list-repos-script Failed to fetch page: 1 RequestError [HttpError]: Not Found
    at /home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/node_modules/@octokit/request/dist-src/fetch-wrapper.js:68:27
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at fetchReposForPage (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/lib/source-handlers/github/list-repos.ts:24:15)
    at fetchAllRepos (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/lib/source-handlers/github/list-repos.ts:59:38)
    at Object.listGithubRepos (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/lib/source-handlers/github/list-repos.ts:91:17)
    at Object.githubEnterpriseRepos [as github-enterprise] (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/scripts/generate-targets-data.ts:34:37)
    at Object.generateTargetsImportDataFile (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/scripts/generate-targets-data.ts:97:11)
    at Object.handler (/home/ec2-user/.nvm/versions/node/v17.7.1/lib/node_modules/snyk-api-import/src/cmds/import:data.ts:79:17) {
  status: 404,
  response: {
    url: 'https://api.github.com/api/v3/orgs/xxxx/repos?per_page=100&page=1',
    status: 404,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      connection: 'close',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Wed, 16 Mar 2022 22:30:24 GMT',
      'github-authentication-token-expiration': '2023-03-15 13:00:00 UTC',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-accepted-oauth-scopes': 'repo',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': 'C3F8:372A:F7BBB:10A1E4:62326500',
      'x-oauth-scopes': 'admin:repo_hook, read:org, repo',
      'x-ratelimit-limit': '5000',
      'x-ratelimit-remaining': '4395',
      'x-ratelimit-reset': '1647472434',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '605',
      'x-xss-protection': '0'
    },
    data: {
      message: 'Not Found',
      documentation_url: 'https://docs.github.com/rest'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/api/v3/orgs/xxxxx/repos?per_page=100&page=1',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-rest.js/18.12.0 octokit-core.js/3.6.0 Node.js/17.7.1 (linux; x64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

Import script & readme file improvements

When trying to use the snyk-api-import script, there are several things that can make the use of script a bit more clear, and small improvements on the readme file that might reduce the questions around the use of this library.

Might be worth adding files and exclusionGlobs to all of the examples. (For me it raised a question does the is should look different two between configurations?)
exclusionGlobs - we should add a comment about being careful from setting the exclusionGlobs to be [] (empty) as this will override all the default exclusions Snyk provides.
SNYK_IMPORT_PATH - the path to the import file - I referred to this at the beginning as a path to the directory, not to a specific file, and more than that not to the import-projects.json file.
In addition, this path (on mac at least) needs to be the full path for the files - and not only the file itself.
Suggestion - make SNYK_IMPORT_PATH optional and check first if the import-projects.json exists in the current directory.

snyk / snyk-api-import Goto Github PK

snyk-api-import's Introduction

snyk-api-import

Table of Contents

Installation

Standalone executables (macOS, Linux, Windows)

More installation methods

Install with npm or Yarn

Usage

FAQ

snyk-api-import's People

Contributors

Stargazers

Watchers

Forkers

snyk-api-import's Issues

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Screenshots

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Expected behaviour

Actual behaviour

Steps to reproduce

Install curl

Download and install the snyk-api-import tool

Create working folder

Set working folder

Copy the python script to the Docker image

Run the command on container startup

Setup logging

Copy json file to root directory

Load the secrets and Snyk organization ID

Set environment variables

Check if the file exists

Import:data command

Check for presence of github-enterprise-import-targets.json

Import command

Sync command

Debug log

Screenshots

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Screenshots

🚨 The automated release from the master branch failed. 🚨

No npm token specified.

🚨 The automated release from the master branch failed. 🚨

Cannot push to the Git repository.

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Screenshots

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Expected behaviour

Actual behaviour

Steps to reproduce

Debug log

Recommend Projects

Recommend Topics

Recommend Org

🚨 The automated release from the `master` branch failed. 🚨

🚨 The automated release from the `master` branch failed. 🚨