cisagov/action-lineage

Welcome to cisagov

Welcome to cisagov, the GitHub home for the Cybersecurity and Infrastructure Security Agency (CISA)!

This repository aims to make it easier to get working with GitHub and Free and Open Source Software (FOSS) for people who work at or with CISA.

For developer-focused documentation and guides, please visit our development-guide repository.

Common questions

Getting started with GitHub

• How do I make a GitHub account?
  • Please see the FISMA-Ready GitHub Guide
• Why do I add my work email instead of making a separate work-only GitHub account?
  • GitHub's terms of service say to use one account per person. Any commits made will be associated with the user who created them, and GitHub allows for granular, role-based access that can also be revoked when someone departs CISA.
• How do I use GitHub? Where do I start?
  • 18F Intro to GitHub
  • Digital.gov intro to GitHub
  • Always remember that GitHub is a version control system, so all changes are saved and there is nothing we can't fix or undo.

GitHub access

• How do I get access to the cisagov organization?
  • For federal employees, send your GitHub username to [email protected]
  • For contractors, ask your fed lead to send in your username
• How do I get put on a team within the cisagov organization?
  • See Github's organizations and teams documentation
  • You'll need to contact a team member or a cisagov administrator
• How do I get access to a specific repository?
  • Most repositories are associated with the teams that use them, so to get access you can contact [email protected], a team member, or a cisagov administrator to get access
• How do I make a new repository?
  • GitHub's documentation on creating a repository
  • cisagov's Project Setup guide

Policies and content guides

• What belongs in cisagov versus another GitHub organization?
  • Here are some questions to ask when considering posting a project:
    • Does CISA use or develop the software? Is it developed by or for one of the groups or divisions within CISA?
      • If not, we recommend the authors create their own GitHub organization and post their work there
    • What source control system is in place currently?
      • Many source control systems, such as Mercurial and GitLab, can export the entire development history for import into GitHub - this is by far the preferred method
    • Is the project still under active development or is it in maintenance?
      • If the project no longer has a team performing maintenance, we recommend the repository be archived to make that clear to people who may want to use it
• What belongs on cisa.gov versus on cisagov?
  • The cisa.gov site is primarily focused at an audience outside of CISA, such as Critical Infrastructure partners or the public
  • cisagov is for both internal and external users, as well as partners. It exists specifically to share projects with the public as well as internal users.
• Working in public (dos and don'ts, best practices)
  • As a best practice, use the cisagov organization issue templates and pull request templates. These templates are available by default in all repositories created in the cisagov organization.
  • As a best practice, we require code reviews before merging pull requests. This is done using branch protection.
• When should I talk to CISA External Affairs (EA)?
  • Early and often!
• What is CISA's open source policy?
  • See CISA's open source policy here

Feedback and contact

Have an idea about how to make these pages better? File an issue!

For any repository-specific questions or feedback, please make an issue in that repository so the appropriate team will see it.

For more about CISA as an agency or any of its subcomponents, please visit the About CISA page on cisa.gov.

For other GitHub-related questions, feel free to email us.

Developer resources

We have a cisagov development-guide repository, which contains coding standards, steps for setting up a development environment, and other information.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

Thanks

We would like to thank the General Services Administration and 18F, the Consumer Financial Protection Bureau, Department of Defense, and Office of Management and Budget for their work in blazing the path for the use of FOSS in the U.S. federal government.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

action-lineage's Issues

Change the design of this Action to target a single (using) repository

💡 Summary

Currently this Action has a workflow that runs this Action against our organization on a regularly basis. This is in line with the Action's current design to scan based on a provided query. This issue proposes changing the functionality of this Action to instead be used in a workflow in individual repositories that will scan on their own schedule.

Motivation and context

This change would align this project with the general way that GitHub Actions are used. It would also provide more configuration in downstream repositories and would simplify the logic of this project.

Implementation notes

Important design consideration (in my mind):

• Use the default Actions permissions by default, but allow configuration to provide specific credentials if desired.
• Use the calling repository by default, but allow this to be overridden with a provided value. This would allow a public repository to host a workflow that would update a private repository which is functionality provided by the current implementation. This ties into the above point for allowing external runs to access other repositories.

Acceptance criteria

• The project is redesigned to run as an Action called from a workflow in individual repositories.

Version inconsistency

The versions of this action are inconsistent.

Problem

I found a v0.1.0 tag that links to a commit more recent than the v1.0.0 branch.

Solution

I recommend to keep working on your main branch (develop in your case) and tag new versions along.

You could enforce conventional commit messages and then use tools to automate new releases and changelogs and make your life easier.

Add label to PRs on Lineage creation

💡 Summary

Have Lineage add labels when creating PRs, i.e. upstream update. Possibly also add blocked to Conflict PRs.

Motivation

This will help us track the type of PRs we have and make the labels system more used/useful.

Implementation notes

Please provide details for implementation, such as:

• Add upstream update label to new Lineage PRs
• Possibly also add blocked to Conflict PRs

Acceptance criteria

How do we know when this work is done?

• Determine which label(s) to add as PRs are created
• Update Lineage to apply this label scheme

Better handle additional commits after a PR has been created

💡 Summary

Lineage should better handle the case where it has already created a pull request, but more changes are merged into the relevant skeleton.

Motivation and context

Lineage currently just blindly tacks the additional commits onto the existing pull request. This is problematic if:

• The existing pull request's description does not indicate that there are conflicts
• The additional commits create a conflict

Implementation notes

• Lineage should at a minimum update the pull request description if the new commits generate a conflict and the existing description does not indicate that.
• Even if the new commits do not generate a conflict, Lineage should possibly add a blurb to the pull request description to inform the user that additional commits were added.
• Lineage probably needs to warn the user (via the PR description) if it detects that the PR has been manually modified and there are additional commits to be added. In many cases the user would want to delete the branch and let Lineage recreate it from scratch.

Acceptance criteria

How do we know when this work is done?

• Criterion

Replace calls to git executable with GitPython

🚀 Feature Proposal

It would be better to use GitPython instead of spawning processes that run the git command line.

Update Conflict PR Description to Follow Team Best Practices

💡 Summary

Update so the directions for a conflict PR follow team best practices.

Motivation and context

As pointed out in the discussion in cisagov/gh-skeleton#1 (comment) the directions produced for a PR with conflicts do not follow team best practices.

Implementation notes

Current PR description output:

git clone [email protected]:cisagov/action-lineage.git action-lineage
cd action-lineage
git remote add skeleton https://github.com/cisagov/skeleton-python-library.git
git remote set-url --push skeleton no_push
git switch develop
git checkout -b lineage/skeleton --track origin/develop
git pull skeleton HEAD
git checkout --ours -- .github/lineage.yml
git add .github/lineage.yml
git status

Expected PR description output:

git clone [email protected]:cisagov/action-lineage.git action-lineage
cd action-lineage
git remote add skeleton-python-library https://github.com/cisagov/skeleton-python-library.git
git remote set-url --push skeleton-python-library no_push
git switch develop
git checkout -b lineage/skeleton --track origin/develop
git pull skeleton-python-library HEAD
git checkout --ours -- .github/lineage.yml
git add .github/lineage.yml
git status

This can be fixed by changing the value of lineage_id that is passed to the template per

git clone {{ ssh_url }} {{ repo_name }}
cd {{ repo_name }}
git remote add {{ lineage_id }} {{ remote_url }}
git remote set-url --push {{ lineage_id }} no_push
git switch {{ local_branch }}
git checkout -b {{ pr_branch_name }} --track origin/{{ local_branch }}
git pull {{ lineage_id }} {{ remote_branch }}
git status

Acceptance criteria

How do we know when this work is done?

• The full skeleton repository name is used in the directions given for conflict PRs

Consider integrating the standard cisagov PR template

💡 Summary

Consider integrating the standard cisagov PR template into the Lineage PR templates.

Motivation and context

Suppose one is handling a Lineage PR for a descendant of cisagov/skeleton-packer that updates the version of the AMI's base OS or otherwise happens to make a change that changes the build product (i.e., the AMI). It would be convenient to have the checklists there reminding you so create a release, since this could be easily overlooked.

Apart from being helpful in certain cases, this would make our cisagov PRs more uniform.

Private repositories can not be handled, missing auth

🐛 Summary

Private repositories can not be processed due to check for lineage configuration never returning result from unauthenticated request:

action-lineage/src/lineage/entrypoint.py

Lines 110 to 118 in 3a0be21

	def get_config(repo: Repository.Repository) -> Optional[dict]:
	"""Read the lineage configuration for this repo without checking it out."""
	config_url: str = f"https://raw.githubusercontent.com/{repo.full_name}/{repo.default_branch}/{CONFIG_FILENAME}"
	logging.debug("Checking for config at: %s", config_url)
	response = requests.get(config_url)
	if response.status_code == 200:
	return yaml.safe_load(response.content)
	else:
	return None

To reproduce

Steps to reproduce the behavior:

1. Setup action normally for private repository scanning
2. Setup private repository and ensure PAT provides access
3. Declare lineage configuration in private repository
4. Trigger action and ensure repository was scanned:

   INFO Querying for repositories: user:Xerkus archived:false
   Xerkus/private-lineage-test
     INFO Checking: Xerkus/private-lineage-test
     INFO Lineage configuration not found for Xerkus/private-lineage-test
   

5. Make repository public and observe action working as expected

Expected behavior

Lineage action can get past the check for lineage configuration existence

Any helpful log output or screenshots

Cannot pull private parent repository due to missing auth in git client

🐛 Summary

Private parent repository can not be pulled because git client does not receive the token required for authentication.

To reproduce

Steps to reproduce the behavior:

1. Setup lineage action as usual and ensure it works
2. Make parent repository private and make sure PAT has access to the now private repository
3. Run action

Expected behavior

Parent repository is pulled and merge branch is created

Any helpful log output or screenshots

Paste the results here:

Xerkus/public-lineage-test
  INFO Checking: Xerkus/public-lineage-test
  INFO Lineage configuration found for Xerkus/public-lineage-test
  INFO Cloning repository: https://github.com/Xerkus/public-lineage-test.git
  INFO ✅ success
  INFO Processing lineage: skeleton
  INFO Upstream: https://github.com/Xerkus/private-lineage-parent.git HEAD
  INFO Attempting to switch to branch: lineage/skeleton
  INFO ✅ (error ok) return code: 128
  INFO Branch did not exist.  Creating: lineage/skeleton from local main
  INFO Creating branch lineage/skeleton from main
  INFO ✅ success
  INFO Switching to lineage/skeleton
  INFO ✅ success
  INFO Pull request branch is new: True
  INFO Fetching https://github.com/Xerkus/private-lineage-parent.git HEAD
  Traceback (most recent call last):
  CRITICAL fatal: could not read Username for 'https://github.com': No such device or address
  
  CRITICAL ❌ ERROR! return code: 128
    File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/src/lineage/__main__.py", line 5, in <module>
      main()
    File "/src/lineage/entrypoint.py", line 431, in main
      fetch(repo, remote_url, remote_branch)
    File "/src/lineage/entrypoint.py", line 149, in fetch
      run([GIT, "fetch", remote_url, remote_branch], cwd=repo.full_name)
    File "/src/lineage/entrypoint.py", line 68, in run
      raise Exception("Subprocess was expected to exit with 0.")
  Exception: Subprocess was expected to exit with 0.

Permission errors should not cause job failure

🐛 Bug Report

A permission error should be reported, but not failed upon. See:

https://github.com/cisagov/action-lineage/runs/772741162?check_suite_focus=true

Any helpful log output

Paste the results here:

2020-06-15T13:16:56.6114233Z INFO Checking: cisagov/cool-system
2020-06-15T13:16:56.8016847Z INFO Lineage configuration found for cisagov/cool-system
2020-06-15T13:16:56.8017470Z INFO Cloning repository: https://github.com/cisagov/cool-system.git
2020-06-15T13:16:57.3166350Z INFO ✅ success
2020-06-15T13:16:57.3167776Z INFO Processing lineage: skeleton
2020-06-15T13:16:57.3168742Z INFO Upstream: https://github.com/cisagov/skeleton-generic.git HEAD
2020-06-15T13:16:57.3169069Z INFO Attempting to switch to branch: lineage/skeleton
2020-06-15T13:16:57.3194201Z INFO ✅ (error ok) return code: 128
2020-06-15T13:16:57.3194784Z INFO Branch did not exist.  Creating: lineage/skeleton from local develop
2020-06-15T13:16:57.3195117Z INFO Creating branch lineage/skeleton from develop
2020-06-15T13:16:57.3219662Z INFO ✅ success
2020-06-15T13:16:57.3221732Z INFO Switching to lineage/skeleton
2020-06-15T13:16:57.3443533Z INFO ✅ success
2020-06-15T13:16:57.3443679Z INFO Pull request branch is new: True
2020-06-15T13:16:57.6403691Z INFO Fetching https://github.com/cisagov/skeleton-generic.git HEAD
2020-06-15T13:16:57.6404068Z INFO ✅ success
2020-06-15T13:16:57.6429427Z INFO ✅ success
2020-06-15T13:16:57.6462396Z INFO ✅ success
2020-06-15T13:16:57.6462555Z INFO Attempting merge of fetched changes.
2020-06-15T13:16:57.7075221Z INFO ✅ success
2020-06-15T13:16:57.7075723Z INFO Remove any incoming modifications to .github/lineage.yml
2020-06-15T13:16:57.7292196Z INFO ✅ success
2020-06-15T13:16:57.7514505Z INFO ✅ success
2020-06-15T13:16:57.7515575Z INFO Committing merge.
2020-06-15T13:16:57.7783416Z INFO ✅ success
2020-06-15T13:16:57.7787494Z INFO Assigning credentials for push.
2020-06-15T13:16:57.7818077Z INFO ✅ success
2020-06-15T13:16:57.7818718Z INFO Pushing lineage/skeleton to remote.
2020-06-15T13:16:57.9309812Z CRITICAL remote: Permission to cisagov/cool-system.git denied to cisagovbot.
2020-06-15T13:16:57.9310313Z fatal: unable to access 'https://github.com/cisagov/cool-system.git/': The requested URL returned error: 403
2020-06-15T13:16:57.9310435Z 
2020-06-15T13:16:57.9310700Z CRITICAL ❌ ERROR! return code: 128
2020-06-15T13:16:57.9380220Z Traceback (most recent call last):
2020-06-15T13:16:57.9380557Z   File "/src/lineage/entrypoint.py", line 379, in <module>
2020-06-15T13:16:57.9380684Z     sys.exit(main())
2020-06-15T13:16:57.9384292Z   File "/src/lineage/entrypoint.py", line 336, in main
2020-06-15T13:16:57.9384606Z     push(repo, pr_branch_name, "git", access_token)
2020-06-15T13:16:57.9384728Z   File "/src/lineage/entrypoint.py", line 190, in push
2020-06-15T13:16:57.9385276Z     run([GIT, "push", "--set-upstream", "origin", branch_name], cwd=repo.full_name)
2020-06-15T13:16:57.9385422Z   File "/src/lineage/entrypoint.py", line 67, in run
2020-06-15T13:16:57.9385553Z     raise Exception("Subprocess was expected to exit with 0.")
2020-06-15T13:16:57.9385682Z Exception: Subprocess was expected to exit with 0.
2020-06-15T13:16:58.3991507Z Cleaning up orphan processes