cisagov/action-lineage

action-lineage

A GitHub Action to automatically generate PR requests from upstream repositories regardless of the fork network.

Repository Lineage configuration

Lineage is configured using .github/lineage.yml in a repository. Each upstream repository is listed in the lineage section.

Key	Description	Required
local-branch	The branch that will receive new changes.	No
remote-url	The https URL of the upstream repository.	Yes
remote-branch	The branch in the upstream repository.	No

Below is an example configuration that defines two upstream repositories. The skeleton repository specifies both the source and destination branches, while the extra-sauce repository uses the default branches for both repositories.

---
version: "1"

lineage:
  skeleton:
    local-branch: develop
    remote-url: https://github.com/cisagov/skeleton-generic.git
    remote-branch: develop
  extra-sauce:
    remote-url: https://github.com/felddy/extra-skel-sauce.git

Usage

Inputs

Name	Description	Interpreted Type	Default	Required
access_token	GitHub personal access token (see GitHub's documentation).	string	n/a	yes
mask_non_public_repos	Whether to mask the names of non-public (private and internal) repositories in the GitHub Actions logs.	boolean	true	no
include_non_public_repos	Whether to process non-public (private and internal) repositories.	boolean	false	no
repo_query	GitHub search query to use when finding repositories for which to create pull requests (e.g. "org:cisagov archived:false").	string	n/a	yes

Outputs

None.

Sample GitHub Actions workflow

The Lineage action requires a personal access token so that it may open pull requests. For public repositories this token must have the public_repo permission enabled. The token is provided using the repository secrets.

---
name: lineage_scan

on:
  schedule:
    - cron: "0 0 * * *"

env:
  ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}

jobs:
  cisagov:
    runs-on: ubuntu-latest
    steps:
      - name: Check all organization repositories
        uses: cisagov/action-lineage@develop
        with:
          access_token: ${{ env.ACCESS_TOKEN }}
          repo_query: "org:cisagov archived:false"

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

action-lineage's Issues

Better handle additional commits after a PR has been created

💡 Summary

Lineage should better handle the case where it has already created a pull request, but more changes are merged into the relevant skeleton.

Motivation and context

Lineage currently just blindly tacks the additional commits onto the existing pull request. This is problematic if:

• The existing pull request's description does not indicate that there are conflicts
• The additional commits create a conflict

Implementation notes

• Lineage should at a minimum update the pull request description if the new commits generate a conflict and the existing description does not indicate that.
• Even if the new commits do not generate a conflict, Lineage should possibly add a blurb to the pull request description to inform the user that additional commits were added.
• Lineage probably needs to warn the user (via the PR description) if it detects that the PR has been manually modified and there are additional commits to be added. In many cases the user would want to delete the branch and let Lineage recreate it from scratch.

Acceptance criteria

How do we know when this work is done?

• Criterion

Private repositories can not be handled, missing auth

🐛 Summary

Private repositories can not be processed due to check for lineage configuration never returning result from unauthenticated request:

action-lineage/src/lineage/entrypoint.py

Lines 110 to 118 in 3a0be21

	def get_config(repo: Repository.Repository) -> Optional[dict]:
	"""Read the lineage configuration for this repo without checking it out."""
	config_url: str = f"https://raw.githubusercontent.com/{repo.full_name}/{repo.default_branch}/{CONFIG_FILENAME}"
	logging.debug("Checking for config at: %s", config_url)
	response = requests.get(config_url)
	if response.status_code == 200:
	return yaml.safe_load(response.content)
	else:
	return None

To reproduce

Steps to reproduce the behavior:

1. Setup action normally for private repository scanning
2. Setup private repository and ensure PAT provides access
3. Declare lineage configuration in private repository
4. Trigger action and ensure repository was scanned:

   INFO Querying for repositories: user:Xerkus archived:false
   Xerkus/private-lineage-test
     INFO Checking: Xerkus/private-lineage-test
     INFO Lineage configuration not found for Xerkus/private-lineage-test
   

5. Make repository public and observe action working as expected

Expected behavior

Lineage action can get past the check for lineage configuration existence

Any helpful log output or screenshots

Update Conflict PR Description to Follow Team Best Practices

💡 Summary

Update so the directions for a conflict PR follow team best practices.

Motivation and context

As pointed out in the discussion in cisagov/gh-skeleton#1 (comment) the directions produced for a PR with conflicts do not follow team best practices.

Implementation notes

Current PR description output:

git clone [email protected]:cisagov/action-lineage.git action-lineage
cd action-lineage
git remote add skeleton https://github.com/cisagov/skeleton-python-library.git
git remote set-url --push skeleton no_push
git switch develop
git checkout -b lineage/skeleton --track origin/develop
git pull skeleton HEAD
git checkout --ours -- .github/lineage.yml
git add .github/lineage.yml
git status

Expected PR description output:

git clone [email protected]:cisagov/action-lineage.git action-lineage
cd action-lineage
git remote add skeleton-python-library https://github.com/cisagov/skeleton-python-library.git
git remote set-url --push skeleton-python-library no_push
git switch develop
git checkout -b lineage/skeleton --track origin/develop
git pull skeleton-python-library HEAD
git checkout --ours -- .github/lineage.yml
git add .github/lineage.yml
git status

This can be fixed by changing the value of lineage_id that is passed to the template per

git clone {{ ssh_url }} {{ repo_name }}
cd {{ repo_name }}
git remote add {{ lineage_id }} {{ remote_url }}
git remote set-url --push {{ lineage_id }} no_push
git switch {{ local_branch }}
git checkout -b {{ pr_branch_name }} --track origin/{{ local_branch }}
git pull {{ lineage_id }} {{ remote_branch }}
git status

Acceptance criteria

How do we know when this work is done?

• The full skeleton repository name is used in the directions given for conflict PRs

Cannot pull private parent repository due to missing auth in git client

🐛 Summary

Private parent repository can not be pulled because git client does not receive the token required for authentication.

To reproduce

Steps to reproduce the behavior:

1. Setup lineage action as usual and ensure it works
2. Make parent repository private and make sure PAT has access to the now private repository
3. Run action

Expected behavior

Parent repository is pulled and merge branch is created

Any helpful log output or screenshots

Paste the results here:

Xerkus/public-lineage-test
  INFO Checking: Xerkus/public-lineage-test
  INFO Lineage configuration found for Xerkus/public-lineage-test
  INFO Cloning repository: https://github.com/Xerkus/public-lineage-test.git
  INFO ✅ success
  INFO Processing lineage: skeleton
  INFO Upstream: https://github.com/Xerkus/private-lineage-parent.git HEAD
  INFO Attempting to switch to branch: lineage/skeleton
  INFO ✅ (error ok) return code: 128
  INFO Branch did not exist.  Creating: lineage/skeleton from local main
  INFO Creating branch lineage/skeleton from main
  INFO ✅ success
  INFO Switching to lineage/skeleton
  INFO ✅ success
  INFO Pull request branch is new: True
  INFO Fetching https://github.com/Xerkus/private-lineage-parent.git HEAD
  Traceback (most recent call last):
  CRITICAL fatal: could not read Username for 'https://github.com': No such device or address
  
  CRITICAL ❌ ERROR! return code: 128
    File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/src/lineage/__main__.py", line 5, in <module>
      main()
    File "/src/lineage/entrypoint.py", line 431, in main
      fetch(repo, remote_url, remote_branch)
    File "/src/lineage/entrypoint.py", line 149, in fetch
      run([GIT, "fetch", remote_url, remote_branch], cwd=repo.full_name)
    File "/src/lineage/entrypoint.py", line 68, in run
      raise Exception("Subprocess was expected to exit with 0.")
  Exception: Subprocess was expected to exit with 0.

Permission errors should not cause job failure

🐛 Bug Report

A permission error should be reported, but not failed upon. See:

https://github.com/cisagov/action-lineage/runs/772741162?check_suite_focus=true

Any helpful log output

Paste the results here:

2020-06-15T13:16:56.6114233Z INFO Checking: cisagov/cool-system
2020-06-15T13:16:56.8016847Z INFO Lineage configuration found for cisagov/cool-system
2020-06-15T13:16:56.8017470Z INFO Cloning repository: https://github.com/cisagov/cool-system.git
2020-06-15T13:16:57.3166350Z INFO ✅ success
2020-06-15T13:16:57.3167776Z INFO Processing lineage: skeleton
2020-06-15T13:16:57.3168742Z INFO Upstream: https://github.com/cisagov/skeleton-generic.git HEAD
2020-06-15T13:16:57.3169069Z INFO Attempting to switch to branch: lineage/skeleton
2020-06-15T13:16:57.3194201Z INFO ✅ (error ok) return code: 128
2020-06-15T13:16:57.3194784Z INFO Branch did not exist.  Creating: lineage/skeleton from local develop
2020-06-15T13:16:57.3195117Z INFO Creating branch lineage/skeleton from develop
2020-06-15T13:16:57.3219662Z INFO ✅ success
2020-06-15T13:16:57.3221732Z INFO Switching to lineage/skeleton
2020-06-15T13:16:57.3443533Z INFO ✅ success
2020-06-15T13:16:57.3443679Z INFO Pull request branch is new: True
2020-06-15T13:16:57.6403691Z INFO Fetching https://github.com/cisagov/skeleton-generic.git HEAD
2020-06-15T13:16:57.6404068Z INFO ✅ success
2020-06-15T13:16:57.6429427Z INFO ✅ success
2020-06-15T13:16:57.6462396Z INFO ✅ success
2020-06-15T13:16:57.6462555Z INFO Attempting merge of fetched changes.
2020-06-15T13:16:57.7075221Z INFO ✅ success
2020-06-15T13:16:57.7075723Z INFO Remove any incoming modifications to .github/lineage.yml
2020-06-15T13:16:57.7292196Z INFO ✅ success
2020-06-15T13:16:57.7514505Z INFO ✅ success
2020-06-15T13:16:57.7515575Z INFO Committing merge.
2020-06-15T13:16:57.7783416Z INFO ✅ success
2020-06-15T13:16:57.7787494Z INFO Assigning credentials for push.
2020-06-15T13:16:57.7818077Z INFO ✅ success
2020-06-15T13:16:57.7818718Z INFO Pushing lineage/skeleton to remote.
2020-06-15T13:16:57.9309812Z CRITICAL remote: Permission to cisagov/cool-system.git denied to cisagovbot.
2020-06-15T13:16:57.9310313Z fatal: unable to access 'https://github.com/cisagov/cool-system.git/': The requested URL returned error: 403
2020-06-15T13:16:57.9310435Z 
2020-06-15T13:16:57.9310700Z CRITICAL ❌ ERROR! return code: 128
2020-06-15T13:16:57.9380220Z Traceback (most recent call last):
2020-06-15T13:16:57.9380557Z   File "/src/lineage/entrypoint.py", line 379, in <module>
2020-06-15T13:16:57.9380684Z     sys.exit(main())
2020-06-15T13:16:57.9384292Z   File "/src/lineage/entrypoint.py", line 336, in main
2020-06-15T13:16:57.9384606Z     push(repo, pr_branch_name, "git", access_token)
2020-06-15T13:16:57.9384728Z   File "/src/lineage/entrypoint.py", line 190, in push
2020-06-15T13:16:57.9385276Z     run([GIT, "push", "--set-upstream", "origin", branch_name], cwd=repo.full_name)
2020-06-15T13:16:57.9385422Z   File "/src/lineage/entrypoint.py", line 67, in run
2020-06-15T13:16:57.9385553Z     raise Exception("Subprocess was expected to exit with 0.")
2020-06-15T13:16:57.9385682Z Exception: Subprocess was expected to exit with 0.
2020-06-15T13:16:58.3991507Z Cleaning up orphan processes

Replace calls to git executable with GitPython

🚀 Feature Proposal

It would be better to use GitPython instead of spawning processes that run the git command line.

Change the design of this Action to target a single (using) repository

💡 Summary

Currently this Action has a workflow that runs this Action against our organization on a regularly basis. This is in line with the Action's current design to scan based on a provided query. This issue proposes changing the functionality of this Action to instead be used in a workflow in individual repositories that will scan on their own schedule.

Motivation and context

This change would align this project with the general way that GitHub Actions are used. It would also provide more configuration in downstream repositories and would simplify the logic of this project.

Implementation notes

Important design consideration (in my mind):

• Use the default Actions permissions by default, but allow configuration to provide specific credentials if desired.
• Use the calling repository by default, but allow this to be overridden with a provided value. This would allow a public repository to host a workflow that would update a private repository which is functionality provided by the current implementation. This ties into the above point for allowing external runs to access other repositories.

Acceptance criteria

• The project is redesigned to run as an Action called from a workflow in individual repositories.

Add label to PRs on Lineage creation

💡 Summary

Have Lineage add labels when creating PRs, i.e. upstream update. Possibly also add blocked to Conflict PRs.

Motivation

This will help us track the type of PRs we have and make the labels system more used/useful.

Implementation notes

Please provide details for implementation, such as:

• Add upstream update label to new Lineage PRs
• Possibly also add blocked to Conflict PRs

Acceptance criteria

How do we know when this work is done?

• Determine which label(s) to add as PRs are created
• Update Lineage to apply this label scheme

Consider integrating the standard cisagov PR template

💡 Summary

Consider integrating the standard cisagov PR template into the Lineage PR templates.

Motivation and context

Suppose one is handling a Lineage PR for a descendant of cisagov/skeleton-packer that updates the version of the AMI's base OS or otherwise happens to make a change that changes the build product (i.e., the AMI). It would be convenient to have the checklists there reminding you so create a release, since this could be easily overlooked.

Apart from being helpful in certain cases, this would make our cisagov PRs more uniform.

Version inconsistency

The versions of this action are inconsistent.

Problem

I found a v0.1.0 tag that links to a commit more recent than the v1.0.0 branch.

Solution

I recommend to keep working on your main branch (develop in your case) and tag new versions along.

You could enforce conventional commit messages and then use tools to automate new releases and changelogs and make your life easier.