ossf/allstar

GitHub App to set and enforce security policies

allstar's Issues

Support creating JIRA tickets with the findings

It would be great if AllStar supported creating JIRA tickets instead of just GitHub issues, as many projects use JIRA instead.

I might find time to contribute this next year, if no-one else jumps in before then.

Allow custom label on raised issue

Allstar always creates it's issue(s) with a gray label called allstar. We use an organization-wide label associated with security / risk that we'd like it to use. I've read through the code and config files and I don't see how to configure allstar to use a custom label, but it seems like something that would be possible based on the way the config works currently.

Thanks for the great tool!

SECURITY.md not found under .github/ folder

example issue:

stackables/cloudevents-router-gcp#3

Security Policy SECURITY.md is out of compliance, status:
SECURITY.md not found.
Go to https://github.com/stackables/cloudevents-router-gcp/security/policy to enable.

github docs:
https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository

Show SECURITY.md can be under root, docs, or .github

Administrator requirement for repo default setting

Forking from: #103 (comment)

My thoughts:

it is also a goal for me to have Allstar's default settings represent the highest level of security best practices that we (OpenSSF) can recommend. ...
That said, I also acknowledge that my point of view is from a large-org perspective, and while the org admins do have transient permissions on all repos, they can't be expected to be responsible for all repos, and a direct admin should be required. I see how this would not be the case for a small org. I'd like to bring this up at the next "Best Practices" WG meeting to discuss further.

Binary artifact issue should list the actual artifacts

"Allstar has detected that this repository’s Binary Artifacts security policy is out of compliance. Status: Scorecard Check Binary Artifacts failed: binaries present in source code" is what the message says today.

This is not super helpful. It would be nice if the message was more specific about exactly which files it was referring to.

New to Allstar

Hello Team,

I am new to allstar and tried to use it in my GitHub repository. I have few questions

1. Is it specific only to the GitHub repo? Does it have a docker image or CLI to use it in local/ other git remote repositories

2. I have configured .allstar folder wiht the specified 4 policy yaml file. I dont see any issue or any process running to indicate a check is running and i need to wait for the result. Where can i see the result?

3. Does allstar use scorecard internally?

Thanks in advance.

Update wording and formatting of Binary Artifacts issue

Proposal is to look like:

(No change to title)

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar

Security policy violation
Security policy Binary Artifacts failed: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see Scorecards Documentation.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts found

• dummy.dll
• dummy.exe
• dummy.jar
• dummy.so

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores security best practices for a project. You may wish to run a Scorecards scan directly on this repository for more details.

---

Staging deploy of Allstar for testing, see https://github.com/ossf-tests/.allstar for config. (This is the custom footer)

This issue will auto resolve when the policy is in compliance. (This is current Allstar footer)

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

---

Currently looks like: https://github.com/ossf-tests/scorecard-check-binary-artifacts-e2e-4-binaries/issues/4

Where will issues be submitted?

For the flutter org, we have lots of repos but (with a few small exceptions) we use flutter/flutter/issues for all of them. Where will allstar try to submit issues if I enable that?

Add retry to transport

We get 502 and 504 occasionally from GitHub. These should be retried.

Include scorecard details in created issue

When creating an issue for scorecard, include the details from:
https://github.com/ossf/scorecard/blob/main/checker/check_result.go#L45
in the issue. Only include DetailWarn and above, only include first ~10 lines and note if some lines were cut.

Handle 451

https://developer.github.com/changes/19/#--the-451-status-code-is-now-supported
repo cannot be accessed: GET https://api.github.com/repos/...: 451 Repository access blocked []

Looks like repos can be listed, apps installed, but unavailable due to DMCA takedown request. Allstar should catch this abort this repo, and continue enforcing on other repos in the installation.

Currently we are aborting the whole installation/org when hitting this.
https://github.com/ossf/allstar/blob/main/pkg/enforce/enforce.go#L107

Administrator requirement for outside collaborators does not seem to work with an admin team

Hi,

We got a recent issue opened using the allstar-app:

"Allstar has detected that this repository’s Outside Collaborators security policy is out of compliance. Status:
Did not find any owners of this repository
This policy requires all repositories to have an organization member or team assigned as an administrator. Either there are no administrators, or all administrators are outside collaborators."

I guess due to this change: 01e4892

However, the repo has a team with admin access, and a member of the org is on this team. Perhaps the change didn't take into account the scenario of an admin team as opposed to a member being an admin directly on the team?

Own instance of allstar

Hello @jeffmendoza,

I am trying to create an own instance of allstar.

• I cloned the repository
• build the cmd/allstar
• tried running it locally using the "allstar" command
  Got the below error,
  {"severity":"CRITICAL","error":"open variable gcpsecretmanager://projects/allstar-ossf/secrets/allstar-private-key?decoder=bytes: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.","time":"2021-08-23T15:16:18+05:30","message":"Could not load app secret, shutting down"}
  Is it because allstar can not be run locally?

Convert Binary Artifacts policy to be a generic scorecard policy

Should have multiple checks enabled (3-5 to start). Default config will be to run all enabled checks, but can override in config to only run certain ones. Should have a "success value" to decide what score (0-10) is passing.

GitHub Issue ping duration should be user configurable.

Currently the bot posts an update to each issue every few days. Since we have around a hundred issues open (primarily because of #108) this causes a huge amount of bugspam to anyone following the repository.

In pondering this I realised that technically the bot is violating our contribution guidelines, which say that a comment shouldn't be added if it doesn't add any new information. In general, if a bug is open, it is taken that it is still valid, and nothing need be said if nothing has changed.

Write DEVELOPER.md

Include Go requirements, how to clone, build, test. Install linter and run lint.

Also include how to setup a test app in GitHub and run locally.

Handle BP on free-tier private repos

Branch Protection is not enabled on free-tier private repos: https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches
GitHub responds with:
GET https://api.github.com/repos/org/repo/branches/master/protection: 403 Upgrade to GitHub Pro or make this repository public to enable this feature. []
Currently we error out of the whole installation/org.

Change to fail BP policy, notify text should include the message from GitHub "Upgrade to GitHub Pro or make this repository public to enable this feature."

Change "optOutStrategy" to "useOptOutStrategy"

Improved readability.

https://github.com/ossf/allstar/blob/main/pkg/config/config.go#L40

Documentation is confusing because of non-existent features being documented

It would be significantly clearer if the documentation only included actual implemented features rather than being aspirational.

For example, "fix" is listed as a possible action, separate from the "Proposed, but not yet implemented actions", but later is revealed to do nothing yet. It would be better to just not list it until it does something.

Organization wide SECURITY.md file does not fulfill SECURITY.md policy

If an organization has an organization level SECURITY.md file defined as outline in About default community health files, I would expect that this would satisfy the SECURITY.md policy check.

Instead, an issue was created with a link to create this policy, but this link is actually just to our policy itself, which is defined here.

OptConfig should have a setting "optOutArchivedRepos"

When we enable allstar on the flutter org, it filed about 37 issues complaining that branch protection wasn't enabled on various repos.

All of these were archived, read-only repos. These don't need branch protection, since they have the ultimate protection of being entirely unmodifiable.

Issues created by allstar-app could be improved

I think that it's crucial to include some actionable information when creating an issue of any kind. This is a follow up from this issue: envoyproxy/java-control-plane#173.

I think that a well created ticket should at least include the following information:

1. Why is this ticked created?
2. Why is the current situation bad?
3. What are the risks?
4. How can it be fixed? (step by step guide)
5. How to know it's fixed.

Let's look at the message in the linked issue:

Security Policy Outside Collaborators is out of compliance, status:
Found 1 outside collaborators with admin access.

In my opinion it does not inform why it's an issue and how to resolve it. A better message would be:

Security Policy "Outside Collaborators" is out of compliance.

This policy requires all users with admin access to be members of the organisation. That way if an account is compromised it can be quickly denied access to all resources of the organisation. To fix this you should:
1. Remove the user from the repository based access by going to settings -> permissions -> ...
2. Add the user to the organisation by going to -> ...

If you don't see the settings tab you probably don't have administrative access. Reach out to the administrators of the organisation to fix this issue.

This issue will auto resolve when the policy is in compliance.

If you have any questions - please reach out to XYZ.

If you have any questions or you'd like to agree/disagree please comment below. Also, I'm creating this issue in good faith, I really think better issue contents will improve this project.

Best regards,
slonka

bug: typo in RepoConfig struct

it seems that requireAppproval has a typo (should it be requireApproval?)

allstar/pkg/policies/branch/branch.go

Line 79 in 84b402d

RequireApproval *bool `yaml:"requireAppproval"`

Update linter config

Look at : https://github.com/ossf/scorecard/blob/main/.golangci.yml
See if there are any more linters we want to enable over default. Only include additions to config file necessary over default settings.

Fix the security advisories

https://deps.dev/go/github.com%2Fossf%2Fallstar

In-memory cache could cause OOM

The in-memory cache is great for simple workflows. But for large repositories it usually causes OOM.

A sample repository that was causing OOM was github.com/ApolloAuto/apollo

related issue ossf/scorecard#589

allstar/pkg/ghclients/ghclients.go

Line 83 in 70d3e36

Cache: httpcache.NewMemoryCache(),

This might not be applicable for allstar but wanted to point out the issues it could run into.

Run scorecard on allstar

Eat your own dog food. Run scorecard on allstar.

Support the ability to define the repository types for security issues

It would be nice to be able to define the repository types to include for the security issues. For example, we might want a SECURITY.md for all public and internal repos, but not care about whether private repos have it or not.

Update Scorecard Interface

After discussing with scorecard folks, we will try to use checker.Runner https://github.com/ossf/scorecard/blob/main/checker/check_runner.go as the interface, update: https://github.com/ossf/allstar/blob/main/pkg/policies/binary/binary.go#L117-L137 to use that.

How to use allstar to actually look into a specific branch in a repository

Hello @jeffmendoza,

Is there a possibility that I can look for a specific branch which using allstar?
Assume i have two different branch ,
Main and dev

Main branch as Security.md file and dev doesnt have.
Does allstar detect this and raise an issue?

Please help me understand

Thanks in advance

Allstar frequency

Hello Team,

I am curious to understand the frequency in which allstar scans the repositories to identify security policy violations?

I believe allstar doesn't use webhooks, so how will it know if a push or any action that happens in the GitHub repository?

Thanks in advance

When is the issue submitted?

I've created a sample repository to see how Allstar works. It was 9PM JST, and an issue is created in 6AM in the next day.
I couldn't find documents about when Allstar app is triggered. Is it once a day?
Can we configure when it's triggered, or manually trigger?

Branch protection out of compliance even tough codereview requirements is set to 1

Dart and Flutter Orgs branch protection requires 1 codereview but we allow some bot accounts to bypass the requirement. This is causing the allstar application to detect that the repositories as out of compliance:

Allstar has detected that this repository’s Branch Protection security policy is out of compliance. Status:
PR Approvals not configured for branch master

Could this be related to ossf/scorecard#1614?

Handle 410 when trying to create issue

POST https://api.github.com/repos/org/repo/issues: 410 Issues are disabled for this repo
If Allstar is configured to create an issue, but issues are disabled, fail silently, I guess, and continue.

Enforce certain status checks in branch protections

Accept an optional yaml list of status check names for the branch protection policy to enforce repos have enabled an organization's expected checks. The yaml list should accept objects containing at least the fields name and required (to enforce whether a particular status check should be marked as required).

I'm happy to take on this pull request myself if the maintainers approve of the issue.

Issues should include footer text with information about opt-in/opt-out strategy configuration

Users who want to opt out of or disable Allstar need to know the strategy configuration (opt-in vs opt-out vs. repo-level), which involves looking in several places to find certain files or settings. Is it possible to add a feature that tells users the configuration in each issue they receive? For example, a line in the issue footer could say:

Allstar is enabled on your repository
Allstar is enabled on your organization in the opt-in strategy
Allstar is enabled on your organization in the opt-out strategy with/without repository override.

Policies documentation is incomplete

https://github.com/ossf/allstar#policies links to golang documentation as part of its Yaml documentation. It's quite confusing. I was unable to determine at a glance what form the .yaml files should take and what options were available.

Allow pattern matching for repos on which to `enforceBranches`

It would be great if the enforceBranches configuration for branch_protection.yaml could be set to apply to any repository which matches a pattern.

Example of usage:
In our repositories across our organization we perform our releases from branches which match *.*.x, we would like to enforce our branch protections on multiple repositories for those branches, but would love to have this requirement centralized in our .allstar repository.

You could just include it in the yaml as the branch name:

enforceBranches:
  "*-releasable":
    - "*.*.x"

For our usage we would have:

enforceBranches:
  "*":
    - "*.*.x"

Organization repo name and config... match probot?

create a repository named .allstar in your organization

Many GitHub apps are based on Probot and default to the .github repository for organization config. The default setting of this app requires splitting config into yet another repository.

At a minimum it would be nice if this app allowed specifying the org config location.

It would be nice to mirror https://probot.github.io/docs/best-practices/#configuration.

Issues should link to the repositories

Since issues for an org can all be sent to one issue database, it would be nice if the issues themselves linked to the offending repositories, so that the admin didn't have to copy-paste the name into a URL to go to the settings of that repo to fix the issue.

Alerting for repos missing dependabot config (`dependabot.yml`)

Similar to the SECURITY.md community health check, for organizations that are interested in having dependabot automated updates enabled on their repositories, it'd be great to have a check that alerts when a .github/dependabot.yml config is not present within the repo.

Ideally, it'd be cool to have this baked into https://github.com/organizations/<org-name>/settings/security_analysis.
cc: @jhutchings1

Allow enabling / disabling on all private / public repos

Right now the opt in/out config must list all repos to opt in or out. It would be good to be able to opt in/out all public or private repos.

It's unclear how "disableRepoOverride: true" in the allstar.yaml file interacts with policy files

The readme (https://github.com/ossf/allstar) links to a sample config repository whose policy files contain "disableRepoOverride: true". It's not clear from the documentation whether this overrides, combines with, or is overridden by the global "disableRepoOverride: true" in the "allstar.yaml" (which also has it set in that sample repo).

Preview mode

I administer an org with many, many repositories with varying levels of security (and varying levels of importance). I would love to enable allstar globally, but I am concerned that it will file hundreds of issues on various repos that may be secret, archived, unimportant, or some combination of the above, as well as filing issues that are known to be bogus (for example some of our repos are intentionally for storing binary files so the whole "don't store binary files" thing won't work in those repos).

What I would like is a way to get a preview of what allstar is going to complain about ahead of me doing something with public visibility.

Feature: require GitHub Actions to be present

Would be great if allstar could verify whether certain GitHub Actions exist and a minimum version.
Often GitHub Actions contain a set of security screenings to block the PR.

Allstar Integration on Enterprise version of GitHub

Hello Team,

I have followed your documentation and enabled allstart github app as below.

1. Create .allstar repo under the org as internal (public is not allowed in our enterprise either private or internal)
2. Create all the four config files and opted for scanning all repositories

Do not see any issues being created automatically, Does it support GitHub enterprise with allstar as internal repository type?
Is there anything else to configure to get the issues created in respective repositories?

Thanks

Search .github repo for allstar org level config as a second priority

To allow flexibility, we should allow org-level config in the .github repo and allstar sub-directory. This should be searched only if the .allstar repo does not exist in the org. Note, we must check for repo existance of .allstar, and not just try to get the particular config file (ex: .allstar/foo.yaml), as some config files will not exist on purpose. If the .allstar repo exists at all, then .github should not be used at all.

This will address #27

Upgrade to v3 of scorecard

allstar/go.mod

Line 13 in 1c2befa

github.com/ossf/scorecard v1.2.1-0.20210722153731-89c8e2af3131

The scorecard version v3 has more reliable features.

Document necessary access rights for repos

It's not immediately clear to me whether I need to give special rights to any particular user for allstar to be able to see the .allstar repo if it's private, or file issues in private repos. The documentation seems to be silent about this. I'm guessing that apps get magic access to everything somehow?

Readme doesn't describe priority order for opt-out features

There are various ways to opt repos in and out, including these:

optConfig:
  optOutStrategy: true
  optOutRepos:
  - ...
  optInRepos:
  - ...
  optOutPrivateRepos: true
  optOutPublicRepos: false

...and including a per-repo file and an org-wide .allstar repo file.

It is highly unclear from the documentation what will win if there is a conflict (additionally it seems there's an undocumented feature that changes which will win, as mentioned briefly in opt-out.md, which makes this even more unclear). (edit: found the documentation for this feature)

For example, does optOutPrivateRepos win vs an explicit optInRepos? What if optOutStrategy is true, but optOutPrivateRepos and optOutPublicRepos are also true? What if a repo opts out locally but is listed explicitly in the global .allstar file?