Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Allow "incomplete" checks for --local repos #3832

Open
pnacht opened this issue Jan 30, 2024 · 1 comment · May be fixed by #4423
Open

Feature: Allow "incomplete" checks for --local repos #3832

pnacht opened this issue Jan 30, 2024 · 1 comment · May be fixed by #4423
Labels
good first issue Good for newcomers kind/enhancement New feature or request

Comments

@pnacht
Copy link
Contributor

pnacht commented Jan 30, 2024

Is your feature request related to a problem? Please describe.
Running scorecard with --local only allows it to run 6 checks. Of the 12 that aren't run, I propose that 6 be allowed to run, even if the results may be incomplete when compared with the upstream:

  • Fuzzing
  • License
  • Maintained
  • Packaging
  • SAST
  • Security-Policy

Describe the solution you'd like

  • Fuzzing only requires a non-local repo for OSS-Fuzz, all other fuzzers are detected using files in the repository itself. With --local, we simply don't look for OSS-Fuzz.
  • License tries to use the platform's API, but it already has fallback methods to scan the repository itself. With --local, we simply use the fallbacks.
  • Maintained can score a --local repo using just the commit activity: use git log to detect the frequency of commits and use the date of the root commit to determine whether the repo is 90+ days old.
  • Packaging currently looks for release workflows and then checks to see if they've ever passed. With --local, it could skip that later step.
  • SAST looks for workflows with CodeQL or SonarCloud and sees if they're run on PRs (or something like that?). With --local, it could simply reward for having a SAST workflow.
  • Security-Policy looks for a policy in the repo or the org's .github repo. With --local, it could just look in the repo.

In all of these cases, Scorecard should log that the results are potentially incomplete/unrepresentative of the upstream repository.

@pnacht pnacht added the kind/enhancement New feature or request label Jan 30, 2024
@spencerschrock
Copy link
Member

spencerschrock commented Jan 31, 2024

We do this in other places already ( see #3756 and #3415):

commits, err := c.SearchCommits(clients.SearchCommitsOptions{Author: "dependabot[bot]"})
if err != nil {
// TODO https://github.com/ossf/scorecard/issues/1709
// some repo clients (e.g. local) don't currently have the ability to search commits,
// but some data is better than none.
if errors.Is(err, clients.ErrUnsupportedFeature) {
return checker.DependencyUpdateToolData{Tools: tools}, nil
}
return checker.DependencyUpdateToolData{}, fmt.Errorf("dependabot commit search: %w", err)
}

With that in mind, these could be accomplished with similar means:

  • Fuzzing
  • License
  • Packaging
  • SAST
  • Security-Policy

However Maintained is a more complex issue. See #1709

@spencerschrock spencerschrock added the good first issue Good for newcomers label Jan 31, 2024
lharrison13 added a commit to lharrison13/scorecard that referenced this issue Nov 27, 2024
lharrison13 added a commit to lharrison13/scorecard that referenced this issue Nov 27, 2024
lharrison13 added a commit to lharrison13/scorecard that referenced this issue Nov 27, 2024
lharrison13 added a commit to lharrison13/scorecard that referenced this issue Nov 27, 2024
lharrison13 added a commit to lharrison13/scorecard that referenced this issue Nov 27, 2024
@lharrison13 lharrison13 linked a pull request Nov 27, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers kind/enhancement New feature or request
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

2 participants