Giter Site home page Giter Site logo

softvis-research / jqa-githubissues-plugin Goto Github PK

View Code? Open in Web Editor NEW
1.0 3.0 3.0 117 KB

This is a GitHub issue scanner for jQAssistant. It enables jQAssistant to scan and analyze GitHub issues.

License: GNU General Public License v3.0

Java 98.90% HTML 0.34% Shell 0.75%
java github-issues jqassistant neo4j software-analytics

jqa-githubissues-plugin's Introduction

jQAssistant GitHub-Issues Plugin

GitHub license Build Status codecov

This is a GitHub issue scanner for jQAssistant. It enables jQAssistant to scan and analyze GitHub issues.

Getting Started

Download the jQAssistant command line tool for your system: jQAssistant - Get Started.

Next download the latest version from the release tab. Put the jqa-githubissues-plugin-*.jar into the plugins folder of the jQAssistant commandline tool.

Create a file named githubissues.xml. The plugin can scan multiple repositories owned by different users. Please note that the GitHub REST-API requires login credentials to access any of its functions. Therefore, login credentials must be provided per repository.

<github-issues-configuration>
    <github-repository>
        <user>github-user</user>
        <name>github-repository-name</name>

        <credentials>
            <user>authentication-user</user>
            <password>authentication-password</password>
        </credentials>
    </github-repository>

    <github-repository>
        ...
    </github-repository>
</github-issues-configuration>

Now scan your configuration and wait for the plugin to finish:

jqassistant.sh scan -f githubissues.xml

You can then start a local Neo4j server to start querying the database at http://localhost:7474:

jqassistant.sh server

Labels

The GitHub-Issues plugin uses the following labels in the resulting graph:

Label Description ID
GitHub-Issues-Configuration-File A configuration file for the plugin. -
GitHub Parent label for all nodes related to the GitHub-Issues plugin. -
Repository Represents a GitHub Repository. "repo-user/repo-name"
Issue Represents a GitHub Issue. "repo-user/repo-name#issue-number"
Milestone Represents a GitHub Milestone which is a collection of Issues. "repo-user/repo-name#milestone-id"
Comment Represents a Comment under a GitHub Issue. -
PullRequest Every PullRequest is an Issue, but not every Issue is a PullRequest. "repo-user/repo-name#issue-number"
User Represents a GitHub User. "user-name"
Commit Represents a GitHub Commit. "repo-user/repo-name#commit-sha"

Here are the possible relations between those labels:

(GitHub-Issues-Configuration-File)  -[:SPECIFIES_REPOSITORY]    ->  (Repository)

(Repository)    -[:HAS_ISSUE]         ->    (Issue)
(Repository)    -[:HAS_MILESTONE]     ->    (Milestone)

(Issue)         -[:HAS_LABEL]         ->    (Label)
(Issue)         -[:HAS_COMMENT]       ->    (Comment)
(Issue)         -[:HAS_ASSIGNEE]      ->    (User)
(Issue)         -[:CREATED_BY]        ->    (User)
(Issue)         -[:IS_PART_OF]        ->    (Milestone)

(PullRequest)   -[:HAS_LAST_COMMIT]   ->    (Commit)

(Milestone)     -[:CREATED_BY]        ->    (User)

(Comment)       -[:FOLLOWED_BY]       ->    (Comment)
(Comment)       -[:CREATED_BY]        ->    (User)

(Issue|Comment) -[:REFERENCES_ISSUE]  ->    (Issue)
(Issue|Comment) -[:REFERENCES_COMMIT] ->    (Commit)
(Issue|Comment) -[:REFERENCES_USER]   ->    (User)

Use Cases

Overview

List all your open Issues over multiple repositories:

MATCH
    (r:Repository)-[:HAS_ISSUE]->(i:Issue {state:"open"})
RETURN
    r.repositoryId, i.title, i.body

Count open Issues per repository:

MATCH
    (r:Repository)-[:HAS_ISSUE]->(Issue {state:"open"})
RETURN
    r.repositoryId, count(*) AS issueCount
ORDER BY
    issueCount DESC

List open issues per user:

MATCH
    (Issue {state:"open"})-[:HAS_ASSIGNEE]->(u:User)
RETURN
    u.login, count(*)

Issue quality

Show issues without description:

MATCH
    (i:Issue)
WHERE
    i.body = ""
RETURN
    i.issueId, i.title

Show issues without labels:

MATCH 
    (i:Issue)
WHERE 
    NOT (i:Issue)-[:HAS_LABEL]->()
RETURN
    i.title, i.issueId

Show issues ordered descending by the amount of comments:

MATCH 
    path=((i:Issue)-[:HAS_COMMENT]->()-[:FOLLOWED_BY*]->())
RETURN
    i.title, i.issueId, length(path) AS pathLength, i.state
ORDER BY
    pathLength DESC, i.state DESC

Show durations it needed to resolve an issue:

WITH
    issue, duration.inDays(date(issue.createdAt), date(issue.updatedAt)).days AS duration
RETURN 
    issue.issueId, issue.title, duration + " days" AS timToSolve
ORDER BY
    duration DESC

Show issues older than 1 month that are still open:

MATCH
    (issue:Issue {state:"open"})
WHERE
    date(issue.createdAt) <= date('20180713')
RETURN 
    *

Why are these issues still open?

Let's have a look at a few indicators:

  • Do these Issues have labels?
MATCH
    (issue:Issue {state:"open"})
WHERE
    date(issue.createdAt) <= date('20180713') AND NOT (issue:Issue)-[:HAS_LABEL]->()
RETURN 
    *

→ If not, then probably no one looked at these issues.

  • Is anyone assigned to this issue?
MATCH
    (issue:Issue {state:"open"})
WHERE
    date(issue.createdAt) <= date('20180713') AND NOT (issue:Issue)-[:HAS_ASSIGNEE]->(:User)
RETURN 
    issue

→ If not, then probably no one feels responsible for this issue.

Known Problems

Performance

The performance of the plugin is limited by the GitHub REST API. We are not allowed to make more than one request per second. See this for more information. For each Issue and every Comment the body text gets parsed to HTML by this endpoint. We need to do this to resolve references to User, Commits and Commits in the markdown texts.

That is why a analysis of ~1800 issues can take a few hours.

Only one configuration file

At the moment only one configuration file is supported. When you scan more than one at a time nodes representing the same real world entity won't be identified. Furthermore, the plugin doesn't print a warning so be careful to avoid wrong analyses of your repositories!

Did you find a bug?

Please have a look at the issue section in GitHub. If you can't find your bug open a ticket with an reproducible example and your error logs.

Contribution

If you want to contribute here are a few tips to get you started:

Build the GitHub-Issues plugin:

cd plugin

# Build a fat-JAR
mvn clean package

# Copy the resulting JAR into the jQAssistant CLI plugins folder
cp target/jqa-githubissues-plugin-0.1-jar-with-dependencies.jar ../run/jqassistant-commandline-neo4jv3-1.4.0/plugins/

Run code coverage via Corbertura:

mvn cobertura:cobertura

The coverage reports can be found under target/site/cobertura.

jqa-githubissues-plugin's People

Contributors

b-pos465 avatar dirkmahler avatar rmllr avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

jqa-githubissues-plugin's Issues

Scanning Lombok fails at issue #59.

Scanning the lombok repository leads to a bug. Here is the configuration:

<github-repository>
      <user>rzwitserloot</user>
      <name>lombok</name>

      <credentials>
          <user>...</user>
          <password>...</password>
      </credentials>
</github-repository>

And here is the error log:

2018-09-04 21:23:03.471 [main] INFO GraphBuilder - Importing issue: rzwitserloot/lombok#59, "[i769] changed ValueHandler for @Value: AllArgsConstructor to"
2018-09-04 21:23:04.917 [main] ERROR ScannerImpl - Unexpected problem encountered while scanning: item='test-project/target/test-project-1.0-SNAPSHOT.jar!githubissues.xml', path='/githubissues.xml', scope='CLASSPATH', pipeline='[com.buschmais.jqassistant.plugin.common.impl.scanner.FileResourceScannerPlugin@12b34bd5, org.jqassistant.contrib.plugin.githubissues.scanner.GitHubIssueScannerPlugin@596343e7]'. Please report this error including the full stacktrace (continueOnError=true).
java.lang.ClassCastException: com.sun.proxy.$Proxy130 cannot be cast to org.jqassistant.contrib.plugin.githubissues.model.GitHubPullRequest
        at org.jqassistant.contrib.plugin.githubissues.scanner.GraphBuilder.issueLevel(GraphBuilder.java:150)
        at org.jqassistant.contrib.plugin.githubissues.scanner.GraphBuilder.repositoryLevel(GraphBuilder.java:106)
        at org.jqassistant.contrib.plugin.githubissues.scanner.GraphBuilder.startTraversal(GraphBuilder.java:65)
        at org.jqassistant.contrib.plugin.githubissues.scanner.GitHubIssueScannerPlugin.scan(GitHubIssueScannerPlugin.java:117)
        at org.jqassistant.contrib.plugin.githubissues.scanner.GitHubIssueScannerPlugin.scan(GitHubIssueScannerPlugin.java:26)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:82)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:59)
        at com.buschmais.jqassistant.plugin.common.api.scanner.AbstractContainerScannerPlugin.scan(AbstractContainerScannerPlugin.java:50)
        at com.buschmais.jqassistant.plugin.common.api.scanner.AbstractContainerScannerPlugin.scan(AbstractContainerScannerPlugin.java:29)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:82)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:59)
        at com.buschmais.jqassistant.plugin.common.api.scanner.AbstractZipArchiveScannerPlugin.scan(AbstractZipArchiveScannerPlugin.java:51)
        at com.buschmais.jqassistant.plugin.common.api.scanner.AbstractZipArchiveScannerPlugin.scan(AbstractZipArchiveScannerPlugin.java:21)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:82)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:59)
        at com.buschmais.jqassistant.plugin.common.impl.scanner.FileScannerPlugin.scan(FileScannerPlugin.java:34)
        at com.buschmais.jqassistant.plugin.common.impl.scanner.FileScannerPlugin.scan(FileScannerPlugin.java:20)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:82)
        at com.buschmais.jqassistant.core.scanner.impl.ScannerImpl.scan(ScannerImpl.java:59)
        at com.buschmais.jqassistant.commandline.task.ScanTask.scan(ScanTask.java:128)
        at com.buschmais.jqassistant.commandline.task.ScanTask.executeTask(ScanTask.java:79)
        at com.buschmais.jqassistant.commandline.task.AbstractStoreTask.run(AbstractStoreTask.java:49)
        at com.buschmais.jqassistant.commandline.Main.executeTask(Main.java:254)
        at com.buschmais.jqassistant.commandline.Main.interpretCommandLine(Main.java:206)
        at com.buschmais.jqassistant.commandline.Main.run(Main.java:92)
        at com.buschmais.jqassistant.commandline.Main.main(Main.java:63)
2018-09-04 21:23:04.919 [main] INFO ScannerImpl - Continuing scan after error. NOTE: Data might be inconsistent.

The bug can't be reproduced by just importing issue #59. Furthermore, pull requests from other repositories work as expected.

Feedbackliste - 24.08.2018

Folgende Punkte wurden beim letzten Gespräch angemerkt:

  • GitHub Descriptor auseinanderziehen: 1 Deskriptor für Konfiguration und ein Super-Label für alle Knoten.
  • POINTS_AT ersetzen durch REFERENCES_ISSUE, REFERENCES_COMMIT, REFERENCES_USER.
  • Modell: Gemeinsame Attribute in eigenes Interface extrahieren. (createdBy, createdAt, etc.)
  • Maven Projekt in Repository-Wurzel legen.
  • Maven Ordner-Pfad: org.jqassistant.contrib.plugin.githubissues.
  • Static Variablen vermeiden.
  • RestTool: URL im Kontruktor übergeben.
  • Singeltons vermeiden.
  • StoreTool: Bennenung
  • FileReader ersetzen durch IOUtils.readFile o.ä. apache commons.
  • Maven Shade Plugin statt Maven Assembly Plugin.
  • Plugin über Maven testen, z.B. JUnit Demo Projekt.
  • Dokumentation: Klassen müssen noch dokumentiert werden.
  • Editor config aus jqa-javasrc-plugin verwenden.
  • SpotBugs verwenden.
  • Dependecy Injection über Konstruktor (GraphBuilder Store, RestTool, etc. übergeben. Modell erst bei Methodenaufruf.)

Folgende Punkte wurden bewusst nicht umgesetzt:

Scanner kann initialize() enthalten für Cache, RestTool, etc.

Es ergibt vermutlich für keine der Komponenten Sinn sie im Scope des Scanners zu definieren. Dieser Punkt muss noch diskutiert werden. Vielleicht wäre eine Persistenz des Caches sinnvoll. Sollen mehrer Konfigurationsfiles erlaubt werden? Aktuell wird dies nicht unterstützt.

URLs mit JaxRS zusammenbauen (keine String Konkatinierung) und Resourcen wiederverwenden.

Ich habe keine Variante via JAX-RS gefunden?

DescriptorCache: IDs als Objekte statt konkatinierte Strings

Siehe branch dev/descriptor-ids.

Support the GitHub REST API pagination.

All rest endpoints of the GitHub REST API use Pagination.

The link pointing at the next pages are stored inside a header. Here is the GitHub documentation for this feature.

The plugin retrieves only the first 30 entries for every entity so far.

Gitlab Issue Plugin

Hey, I am applying for GSOC and wondering if the same can be developed for Gitlab etc? I dont think that will be much change to do.

Plugin prevents jQAssistant from starting

Unfortunately this plugin doesn't work at all at this time.
Here's the output you will get with ./bin/jqassistant.sh scan -f githubissues.xml:

2020-02-04 20:03:42.410 [main] INFO PluginConfigurationReaderImpl - Loaded jQAssistant plugins [CDI, Common, Core Analysis, Core Report, EJB3, GitHub-Issues, GraphML, JAX-RS, JPA 2, JSON, JUnit, Java, Java 8, Java EE 6, Maven 2 Repository, Maven 3, OSGi, RDBMS, Spring, TestNG, Tycho, XML, YAML]. 2020-02-04 20:03:42.817 [main] INFO StoreFactory - Connecting to store at 'file:/home/adrian/Downloads/jqassistant-commandline-neo4jv3-1.7.0/jqassistant/store' Exception in thread "main" com.buschmais.xo.api.XOException: com.buschmais.xo.neo4j.embedded.api.EmbeddedNeo4jXOProvider specified as XO provider must implement com.buschmais.xo.spi.bootstrap.XODatastoreProvider at com.buschmais.xo.impl.XOManagerFactoryImpl.<init>(XOManagerFactoryImpl.java:46) at com.buschmais.xo.impl.bootstrap.XOBootstrapServiceImpl.createXOManagerFactory(XOBootstrapServiceImpl.java:39) at com.buschmais.xo.api.bootstrap.XO.createXOManagerFactory(XO.java:49) at com.buschmais.jqassistant.core.store.impl.AbstractGraphStore.start(AbstractGraphStore.java:50) at com.buschmais.jqassistant.commandline.task.AbstractStoreTask.run(AbstractStoreTask.java:55) at com.buschmais.jqassistant.commandline.Main.executeTask(Main.java:259) at com.buschmais.jqassistant.commandline.Main.executeTasks(Main.java:208) at com.buschmais.jqassistant.commandline.Main.interpretCommandLine(Main.java:197) at com.buschmais.jqassistant.commandline.Main.run(Main.java:79) at com.buschmais.jqassistant.commandline.Main.main(Main.java:50)

Tested on

  • macOS, Java 11, jQAssistant 1.7, plugin v0.0.4
  • macOS, Java 11, jQAssistant 1.6, plugin v0.3
  • Linux, Java 13, jQAssistant 1.7, plugin v0.0.4
  • Linux, Java 13, jQAssistant 1.6, plugin v0.3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.