uniba-mi/quare

QuaRe is a tool for testing whether GitHub repositories of interest comply with certain quality criteria that they should fulfill according to the project type.

quare's Issues

Unable to run the Quare app directly via "docker compose up"

Hello, I came to test this software after reading your paper: "Specification and Validation of Quality Criteria for Git Repositories using RDF and SHACL"

The README states that we can launch the app via docker compose up after cloning the repository. In my experience (OS: Ubuntu 20.04), two additional steps are needed:

1. In the frontend directory, run npm install to install all the dependencies.
2. In the backend directory, run chmod +x backend_api.py to add execution permission.

Afterwards, docker compose up works correctly.

Thank you!

Recognize all keywords per heading in process_readme_sections

If a heading contains multiple relevant keywords, currently only one of those is recognized. After that, the next heading is checked. For example in "Installation and usage", only the installation part is recognized.
→ Remove "continue" from the for-loop in process_readme_sections.

Augment repository representation to facilitate validation

Extend the file shacl_validator.py so that the repository representation then includes all necessary information for the new FAIR project type.

Update Python version in Dockerfile and use match-case syntax in shacl_verbalizer.py

1. Switch to the newest Python version in the Dockerfile.
2. Replace the if-elif-else section of shacl_verbalizer.py with the new match-case syntax.

Evaluate SHACL severity values

According to the specification, "shapes can specify one value for the property sh:severity in the shapes graph" (https://www.w3.org/TR/shacl/#severity).
Check if this is useful for QuaRe and how pySHACL handles this information.

Remove OWL part from QuaRe

In this milestone 1 (Implement FAIR Project Type for Validation), the FAIR project type was added only to the SHACL part of QuaRe. Moreover, as discussed in the paper (cf. p. 5), SHACL is better suited for the overall use case than OWL. Therefore, the OWL part should be removed from QuaRe.

Understand and detail FAIR best practices

Understanding and detailing the "Research Software Best Practices" presented in the poster by A. Iglesias-Molina and D. Garijo.
--> What exactly has to be present/absent in a GitHub repository so that it fulfills these criteria?

Validating a repository without a license leads to an internal error and no result

The validation of (public and private) repositories without a license file returns no results in the frontend. Instead, the spinner in the grey result button contiunes to spin.
At the same time, the terminal where Docker Compose is running, shows this error message (here shortened):

File "/src/shacl_validator.py", line 104, in create_repository_representation
license = repo.get_license()
. . .
github.GithubException.UnknownObjectException: 404 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest/licenses/licenses#get-the-license-for-a-repository"}

Test implementation of FAIR project type with famous repositories

Test the new project type with e.g., the repositories with the most stars on GitHub.
This issue has two purposes:

1. Compare the results of QuaRe with those from a manual check.
2. Get an insight into how well the repositories fulfill the FAIR principles.

Model FAIR principles as SHACL rules

In the file shacl_project_shapes.ttl, extend the type "FAIRSoftware" with the rules defined in issue #10.

Check if comments can/should be added to the file.

Show share of fulfilled criteria in all criteria

After validation, show the total number of criteria for the project type and how many of them were met by the repository.

Visualise ontology with mermaid-js

Use mermaid-js to visualise the repository ontology (all possible node and edge types) in the README file.

Running benchmark.py leads to a ZeroDivisionError

The following error occurs when you run the benchmarks as explained in the readme file:

"ZeroDivisionError: float division by zero"

The variable total is 0, so there seems to be a problem with the values of all_finished and all_interal on which total is based.

Add OR-cases to existing project types

For the FAIR project type, often alternatives were modelled. For example, usage documentation is present if there is a section "Usage", "How to use", "Manual" or "User manual" in the README file.
The other project type definitions only have one possible value for each property. --> Extend them with reasonable values.

Use re-rendering for specification page with Markdown

The quality criteria on the specification page originate from Markdown content in the descriptions of node and property shapes. All descriptions contain tables that need to be formatted. For this, a CSS file is used because Svelte ignored selectors like table in <style> tags. This is because these tags are added at runtime.
Use re-rendering so that the style can be defined in the file SpecificationPage.svelte.

Write integration/system tests for the shacl_validator with the shapes graph

Use for example pytest to test the shacl_validator with the SHACL specifications (focus on FAIR project type). The main goal is to find possible errors in the specification.

Evaluate LLMs for report verbalization

Adapt verbalized explanation for FAIR project type

For example, a "Constraint Violation in OrConstraintComponent" does not contain a result path and therefore does not lead to an explanation. → Investigate new possible violations and adapt shacl_verbalizer.py accordingly.

Use properties from common ontologies instead of custom ones

Evaluate which ontologies/vocabularies fit best. Next, use them in the repository representation and SHACL shapes graph.

Improve validation report verbalization on validation page using sh:message

Evaluate SHACL sh:message for verbalization.

https://www.w3.org/TR/shacl/#message

Limit repository representation to requirements of respective project type

At the moment, the repository representation is always constructed completely, independent of how many attributes are important for the project type. This increases the time until the validation result is presented to the user.

Therefore, the required methods / code words should be mentioned in the SHACL files. In the Python code, they should be mapped to functions via globals or dictionaries.

Find a way to improve criteria descriptions on specification page