vemonet/shapes-of-you

💠 An index for linked open data & standard knowledge descriptions (ontologies, vocabularies, shapes, queries, mappings)

Access standard knowledge indexed from code repositories, connected to the Linked Open Data access points

🖥 Access the web app at index.semanticscience.org

📬 Query our knowledge graph using the OpenAPI at grlc.io/api-git/vemonet/shapes-of-you/subdir/api (powered by grlc.io and SPARQL)

✨ Directly query the SPARQL endpoint on YASGUI at https://graphdb.dumontierlab.com/repositories/shapes-registry.

The SPARQL endpoint is also conveniently accessible in the webapp Active endpoints tab, since Shapes of You indexes its own SPARQL query files, and computes metadata for its SPARQL endpoint.

Shapes of you is a global index for semantically descriptive files published to public Git repositories (GitHub, GitLab, and Gitee), it enables semantic web enthusiast to connect those standard knowledge definitions to active Linked Open Data access points (SPARQL endpoints).

To be found by our indexer, make sure your repository description, or topics, on GitHub, GitLab, or Gitee includes one of the resources mentionned below, we automatically index files from public repositories every week on saturday at 1:00 GMT+1 🕐

• SHACL shapes: we index RDF files such as .ttl, .rdf, .jsonld, etc), with all sh:NodeShape they contain
• ShEx expressions: we index .shex files, and ShEx shapes defined in RDF files
• SPARQL queries: we index .rq and .sparql files, and parse grlc.io APIs metadata
• OWL ontologies: we index all RDF files with all owl:Class they contain
• SKOS vocabularies: we index all RDF files with all skos:Concept they contain
• RML mappings: we index RDF files, with all r2rml:SubjectMap and rml:LogicalSource they contain
• R2RML mappings: we index RDF files, with all r2rml:SubjectMap they contain
• CSVW metadata: we index RDF files, with all csvw:Column they contain
• Nanopublication templates: we index RDF files, with all nt:AssertionTemplates and inputs they contain
• OBO ontologies: we index all .obo files with all terms they contain
• OpenAPI specifications: we index .yml, .yaml and .json files, and parse the spec to retrieve API metadata
• DCAT datasets: we index RDF files, with all dcat:Dataset they contain

If your repository or endpoint is missed by our indexer:

• Additional GitHub repositories in the file EXTRAS_GITHUB_REPOSITORIES.txt

• Additional SPARQL endpoints in the file EXTRAS_SPARQL_ENDPOINTS.txt

Technical overview 🧭

This web service is composed of those 4 main parts, described more in details below:

• A python script to retrieve SPARQL queries, SHACL & ShEx Shapes files with some metadata from GitHub repositories. The retrieved data is defined using RDF.
  • A GitHub Actions workflow runs every week on saturday night to execute the python script, and publish the RDF output to the triplestore
• A React web app written in TypeScript, which displays the files and metadata from the SPARQL endpoint with filters, and search
  • The website is automatically deployed by GitHub Actions workflows to GitHub Pages at each push to the main branch.
  • We use expo to build this Progressive Web App (aka. PWA), it can be installed as a native app on any computer desktop (using Chrome is recommended), or smartphones.
• A triplestore with a publicly available SPARQL endpoint at https://graphdb.dumontierlab.com/repositories/shapes-registry
• A grlc.io powered OpenAPI to query the SPARQL endpoint at http://grlc.io/api-git/vemonet/shapes-of-you
  • Most SPARQL queries used by the webapp are also provided as API calls

---

Data model 📋

We defined and published a simple schema for our data as a OWL ontology, mainly re-using schema.org concepts.

Checkout the OWL ontology in website/assets/shapes-of-you-ontology.ttl 🦉

Here is an overview of the ontology (generated by gra.fo):

Prefixes

Just copy/paste this if you are missing some prefixes to query the Shapes of You knowledge graph:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX dcat: <http://www.w3.org/ns/dcat#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX sio: <http://semanticscience.org/resource/SIO_>
PREFIX schema: <https://schema.org/>
PREFIX sh: <http://www.w3.org/ns/shacl#>
PREFIX shex: <http://www.w3.org/ns/shex#>
PREFIX void: <http://rdfs.org/ns/void#>
PREFIX void-ext: <http://ldf.fi/void-ext#>
PREFIX sdm: <https://w3id.org/vocab/sdm#>
PREFIX r2rml: <http://www.w3.org/ns/r2rml#>
PREFIX rml: <http://semweb.mmlab.be/ns/rml#>
PREFIX nt: <https://w3id.org/np/o/ntemplate/>
PREFIX csvw: <http://www.w3.org/ns/csvw#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

Classes

• "Shape" files: schema:SoftwareSourceCode
  • Properties:
    • dcterms:hasPart
    • rdfs:comment
    • schema:codeRepository > schema:DataCatalog
  • Subclasses:
    • sh:Shape (SHACL shape)
    • shex:Schema (ShEX schema)
    • sh:SPARQLFunction (SPARQL query) - additional properties: void:sparqlEndpoint, schema:query
    • owl:Ontology (OWL ontology)
    • skos:ConceptScheme (SKOS vocabulary)
    • sio:000623 (OBO ontology)
    • schema:APIReference (OpenAPI)
    • rml:LogicalSource (RML and YARRRML mappings)
    • r2rml:TriplesMap (R2RML mappings)
    • nt:AssertionTemplate (Nanopublication templates)
    • dcat:Dataset (DCAT datasets)
• Git repositories: schema:DataCatalog
  • Properties:
    • rdfs:comment
• Active SPARQL endpoints:schema:EntryPoint

---

Run the web app 🛩️

Requirements: npm and yarn installed.

In development 🏗

Clone the repository:

git clone https://github.com/vemonet/shapes-of-you
cd shapes-of-you

Install dependencies 📥

yarn

Run the web app on http://localhost:19006, it should reload automatically at each changes to the code 🔃

yarn dev

Upgrade the packages versions in yarn.lock 🔒

yarn upgrade

In production 🌍

This website is automatically deployed by a GitHub Actions workflow to GitHub Pages which is accessed from http://index.semanticscience.org

You can also build locally in the /web-build folder and serve on http://localhost:5000 (checkout the Dockerfile)

yarn build
yarn serve

Deploy the backend

Deploy the Oxigraph triplestore and ElasticSearch index using Docker 🐳 (requires docker installed)

1. Make sure the folder for ElasticSearch has the right permissions

mkdir -p /data/shapes-of-you/elasticsearch
sudo chown -R 1000:0 /data/shapes-of-you/elasticsearch

2. Deploy the stack

docker-compose up -d

Checkout the docker-compose.yml file to see how we run the Docker image.

---

⛏️ Index structured and semantic files

Requirements: Python 3.6+, git

🗃️ Index files from code repositories

This script is run every day by the mighty .github/workflows/index-shapes.yml workflow

The Python script retrieves shapes files from various popular Git services API (GitHub GraphQL API, GitLab API , Gitee API), and generates RDF data. The RDF data is then automatically published to the publicly available triplestore by the GitHub workflow.

You can find the python scripts and requirements in the etl folder.

Use this command to locally define the API_GITHUB_TOKEN, GITLAB_TOKEN and GITEE_TOKEN environment variables required to run the script (you might need to adapt on Windows, but you should know better than me):

export API_GITHUB_TOKEN=MYGITHUBTOKEN000
export GITLAB_TOKEN=MYGITLABTOKEN000
export GITEE_TOKEN=MYGITEETOKEN000

Add those commands to your .zshrc or .bashrc to make it permanent

For GitHub you can create a new GitHub API key (aka. personal access token) at https://github.com/settings/tokens

Go to the etl folder:

cd etl

Install the requirements:

pip install -e .

Retrieve shapes files from search the GitHub GraphQL API (you can also use a topic to search, e.g. topic:sparql):

python3 main.py github vemonet/shapes-of-you

Retrieve shapes files from GitLab API using the python-gitlab package:

python3 main.py gitlab sparql

Retrieve shapes files from Gitee API:

python3 main.py gitee ontology

✨ Generate SPARQL endpoints metadata

This task is performed every day by the swifty .github/workflows/analyze-endpoints.yml workflow

We use the d2s tool (aka. data2services) to generate HCLS metadata for a SPARQL endpoint:

pip install d2s
d2s metadata analyze https://graphdb.dumontierlab.com/repositories/shapes-registry -o metadata.ttl

We commit the generated metadata file to the metadata branch, to experiment using git to version and keep track of changes of the metadata generated for the SPARQL endpoints over time.

Enable Virtuoso Linked Data Platform

Enable WebDAV LDP on Virtuoso 7 (from the official Virtuoso documentation)

Start the virtuoso-opensource-7 docker image

docker-compose up -d

The first time you start Virtuoso, or after you reset the database, you will need to run this script to prepare the Linked Data Platform:

./prepare_virtuoso.sh

To prepare for shapes-of-you, create folders github, gitlab, gitee, apis and endpoints using the same owner and permission as for the ldp folder.

Test by uploading a turtle file to the LDP (change the password before):

curl -u ldp:$ENDPOINT_PASSWORD --data-binary @shapes-rdf.ttl -H "Accept: text/turtle" -H "Content-type: text/turtle" -H "Slug: test-shapes-rdf" https://data.index.semanticscience.org/DAV/home/ldp/github

Enable CORS to query the Virtuoso SPARQL endpoint from JavaScript. See the Virtuoso CORS documentation.

• Go to Web Application Server > Virtual Domains & Directories
• Expand Interface for the Default Web Site
• Locate the /sparql Logical Path > click Edit
• Enter \* in the Cross-Origin Resource Sharing input field.

👩‍💻 Contribute

Contributions are welcome! See the guidelines to contribute.

🤝 Acknowledgements

RDF data hosted in a Oxigraph triplestore (open source)

OpenAPI powered by grlc.io

SPARQL query UI powered by Triply's YASGUI

Ontology built with gra.fo

Data processing workflows run for free using GitHub Actions open source plan

Files parsed using python libraries: rdflib, obonet, prance

shapes-of-you's Issues

Add a new page for each file

Describe the bug/improvement

When clicking on a file, it opens a new page given more informations about the file content and metadata:

• Retrieve the file metadata and concepts in our graph
• Download the file, parse it with JavaScript, and display additional relevant infos? e.g. metadata, classes, properties, instances, subClassOf hierarchy...

Implement SOLID

• Implement SOLID login
• Store user preferences to their SOLID pod (star shapes)
• Retrieve user preferences from their SOLID pod
• Enable user to choose to make their user preferences private (default) or public

Checkout the SOLID React SDK docs: https://docs.inrupt.com/developer-tools/javascript/react-sdk/

Seems only to be UI components to edit the SOLID user profile

Try: https://github.com/solid/react-components (and https://github.com/solid/query-ldflex)

solid.data['https://ruben.verborgh.org/profile/#me']['https://maastrichtu-ids.github.io/shapes-of-you#preferences'] = 1

Replace Like by Star? https://github.com/solid/react-components#-social-interactions

Use this URI to store preferences in SOLID pod: https://maastrichtu-ids.github.io/shapes-of-you#preferences

Example to create a custom Star button:

• https://github.com/solid/react-components/blob/master/src/components/LikeButton.jsx
• https://github.com/solid/react-components/blob/master/src/components/ActivityButton.jsx

star.tsx:

import { customActivityButton } from './ActivityButton';

/** Button to view and perform a "Star" action on an item. */
export default customActivityButton('ShapeStar', 'Star', 'You starred', 'Starred');

Then star it:

<Star object="https://shape-file-url">Star Icon</Star>

Fix indexing issues

python3 etl/index_shapes.py gitlab shacl-shapes

Found 3 relevant SPARQL endpoints, but encoding error

Returns:

INFO:[04/19/2021, 09:55:12] SPARQL endpoint failed: http://digital-agenda-data.eu/data/sparqlbyte indices must be integers or slices, not str
INFO:[04/19/2021, 09:55:21] SPARQL endpoint failed: http://el.dbpedia.org/sparqlEndPointInternalError: endpoint returned code 500 and response. Response:b'Virtuoso 42000 Error The estimated execution time 2928956 (sec) exceeds the limit of 900 (sec).\n\nSPARQL query:\nSELECT * WHERE { ?s ?p ?o } LIMIT 10'
INFO:[04/19/2021, 09:55:25] SPARQL endpoint failed: http://cr.eionet.europa.eu/sparqlbyte indices must be integers or slices, not str

Extract metadata from README files

Describe the improvement

For each git repo try to extract metadata from the readme

Relevant links

https://github.com/KnowledgeCaptureAndDiscovery/somef

Add void files which contain alignments

Hello @vemonet , great stuff! Here is a small suggestion, that maybe you had already in mind.

Describe the improvement

It would be great to also have alignments between KG using void vocabulary. This could also be by looking up the endpoints for .well-known/void files

Relevant links

• An example of void file at https://270a.info/void.ttl

Corresponding concept

void:Dataset; void:Linkset

Report what tests have been run on a resource

Reporting of tests done

It would be really nice if the GUI would report what tests have been run on a resource, similar to what YummyData does, so that we can learn how resources can be made more FAIR.

Extract if service call in a SPARQL query

To know which queries are querying "outside"

Be careful as most Wikidata queries uses service calls by default

Supports shapes in JSON-LD

https://github.com/RDFLib/rdflib-jsonld

from rdflib import Graph, plugin
from rdflib.serializer import Serializer

g = Graph().parse(data=testrdf, format='json-ld')

e.g. https://github.com/INCF/neuroshapes

Improve metadata extracted from RDF files

Describe the improvement

Improve metadata extracted from RDF files:

• Catch and test all void:sparqlEndpoint
• Check for SPARQL services descriptions
• Better support of VOID
• Support BioSchemas

Relevant links

• https://bioschemas.org/
• VOID: https://www.w3.org/TR/hcls-dataset/

Use git clone instead of getting file content from GraphQL API

GraphQL API does not allow to recursively iterate over all files and folders in a repo (and returning all files content is quite intense)

Solution: git clone each repo, and iterate over folders and files locally

e.g. https://github.com/INCF/neuroshapes

Categorize resources

Describe the bug/improvement

Filtering/categorizing files, repositories and SPARQL endpoints based on categories?

Perform community detection?

duplicate info in the README

duplicate info

See this screenshot:

Relevant links

• https://github.com/vemonet/shapes-of-you/blame/main/README.md#L30-L32

Improve scalability of the user interface

Describe the bug/improvement

The user interface currently load the 60k+ files in a simple JSON file, which can slow or freeze the user computer

To be able to properly scale it would be interesting to look into using a local database, the IndexedDB w3c recommendation and Dexie JS seems to be good candidates

Relevant links

• IndexedDB 2.0 recommendation: https://www.w3.org/TR/IndexedDB-2/
• IndexedDB 3.0 draft: https://w3c.github.io/IndexedDB/
• Dexie JS: https://dexie.org/
• Using local storage in PWA: https://www.monterail.com/blog/pwa-offline-dynamic-data

Index mappings files (sssom)

Parse files from the "Simple Standard for Sharing Ontology Mappings (SSSOM)": https://mapping-commons.github.io/sssom/usecases/

Implement better filtering

Improve repositories and shapes files display and filtering:

• Add faceted search with categories? (biomedical, organization, etc)
• Group shapes files under their respective github repository?
• Enable to sort files by date of last update?

Make it a React component easy to reuse in other apps?

We have a use-case with multiple types of inputs for the different facets (checkboxes for file types, MaterialUI autocomplete for repositories, text box for full text search), and we want the options (e.g. filter displayed repositories based on the search input)

We could not find a good dynamic faceted search, the most relevant options for React are:

• ebi-gene-expression-group/scxa-faceted-search-results: multiple inputs faceted search, but does not seems to support dynamic filtering of displayed options
• eduardobcastro/in-memory-faceted-search

Currently we prefer to implement it ourself (search for "faceted" in the SemanticIndex page). But it could be tricky when we will start to have more categories to filter on.

Use Virtuoso instead of GraphDB as triplestore

Use Virtuoso webDAV to easily upload RDF files through HTTP.

We might need to play around with the configuration as Virtuoso current version (7) does not include LDP capabilities out-of-the-box

• LDP docs: http://vos.openlinksw.com/owiki/wiki/VOS/VirtLDP#What%20is%20the%20Linked%20Data%20Platform%20%28LDP%29%3F
• cf. discussion to solve it https://community.openlinksw.com/t/permissions-on-ldp-containers-and-resources/290/9
• GitHub issue: openlink/virtuoso-opensource#466
• Download VAD packages: http://download3.openlinksw.com/index.html?prefix=uda/vad-vos-packages/7.2/

GitHub repo with some instructions to enable LDP in Virtuoso: https://github.com/markwilkinson/to_build_virtuoso_ldp
With the corresponding image on DockerHub: https://hub.docker.com/r/markw/ldp_server

Note: delete a file from the webDAV with iSQL

select DB.DBA.DAV_DELETE ('/DAV/home/dba/rdf_sink/gitee.ttl', 0, 'dba', 'dba');

Trying with current Virtuoso deployment

Create a folder:

curl -iX PUT -u dav:password -H 'Content-Type: text/turtle' https://data.index.semanticscience.org/DAV/home/dav/rdf_sink/test-folder

Upload with HTTP:

curl --verbose -iX PUT -F "[email protected]" -u dav:password -H 'Content-Type: text/turtle' https://data.index.semanticscience.org/DAV/home/dav/rdf_sink/shapes-of-you.ttl

Error (even if the prefix is properly define in the file):

<p>TURTLE RDF loader, line 6: SP029: TURTLE RDF loader, line 6: Undefined namespace prefix at ns1:EntryPoint</p>%