vocol/vocol

An integrated environment to support collaborative ontology / vocabulary development in distributed settings

VoCol - Vocabulary collaboration and build environment.

Inspired by agile software and content development methodologies, the VoCol methodology and tool environment allows building leight-weight ontologies using version control systems such as Git and repository hosting platforms such as Github. VoCol is implemented without dependencies on complex software components, it provides collaborators with comprehensible feedback on syntax and semantics errors in a tight loop, and gives access to a human-readable presentation of the vocabulary. The VoCol environment is employing loose coupling of validation and documentation generation components on top of a standard Git repository. All VoCol components, even the repository engine, can be exchanged with little effort.

Installation on a local machine or on a Web Server

The following steps are needed to setup the VoCol Environment either on a local machine or a web server. These steps are valid in the Linux-based operating systems and with slight modifications can be used in Windows-based as well.

1. You should have the following libraries installed: Java JDK, NodeJS, NPM, and Git packages with their respective versions or higher. For more info see in Required libraries and tools.

2. Create a new directory e.g. "newFolder", clone the VoCol repository, and give the execution permissions as follows:

mkdir newFolder
cd newFolder
git clone https://github.com/vocol/vocol.git
chmod u+x  .

4. Enter inside the "VoCol" folder and run the following script to clean up any not necessary file:

cd vocol
./helper/scripts/resetApp.sh

5. Install the dependent packages (assuming that node package manager is installed already):

sudo npm install

Semantic-Ui framework is used in VoCol development, a couple of selections need to be given while installing it. Select "Skip install" as follows:

? It looks like you have a semantic.json file already.
  Yes, extend my current settings.
> Skip install

Then "Yes" for that Vocol is using "NPM Nice".

? We detected you are using NPM Nice! Is this your project folder? D:\vocolrepo\vocol
> Yes
  No, let me specify

Finally, give "public/semantic" as the location of Sematic-Ui in VoCol Project.

? Where should we put Semantic UI inside your project? (semantic/) public/semantic/

6. The last step is to start VoCol with npm start [VocolPortNumber] [SparqlEndPointPortNumber]. In the following command, we are going to start Vocol on port 3000 where Fuseki Server is runing at port 3030

npm start 3000 3030

8. You can access VoCol start page with http://localhost:3000 , if the port number was not changed. If you clear old data as step 4 describes, then the configuration page will be displayed. Otherwise, you can use http://localhost:3000/config URL for configuring of the VoCol. Sometimes, the port number is also changed during our project's development, for that, you have a possibility to look-up the vocol access's port number and as well change it, by opening bin/www file if you are on the root path of VoCol.

9. To keep your repository synchronized with VoCol instance (for example when you push something), you should configure a webhook path on the repository hosting platform such as Github, GitLab and BitBucket to point with the VoCol API: http(s)://hostname(:port or /vocolInstancePath)/listener. The connection between both hosting server and VoCol instance should be available in such a way that hosting platform can send the notification to the VoCol instance. Please the fundamental explanations of WebHooks in the following link: https://developer.github.com/webhooks/.

For more details about VoCol repository, please have a look on our VoColWiki.

Check out a list of projects that are currently using VoCol.

Moreover, you can use the docker image of VoCol here or use the included Dockerfile to build the docker image.

License

VoCol is licensed under the MIT License. See LICENSE.txt for more details. For respective licenses of individual components and libraries please refer to the Required libraries and tools section.

Current Work

We have extened this VoCol version to work As A Service in VoCoREG. Please, visit us on the following link: https://www.vocoreg.org or https://www.vocoreg.com

vocol's Issues

Another alternative to consider: LODE

@kurzum thanks for pointing me to https://github.com/NLP2RDF/ontologies/blob/master/publish.sh. Indeed LODE is a great documentation tool to consider. As we do not yet have a web server in place I modified the invocation so as to upload a file to the server where LODE is running, but actually for our ontology (sources at https://github.com/mobivoc/mobivoc/), LODE's documentation generated from the uploaded file is empty, and right now I do not have time for debugging.

@jerdeb @badmotor the bad thing about LODE is that it relies on the ontology to be published somewhere, or uploaded, as it fetches it using JavaScript. Whereas SpecGen generates static HTML, which, I think, would fit better into an automated workflow. (However I checked the LODE sources; it should be possible to modify them.)

Parrot seems to be more promising: it allows for file upload and gives you a static HTML, so I'll continue with that.

Of course, when using LODE or Parrot, we should first upload our files somewhere, or modify them to generate static HTML offline, as uploading lots of files there is against the netiquette.

BTW here are some reasonable ways to invoke curl with a file upload:

curl -s -i -F "[email protected]" -F "lang_label=en" -F "caller=http://www.essepuntato.it/lode" http://www.essepuntato.it/store.php

will tell you the URL where to retrieve the HTML documentation.

curl -s -L -o mobivoc.lode.html -F "[email protected]" -F "lang_label=en" -F "caller=http://www.essepuntato.it/lode" http://www.essepuntato.it/store.php

will download the file (but it's not really useful as it's not static).

In-browser validation, implemented as a PEG.js based browser extension

Vocabulary source files can be edited in the GitHub UI in the browser, but you don't have validation while you are typing. This could be realised by a browser extension using the PEG.js parser generator.

(BTW you don't have syntax highlighting either…)

Notify vocabulary engineers of errors

To make more precise what we wrote in the paper: I think there is an easy way of assigning vocabulary engineers to all generated validation issues. We could assume that a team vocabulary_engineers has been set up in the repository, and determine all users in this team. Thus, issues would be Cced to all vocabulary engineers. (I don't see an easy way of notifying the single vocabulary engineer who should be “in charge” of a specific issue.)

Create automatic documentation in schema.org style

Source of the tool: https://github.com/rvguha/schemaorg/

Requires Google App Engine (@np00 please show @lavdim how it works)

test issue

some body text

Wrap specgen call into a Makefile rule

… so that we can generate HTML from as many vocabularies as we like with a single, simple call.

Into such a Makefile we can also wrap any pre-processing (e.g. if specgen required RDF/XML input, which we'd have to produce from our Turtle source).

Generalise successful parts of mobivoc/mobivoc/Makefile and put them here

Let's not do the following, but let's archive the Makefile here to preserve the knowledge.

We now have a concrete instance of the build process running in https://github.com/mobivoc/mobivoc. Next step is to factor out the successful parts into a generic Makefile in vocol.

make sure that cronjob is run in the right directory

@np00's new vocolJob.sh invokes some other programs using relative paths. For them to be found, the cronjob must be executed in the right directory. Crontab should look something like this:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
*/5 * * * *     cd $HOME/place/of/the/github/repository/named/vocol; ./vocolJob.sh

(BTW this is a more elegant way to refer to the shell.)

It may be easier to prepare crontab as a file, instead of generating it on the fly with "echo".

Evaluate OOPS! for semantic validation

After syntax validation we can do an ontology-specific validation using the RESTful web service interface of OOPS!: http://oops.linkeddata.es/

@jerdeb this may also be of interest for you. OOPS! implements a number of quality metrics for ontologies (many of them OWL-specific though).

test issue

some body text

Fix URLs instances

Change static URL instances on rdfa document to mobivoc representation

Register Final application to someones account for using OAuth

Remove root permission when is not neccesary

When no root permission is not necessary any more, execute the further commands as user "vagrant"mobivoc.

Using Git Functionalities for Vocabulary Collaborations

Investigating how Git functionalities can be used for vocabolary collaborations i.e. branching. @clange @np00

Parrot - A RIF and RDF Ontologies documentation Tool

This tool provides users (both business users and developers) with useful reference documentation about rulesets and ontologies expressed in standard languages, such as OWL and RIF.

repo: https://bitbucket.org/fundacionctic/parrot/wiki/Home
demo: http://idi.fundacionctic.org/parrot/parrot

We can check it out and compare with specgen.

See #8 for further details.

Consider better web based editors to offer similar functionality as XTurtle

Adding an evaluation of already existing Class & Property definitions in LOV or SINDICE

Get specgen running without limitations

… i.e. the full version, without the workarounds currently described at https://github.com/mobivoc/vocol/wiki

Set up a cron job that runs the validation and publication process

A task for either @lavdim or @np00 – please discuss who has most capacity to do it

Inside the VM please run the process

if (vocabulary sources are valid)
then publish them as HTML and LOD
else report errors as issues

every 5 minutes or so, using a cron job.

See https://help.ubuntu.com/community/CronHowto (don't install gnome-schedule, which is just a GUI frontend, but … – maybe anacron?)

Make modules of the VoCol workflow (e.g. validation/publication component) easily choosable by configuration

E.g. a configuration file whose entries somehow feed into the Makefile and Vagrantfile (and its provisioning bootstrap script).

In the spirit of #36 there might even be a web frontend for choosing the desired components when setting up a new VoCol project.

Desirable documentation generators:

• schemaorg (supported already)
• specgen (or some of its newer forks; see https://github.com/mobivoc/vocol/wiki for some tips)
• LODE; see #5
• Parrot; see #8

Desirable validators:

• rapper (supported already)
• rdfcat
• eyeball; see #17
• OOPS; see #21

Once we start integrating more of the above, let's use separate issues for each one.

syntax error

Line number 23 consists of the following Error: syntax error

Compare RDF parser for best error output

Study at least the following ones:

• rapper (BTW in order to run make in https://github.com/mobivoc/mobivoc you will need rapper anyway)
• rdfcat

Use stow package manager for installations w/o a package manager

We are aiming at installing most packages using the system's package manager (e.g. apt-get on Ubuntu, zypper on SUSE, etc.), as this is the cleanest way. It permits uninstallation and easy updates and therefore helps to maintain a VoCol installation over a long time.

(BTW if PyGithub is available through apt-get on Ubuntu – on SUSE it is! – we should install it using apt-get rather than pip. pip permits uninstallation but is, of course, not integrated with the system-wide update manager.)

Raptor remains to be installed manually. To make it uninstallable, we can use stow, a poor man's package manager. stow will have to be installed using configure && make && make install, but from that point onwards one can change the following lines

tar -zxvf raptor2-2.0.15.tar.gz
cd raptor2-2.0.15
./configure # no sudo required here, only required for "make install"
make
sudo make install

to …

# as user mobivoc:

tar -zxvf raptor2-2.0.15.tar.gz
cd raptor2-2.0.15
./configure --prefix /usr/local/stow/$(basename $PWD)
make
sudo make install # BTW everything except this command should be executed as non-root, e.g. as user "mobivoc"
(cd /usr/local/stow/$(basename $PWD); sudo stow -v .)

FYI the $(basename $PWD) construction is not strictly necessary, but please enter basename $PWD once to understand how it works :-)

make install now installs stuff into /usr/local/stow/packagename. In the latter directory, stow . creates symlinks such as /usr/local/bin/program -> /usr/local/stow/packagename/bin/program, whereas stow -D . "uninstalls" the package by removing these symlinks.

Add a script that checks out all revisions from the repository

… and to any such revision applies whatever the user wants to apply, e.g. https://github.com/mobivoc/vocol/blob/master/validator/create_valid_stat.py.

The script I'd like you to add would be the generic implementation of the "evaluation methodology" we are selling.

Create Vagrant config file which includes all dependent libraries for our server

We would like to use VirtualBox, generated by Vagrant.

As an example, see https://github.com/pgroth/PROVTutorial/blob/master/Vagrantfile and https://github.com/pgroth/PROVTutorial/blob/master/bootstrap.sh

Use Eyeball for semantic validation

@np00, once syntactic validation (#9, #10) works I would also like us to implement some level of semantic validation. First a bit of generic semantic validation, later maybe extending it to MobiVoc-specific rules and "code styles".

Jena's Eyeball looks suitable for this. On the generic level I have used it before, but it is also highly configurable beyond that.

Eyeball has a comprehensive command-line frontend, but if the output becomes too complex to process, we can also invoke Eyeball from the Java side.

I told @lavdim about it, too, but will for now assign all validation-related issues to you.

test issue

some text

Clean up copies of *.jar files

There are now JAR files (of Jena etc.) in HtmlGenerator/libs and, once more, bootstrap.sh also downloads such files. I'd recommend not to add anything to the repo that bootstrap.sh can also download :-)

Script for self-service setup of VoCol on top of a GitHub repository

The following tasks should be automated:

• setting up the VoCol validation and publication service as a client to the given GitHub repository
• setting up suitable teams (domain experts, knowledge engineers, …) in the organization
• setting up suitable labels (e.g. validation error) in the issue tracker
• … (@soeren1611, @np00, @lavdim, please add more here)

Add back-links to the source git repository and to "new issue"

An idea for the future – very useful but requires a lot of work.

Concrete example: look into https://schema.org/Mountain. Why does a mountain have a fax number? Imagine that a link from faxNumber on the Mountain documentation page takes the reader directly to the source code of the affected class or property. Or that you have a "create new issue" button that takes you to a GitHub "new issue" form, pre-filled with some information about the class/property. (Not sure the latter is possible in GitHub.)

This can be done either by modifying the schema.org HTML generator's sources or by post-processing its HTML output.

Create a general script for executing processes

Create a script that will call all scripts (validation, turtle file generation, html generation)
@clange , @np00

Re-add "assignee" to generated issues

… in issueGenerator.py. At the moment there is a bug, which prevents us from using "assignee".

Set up cronjob (e.g. once a day) to update manually installed services (Google App Engine, Fuseki, …)

Google App Engine (the same holds for Fuseki and anything else) is currently installed manually because it's not normally available through the package manager. In the interest of security it would be important to keep these installations up to date. A daily cronjob could take care of checking whether there is a new version (e.g. looking here: https://storage.googleapis.com/appengine-sdks/), and if so, download it and restart the service.

Failed to convert qname xsd:integer to URI

Line number 115 consists of the following Error: Failed to convert qname xsd:integer to URI

Failed to convert qname xsd:integer to URI

Line number 115 consists of the following Error: Failed to convert qname xsd:integer to URI

Adapt Parrot output to publication on our own web server

To adapt the documentation generated by Parrot (#4), we need to do the following:

At the very least we need to copy (if permitted by the license) CTIC's images, CSS files, etc. Changing the base URL, which is what https://github.com/mobivoc/mobivoc/blob/master/Makefile currently does, is a crude hack and breaks links inside the documentation.

Or we need to adapt the links.

Or get used to invoking the command-line version of Parrot (which would be more scalable anyway).

Serve vocabulary through a SPARQL endpoint (proxied through Apache)

… and wrap the SPARQL endpoint into port 80 by using some kind of an Apache proxy configuration (because other ports will not necessarily be open on the server that runs VoCol, for security reasons).

Here's how this proxy configuration works: http://stackoverflow.com/questions/7529324/how-to-rewrite-proxy-an-apache-uri-to-an-application-listening-on-a-specific-p. The same pattern will be applicable to making Google App Engine (for schemaorg) on port 8080 publicly available through port 80.

Publish ontology and documentation in LOD style

e.g. so that http://purl.org/net/mobivoc/ redirects to the ontology or its documentation, depending on the content type requested by the client during content negotiation.

In our case, http://www.w3.org/TR/swbp-vocab-pub/#recipe5 applies. It's a matter of taste whether we do the configuration in .htaccess files somewhere in the virtual host's directory (/srv/www/htdocs/ok-mobivoc), or whether we do it right in the virtual host configuration file. In any case this requires that not the complete root (/) is "proxied" to Google App Engine, but Apache itself needs to remain in charge at least for serving the RDF/XML. (See #38)

This requires an all-in-one Mobivoc.rdf file (RDF/XML), which can be produced from the existing all-in-one Mobivoc.ttl file with a simple invocation of rapper. (@np00 should know how.)

Install cron if needed

Apparently the Ubuntu base "box" for Vagrant already had cron installed. On our demo server this was not the case. bootstrap.sh should therefore at least check whether cron is available and, if not, install it.

Recursively download schema.org-generated documentation from local web server into files

… using some of wget's or curl's mirroring options. Start with wget, but change to curl if wget doesn't get the job done.

The goal is to be able to call the download tool from the command line, and to serve the generated HTML files from a web server in a linked data way.

Write script which generates GitHub Issues (API) based on RDF errors

Deploy complete generation workflow on a server

… where the server would run a cronjob, which would periodically …

1. pull from some git repository using a special user account with sufficient permissions
2. run the HTML generation machinery (see #6)
3. push the generated HTML to the gh-pages branch

(2) and (3) are already covered by make .sync.

What do we do with error messages if we encounter any? Email them to someone, or output them to a “hidden” file in gh-pages?

Prevent race conditions: record IDs of all commits that have been processed already

vocolJob.sh currently calls issueGenerator.py and vocabularyGenerator.py with an INTERVAL of 5 minutes. vocolJob.sh itself is run by cron every 5 minutes. Thus it's theoretically possible that we lose some commits. vocolJob.sh should therefore consider more commits than just those of the past 5 minutes. To prevent double processing of a commit, it should record all commits it has processed already.

Please document Vagrant bootstrap script

Great to see a bootstrap script with so many parts. Could you please add some basic documentation, such as one comment every few lines?

Auto-created validation-related issues should have their own label

… so that one can filter for them. Does the GitHub API allow assigning labels?

See what we can do with hooks

Maybe GitHub does support server-side hooks? This would make our job a bit easier. Please see https://developer.github.com/v3/repos/hooks/ and, e.g., https://github.com/EIS-Bonn/Papers/settings/hooks.

Restructuring: Creating an own repository for VoCol and the other planned initiatives.

test issue

some body text

Convert Turtle input to the RDFa format that the schema.org HTML generator expects

Here is an example of the expected input: https://github.com/rvguha/schemaorg/blob/master/data/schema.rdfa

BTW the file that should ultimately be converted is one of mobivoc.* in https://github.com/mobivoc/mobivoc. These are generated by running make (which requires rapper to be installed). https://github.com/mobivoc/mobivoc/Makefile (which should ultimately become part of vocol; see #6) generates mobivoc.nt from the *.nt (N-Triples) versions of each *.ttl module, and then, from mobivoc.nt, the other mobivoc.* files. Processing mobivoc.nt is the most direct way, but you can also process mobivoc.ttl or mobivoc.rdf.

Validation issues should mention names of affected files

not as in #24 – there you don't know what files are affected