onroerenderfgoed/atramhasis

An online SKOS editor

Atramhasis

---

Atramhasis is an online SKOS editor. This webapplication enables users to create SKOS vocabularies consisting of Concepts and Collections. It tries to stay close to the SKOS specification. This makes it suited for different types of vocabularies such as simple pick lists, authority files, flat lists and basic to relatively complex thesauri. For a full overview of capabilities, please consult our documentation. If you want a quick demo of what the system provides for a typical end user, have a look at the Flanders Heritage Thesaurus. This is an implementation of Atramhasis for the Flanders Heritage Agency containing several vocabularies regarding cultural heritage, such as heritagetypes, styles and cultures, materials and periods. Most of the vocabularies are in Dutch, but the interface is available in both Dutch, English and French.

All development is done through Github. If you run into bugs or would like to request a new feature, please open a Github issue. Please provide some context to your question such as the operating system, Python and Atramhasis versions you are running. If you have specific questions about the software or the datasets hosted at Flanders Heritage Thesaurus you would rather not address in a public forum, please mail us at [email protected].

Contributions are welcome. This could be improvements to the code, updated documentation or adding a new translation to the User Interface. Please see our contribution guidelines to find out how we can collaborate.

atramhasis's Issues

Alembic script only works on python 3.x

Currently the alembic script tries to from configparser import ConfigParser. This only works on python 3.x. See https://docs.python.org/2/library/configparser.html

Validation rule: narrower concept

When editing or adding a concept, a narrower concept should be a concept (not a collection) and it should not already be a broader concept of the concept being edited.

So if the database contains the following hierarchy of places: World > Europe > Belgium > Flanders > East-Flanders, it should not be possible to set Belgium as a narrower concept for East-Flanders.

Public interface has links to RDF download

Provide an export to SKOS RDF format for:

• An entire conceptscheme
• A single concept

Since this is interesting functionality for any vocabulary loaded through a skosprovider, not just through Atramhasis, we can split this into a separate skosprovider_rdf library.

CRUD service has authorisation hooks

CRUD service has hooks for authorisation. Resources that provide principals.

Our security needs at this moment (this is an oversimplification just to get started):

• public interface is public
• read service is public
• write service needs an authenticated user with the role editor or the role <conceptscheme_id>.editor (basically, user is allowed to edit only the concepts and collections in a certain scheme).

We need to implement a security.py module that defines the necessary Factories and Resources and defines the acl attributes on them (they determine the actual principals needed). See http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html for documentation on pyramid security. It might be possible to reuse the skosprovider.skos or skosprovider_sqlalchemy.skos objects as Resources.

Provide customisability

/cc @BartSaelen /cc @dieuska

A few aspects of Atramhasis will need to be customised for every deployment

• Look and feel of the public interface
• Security (provide custom authentication and authorisation policies)
• Define conceptschemes / providers: A user will need to provide the conceptschemes he needs to manage and some information about them

The ideal way to do this is by using pyramid scaffolding. See http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/scaffolding.html for more info.

Sample from the perspective of an end user:

$ mkvirtualenv my_thesaurus
$ pip install atramhasis
// installs a version of Atramhasis and needed dependencies like pyramid and sqlalchemy
$  pcreate -s atramhasis my_thesaurus
// our scaffold provides hooks for the custom functionalities and some utiltity files like:
// setup.py
// requirements.txt
// my_thesaurus/skos.py (module to define conceptschemes, we might like to provide a sample implementation)
// my_thesaurus/templates/ (place to store overriden templates)
// development.ini
// production.ini

Languages

/cc @dieuska

Figure out what language codes are acceptable and supportable. SKOS refers to http://www.rfc-editor.org/rfc/bcp/bcp47.txt

See also: http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry

Switch from nose to py.test

We'll switch to py.test for unit tests and pytest-cov for coverage.

Todo:

• Document how to run unit tests in documentation (docs/source/development.rst)
• Update .travis file See https://github.com/koenedaele/skosprovider/blob/master/.travis.yml for inspiration

Make label language aware

Currently, we always showing the label of a Concept or Collection in the default language of the provider. We should pass the currently selected language to the label() method when rendering.

This can be applied to concept.label() (http://skosprovider.readthedocs.org/en/latest/api.html#skosprovider.skos.Collection) and to provider.find (http://skosprovider.readthedocs.org/en/latest/api.html#skosprovider.providers.VocabularyProvider.find) or provider.get_all.

Validation rule: related

When setting a related concept, this should always be a concept, not a collection.

Documentation about authentication

Document the authentication mechanism and how this can be overriden and changed in a deployment package.

Provide download of search result as csv

Provide a download of a search action as a csv file. Inluded information:

conceptscheme, id, uri, type, label, prefLabels, altLabels, definition, broader, narrower, related
'TREES', '1', 'urn:x-skosprovider:TREES:1', 'concept', 'The larch', 'The Larch (en), De Lariks (nl)',,'A type of tree.',,,

Feel free to reassign to @BartSaelen if needed. This is a fairly simple, non-urgent feature.

Create a basic public interface

Create a basic public user interface

• A home page
• A search results page
• A page with detailed information about a Concept of Collection
• Based on standard Foundation css.

Documentation about setup, installation, customisation

Write basic documentation about:

• Installing atramhasis
• Participating in development
• Creating a deployment package
• Customising your deployment

Authentication mechanism for demo

In our own deployment we will be using our own security mechanism that integrates with our SSO environment.

For the open source version we will provide a demo package that will need some form of authentication. This can be something as simple as a hardcoded password, but could be something more like integration with Google/Twitter/Facebook Oauth.

Validation rule: prefLabel

A concept or collection can have only one prefLabel per language.

So, this is acceptable:

• prefLabel: Kerken (nl)
• prefLabel: Kerken (nl-BE)

This is not:

• prefLabel: Kerken (nl)
• prefLabel: Kerkgebouwen (nl)

Show collection on concept detail

A concept can be a member of one or more collections. Currently we're showing the members of a collection on the collection detail page. But we should also show the collections a concept is a member of on the concept detail page.

One main problem here is that the skos specs do not have anything like a memberOf of inCollection property. Therefor, currently skosprovider doesn't support it either.

Possible avenues:

• Add some sort of inCollection property to skosprovider
• Add something to Atramhasis and only there.
• See if someone else has already defined an RDF predicate like this.

/cc @BartSaelen @cedrikv @dieuska

Managing dependencies

Managing dependencies

• setup.py contains direct dependencies with as little version information as need (eg. pyramid>1.5)
• requirements.txt contains a concrete set of dependencies that is known to work (can be generated from pip)
• requirements-dev.txt imports requirements.txt and adds everything needed for development: pytest, coverage, sphinx, flake8, tox, ...

See https://github.com/OnroerendErfgoed/dcextensions for an example.

Add code coverage integration

Add integration with coveralls.io

Add skeleton for Sphinx docs

Adding a skeleton for Sphinx docs. Deploy on Read-The-Docs.

include locales in distribution

/cc @BartSaelen

When trying to run the demo via http://atramhasis.readthedocs.org/en/latest/demo.html the last step fails because there is no folder atramhasis/locales.

Something probably needs to be added to MANIFEST.in

Admin interface

A fully functional Admin interface, on url /admin.

Update public interface

Further work on the public user interface. Release 0.2.0 should bring this more or less to completion.

Mark active language in public interface

Currently, there's no visual clue about what language you're surfing the site in. Is there some class="active" or so available in Foundation?

/locale needs referer

If you go to eg. /locale?language=nl without clicking on a link, no HTTP_REFERER is present. In this case, we should just fallback on something else, eg. the home page.

Problem with translations

When starting the current development version, I end up with a home page with the string tr_search on it, no matter what language I choose.

Is there a step needed to compile language files? Can we add this to the documentation?

Add dependencies

Add a dependency on skosprovider, skosprovider_sqlalchemy and pyramid_skosprovider.

Translation and scaffolding

Currently generation of .po files only works from within the atramhasis package itself, not from within a scaffold.

Create basic concept detail

Part of #15 .

Add a page that displays basic info about a Concept or Collection.

url: /conceptscheme/{scheme_id}/c/{c_id}

This is the same URI as http://pyramid-skosprovider.readthedocs.org/en/latest/services.html#get--conceptschemes-{scheme_id}-c-{c_id}. Might be an issue there with handling of Accept headers. The service should listen for application/json requests, the public UI for text/html requests.

This page contains the basic information about the concept or collection:

• id
• type (concept or collection)
• labels
• notes
• broader (with generated links)
• narrower (with generated links)
• related (with generated links)

Reorganise label display

<h2>labels</h2>
<h3>preferred labels</h3>
<ul>
 <li>The Chestnut <em class="language">(en)</em></li>
</ul>
<h3>alternative labels</h3>
<ul>
 <li>De Paardekastanje <em class="language">(nl)</em></li>
</ul>

Basic idea: Under the h2 there are 3 possible h3's: pref labels, alt labels and hidden labels. If a certain category is empty, don't show it. Within each category is a list of labels, with the language in parentheses. The class language is to allow future implementations to style the languages differently if needed.

Create a basic home page

This is part of #15

Create a basic home page:

URL: /

This page contains a search box and a drop down. This drop down contains a list of available conceptschemes / providers. The user can enter a search term and choose a conceptscheme (the first conceptscheme is the default). Upon submitting a search form, the user goes to the search results page.

Create a basic search results page

This is part of #15

We create a basic search results page that shows search results within a single conceptscheme.

URI: /conceptschemes/{scheme_id}/c

This is the same endpoint as the http://pyramid-skosprovider.readthedocs.org/en/latest/services.html#get--conceptschemes-{scheme_id}-c service, but serves html data.

The searchparameter is called label. eg /conceptschemes/TREES/c?label=linde will look for concepts with a label of linde. This gets processed by http://skosprovider.readthedocs.org/en/latest/api.html#skosprovider.providers.VocabularyProvider.find

The search results page contains a list of concepts or collections. For each the following information is present:

• id
• label
• type (concept or collection)

Add travis.yml file

Add a .travis.yml file for continuos integration testing

Limit available languages

Currently the available languages are defined in two places: on the file system as a set of /locale folders and in the base template as a list of available languages to be selected by the user.

Atramhasis might come with more language than a user cares for in his/her implementation. it sohuld be possible for an implementer to determine which languages should be presented. Something similar to http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/i18n.html#detecting-available-languages.

The base template should use this list to generate the language switch buttons.

Add searching on type to public interface

Add the possibility to search based on type: 'all', 'concept', 'collection' to the public user interface.

Provide testdata

Validation rule: broader concept

The broader concept of a concept must also be a concept (not a collection) and must not itself be a narrower concept of the concept being edited.

So if the database contains the following hierarchy of places: World > Europe > Belgium > Flanders > East-Flanders, it should not be possible to set Flanders as a broader concept for Europe.

Provide demo

/cc @BartSaelen /cc @dieuska

It would be possible to provide a demo application for users who want to test the software. For this we need some testdata (see #5) and a basic configuration.

Similar to #11, we can do this through the use of scaffolding.

From the perspective of an end use:

$ mkvirtualenv atramhasis_demo
$ pip install atramhasis
// installs a version of Atramhasis and needed dependencies like pyramid and sqlalchemy
$ pcreate -s atramhasis_demo atramhasis_demo
$ pserve development.ini

Create project skeleton

Set up alembic migrations

We will use Alembic to trac migrations in the database schema. See http://alembic.readthedocs.org/en/latest/

Validation rule: broader, narrower, related and members within one scheme

When editing relations between concepts or concepts and collections, care should be taken that all relations stay within one conceptscheme.

So, when setting the members of a collection, these should be concepts or collections within the same conceptscheme.

Relations between concepts in different conceptschemes will be handled with skos:matches.

Document REST service

Providing documentation about the REST service using https://pythonhosted.org/sphinxcontrib-httpdomain/

Make sure public interface can be translated

Make sure public interface is translatable and locale aware.

Add ability to add images to a concept

Nice to have.

Public interface has links to print pages

Switch order of narrower/broader

Currently the display first displays narrower concepts, followed by broader concepts. Would make more sense to first place the broader concepts, followed by the narrower concepts.

Add jinja2 as templating language

For the public interface we use Jinja2 as the main templating language. Install pyramid_jinja2 and run the scaffold:

http://docs.pylonsproject.org/projects/pyramid_jinja2/en/latest/#jinja2-starter-template

Create CRUD Rest Service

Create a REST service to add, update and delete concepts and collections.

Package doesn't include jinja2 templates

When building a package with

$ python setup.py sdist

the MANIFEST.in file is used to determine what to include in the package. Currently jinja2 templates are not included.

Add tox configuration

Add a tox.ini file for testing in different environments.

Python versions to be supported:

• 2.7
• 3.2
• 3.3

Adding html lang attributes

It seems to be possible to set a natural language for a certain block of html code, eg:

<p lang="en">This is English.</p>
<p lang="nl">Dit is Nederlands.</p>

Not sure if this already implemented in all major browsers. We could set the lang attribute on the html tag to the currently selected locale (eg. ) and then override specific labels and notes in other languages (eg. An english label).

Something to investigate.