wikidp/wikidp-portal

Prototype wikidata portal project.

Wikidata DP Portal Prototype

A prototype for the Wikidata digital preservation portal.

Pre-requisites

• Git or a copy of the latest source code
• MacOS or Linux. Sorry Windows isn't currently supported.
• Python 3. Python 2 isn't supported.
• Python pip for installing Python modules.

Using virtual environments for Python will save a lot of pain and allow you to run Python 3 and Python 2 applications in harmony. If that sounds good then read this primer.

Quick Start

The portal is a Flask web application written in Python. It's currently easy to install as long as you have a Python 3 environment. Be aware that this is currently a local development installation, it's not ready for deployment as a reliable application to a server. We're working on that. That said there's a few stages to getting going:

1. Getting the code.
2. Setting up a Python 3 virtualenv (optional but reccommended).
3. Installing the portal prototype and its dependencies.
4. Running the portal.

Let's take a look in a little more detail. We've provided some helper scripts for virtualenv setup, deployment and running which we'll also point you to.

Getting the code

There's no helper script for this. Clone this repository and move into the project root directory:

git clone https://github.com/WikiDP/portal-proto.git
cd portal-proto

Alternatively download and unpack the source archive from this git repository.

wget https://github.com/WikiDP/portal-proto/archive/master.zip
unzip master.zip
rm master.zip
cd portal-proto-master

Once you've done this you can create the virtualenv for installation, or skip the next step if you have a Python 3 environment you're happy to use.

Setting up a Python 3 virtualenv (optional)

We need to create a Python virtualenv in the project root folder in a venv subdirectory and activate it thus:

virtualenv -p python3 venv
source ./venv/bin/activate

There's a helper script in the root directory you can run instead: $ ./venv.sh. If this has worked your terminal prompt should be adorned with a venv marker, e.g. (venv) $.

You only need to create the virtualenv once, although you can remove it by simply deleting the venv subdirectory. You'll need to activate the virtualenv source ./venv/bin/activate every time you start a new terminal session.

Installing the application and dependencies

Installing the application is straightforward using pip:

(venv) $ source ./venv/bin/activate
(venv) $ pip install -e .

Where the -e switch tells pip to monitor the directory and recompile changes. This is useful for development but should be omitted for stable deployments.

Again there's a helper script for this: (venv) $ ./setup.sh.

Running the application

The following steps need to be followed for every new terminal session.

You'll need to set up your Wikidata user name and password credentials, these can be exported as environment variables for now along with another variable that sets the flask application name:

(venv) $ export WIKIDP_BOT_USER='<username>'
(venv) $ export WIKIDP_BOT_PASSWORD='<password>'
(venv) $ export FLASK_APP='wikidp'

where <username> and <password> are your user name and password.

NOTE these will need to be set for every new session for now.

Finally run the Flask application:

(venv) $ flask run

There's a script that you can use for this in the project root, run.sh. You'll need to edit it once to provide your Wikdata credentials by replacing the placeholders in the script here:

export WIKIDP_BOT_USER='<username>'
export WIKIDP_BOT_PASSWORD='<password>'

Point your browser to http://127.0.0.1:5000 and you should see the prototype of the portal.

Testing the application

All Tests are currently in the tests/ directory. We use py.test to manage our testing interface.

Running Tests

(venv) $ pytest

Checking test coverage

We have provided a bash script to handle the coverage reporting

(venv) $ ./coverage_report.sh

If you would prefer a more visual web-interface for the coverage report, this creates an htmlcov/index.html file for browser viewing

(venv) $ ./coverage_report.sh visual

Ansible roll out

Local vagrant VM

vagrant up

Install Ansible roles locally

ansible-galaxy install -r ansible/requirements.yml

Ansible CLI roll out to local Vagrant VM

ansible-playbook --key-file=.vagrant/machines/default/virtualbox/private_key -v --inventory ansible/vagrant vagrant.yml

First time setup for any server via SSH, needs root password for server

ansible-playbook ansible/setup.yml -i ansible/staging.yml -u root -vv -k

https://unix.stackexchange.com/questions/332641/how-to-install-python-3-6 wget https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz tar xvf Python-3.6.3.tgz cd Python-3.6.3 ./configure --enable-optimizations make -j8 sudo make altinstall python3.6

Troubleshooting

Set FLASK_APP env variable

If you see something like:

Usage: flask run [OPTIONS]

Error: Could not locate Flask application. You did not provide the FLASK_APP environment variable.

For more information see http://flask.pocoo.org/docs/latest/quickstart/

You need to set the FLASK_APP environment variable, see Quick Start above.

This code is released under the GPLv3 license.

wikidp-portal's Issues

Add Statement Panel

Users of the Digital Preservation Portal will need to be able to view the statements associated with an item in Wikidata. They will need to be able to create additional statements.

Consider adding a workflow for creating a new item.

While many users will want to use the portal to contribute statements to existing items, some people may want to create entirely new items. We do not yet support creating new items. We will need to design a workflow for new item creation.

Not all file extensions are searchable

A search for the string ".uef" does not return http://www.wikidata.org/entity/Q7884922. Is it the case that "." + the value of P1195 is supported in our search function?

OAuth integration with Wikidata

Wikidata integrator seems to offer straightforward OAuth integration with Wikidata. Downside is that anyone with a Wikidata account has a working account on our system.

Consider extending structure for schema gallery to allow nesting

For our schema for "file format with identification pattern" in our schema gallery (currently available on staging) qualifiers are showing up on the same level as properties. I would be helpful to extend our structure to accommodate nesting so that the qualifiers can be shown in relation to the property they modify.
For example, in the Wikidata UI:

Consider using EditGroups to allow for rollbacks

We may want to use EditGroups.
Here are the directions for tool developers:

Tidy unwanted GitHub artifacts

e.g. cache files, .pyc, etv

Web GUI doesn’t adapt to different screen sizes

When viewing on an iPad the menu drops into the page, also the display becomes untidy.

Consider creating user-facing message for OAuth failures

Currently we need to inspect the logs to view details of an OAuth failure. We'd like to have a user-facing message that will give people further details on what when wrong if we are not able to authenticate them.

Add in support for creating references for statements

Create inspiration page

Gadget info

https://www.wikidata.org/wiki/User:Daniel_Mietchen/wikidp.js

Think about back end mechanism for serialisation

Is Python's native pickle good enough or do we use a DB back end instead? @carlwilson to investigate.

Redesign "contribute" button

Based on feedback from WikidataCon, we might consider changing the placement of the "contribute" button. After users browse, how will they know they need to hit "contribute" to add their statements?

embed SPARQL query on reports page

<iframe style="width: 80vw; height: 50vh; border: none;" src="https://query.wikidata.org/embed.html#SELECT%20DISTINCT%20%3Fapp%20%3FappLabel%20%0AWHERE%20%7B%0A%20%20%3Fapp%20wdt%3AP31%2Fwdt%3AP279%2a%20wd%3AQ235557.%0A%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22.%20%7D%0A%7D" referrerpolicy="origin" sandbox="allow-scripts allow-same-origin allow-popups" ></iframe>

Feedback from GeneWiki demo

On 13 Feb 2018 @kennethsn and @emulatingkat demo-ed the portal for the GeneWiki team at the Su Lab. Here is a summary of feedback.
"looking for an item id"- perhaps infer the class of the item and also display that to aid selection

could prioritize search results by subclass of/instance of

a class-level diagram of the domain would be helpful

search by mediatype not working

I tried searching by mediatype but the portal did not return search results.

I don't think search by mediatype is working right now.

Create About page

Enhanced dashboard
Creators
License

Learning more about Wikidata

Acknowledgements:
Su Lab
WikiCite
GeneWiki
Logo designer

Select schema to use in property checklist

The current property checklist for software is for the minimal shape. I'd like to allow users to select from a list of schemas to use to indicate the inventory for the property checklist.

Search by mime type

Implement the versioning-out interaction.

We often want to create multiple items per software title, ex. expressing different versions. This interaction will allow the user to select statements from the 'base' item and auto populate the new item.

Documentation required for PyWikiBot user security configuration

The first setup requires that the user adds their Wikidata login details to a file called user-config.py in the application directory. The problem manifests itself with the following output when the app is run without the config file:

RuntimeError: No user-config.py found in directory '/home/euan/portal-proto'.
  Please check that user-config.py is stored in the correct location.
  Directory where user-config.py is searched is determined as follows:
 
    Return the directory in which user-specific information is stored.
 
    This is determined in the following order:
     1.  If the script was called with a -dir: argument, use the directory
         provided in this argument.
     2.  If the user has a PYWIKIBOT2_DIR environment variable, use the value
         of it.
     3.  If user-config is present in current directory, use the current
         directory.
     4.  If user-config is present in pwb.py directory, use that directory
     5.  Use (and if necessary create) a 'pywikibot' folder under
         'Application Data' or 'AppData\Roaming' (Windows) or
         '.pywikibot' directory (Unix and similar) under the user's home
         directory.
 
    Set PYWIKIBOT2_NO_USER_CONFIG=1 to disable loading user-config.py

SPARQL endpoint for our Wikibase instance

Create reports page

The Reports page will allow us to embed sparql queries that will allow us to monitor the growth of different slices of Wikidata data.

UI Review for first release

Review the web GUI with the following in mind:

• consistency, is it easy to identify buttons and other UI elements
• where can we use more "standard" controls
• accessibility, our current contrasts are poor in places.

Add API Documentation to Docs/User Guides

It would be helpful to have a documented source to point users to access API calls that we expose perhaps in our Docs

Dropdown menu background color and font color very similar in Chrome

We may want to consider changing the background color for dropdown menus. In Chrome, the font is difficult to read until you mouse over the selection.

Add Item

Users of the Digital Preservation Portal will need to be able to create new items.

Consider putting file formats first in results

As a user, when I enter 'KDC' into the search bar, I would like to see Q28205360, and Q28205363 as the top two results.

Limit searches by domain

As a user of WikiDP when I search for the following:

• Formats
• Software
• Software environments

I expect to only see results relevant to those searches.

When I search, however, I can bring up anything in Wikibase:

If these searches were restricted by domain/s then the results a user discovers through the system are relevant to their digital preservation needs. It also promotes accuracy in Wikidata as WikiDP generates a feedback loop that promotes accurate data and classification in the source system, i.e. Wikidata.

Schema Gallery

I'd like a browsable gallery of schemas that can be used in the property checklist panel.

These will be created by our team. Others are welcome to contribute additional schemas.

The current set of schemas we have are shex.c files.

Home page doesn't work

When I try to look at the home page I get the following stack trace from the terminal running the flask app:

[2017-04-03 15:09:03,385] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "wikidp/portal_routes.py", line 20, in hello
    formats = get_formats_generator()
  File "/home/cfw/dev/git/WikiDP/prototype/wikidp/wikidata.py", line 32, in get_formats_generator
    return get_sparql_generator(query_file_path, SITE)
  File "/home/cfw/dev/git/WikiDP/prototype/wikidp/wikidata.py", line 37, in get_sparql_generator
    with open(query_file_path, 'r') as query_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/cfw/dev/git/WikiDP/prototype/venv/lib/python3.5/site-packages/pywikibot/families/list-formats.rq'

Consider removing 'support' link or updating it

Currently the 'support' link doesn't lead anywhere. We could point to our issues on github? Or elsewhere? Alternatively we could consider removing this.
Reported during iPres by Joshua Ng.

Create login form

Users will use this form to enter their username and password.

Consider adding default property set

Some users may want to make use of properties beyond those we offer in schemas. Create a list of additional properties that will be available for use.

Add in support for adding qualifiers to statements

Use WikidataIntegrator for Writing Claims to Wikidata

Remove

README local host link could be a link.

Add acknowledgements section on the about page

Make the urls on the about page into links

When linking directly to an item in preview mode "other details" info does not seem to display

I was looking at http://wikidp.org/Q1593782 today and it seems like other details info is not showing up in this view.

Consider adding a workflow for editing label, description, alias for existing items

Our current contribute workflow allows users to contribute claims and qualifiers and references. We do not yet support editing or adding labels, descriptions and aliases. Some users may wish to edit these, so we will need a new interaction to support this.

Search bug

I think I understand how the server went down last week. Unfortunately, I think it may have just gone down again. I performed a search just now for "x-fmt/83" which I was expecting to NOT be found. There was no response, and now the site is down. Now that I've seen this, I think this is also what caused it to go down last week.

Preview Panel for Item Search

Allowing user to text search for an item and be presented the relevant details of that item (statements/claims, external links, aliases, descriptions) based on the current wikidata API, wikidataintegrator, and wikidata's public JSON.