datopian/ckanext-datapub

CKAN extension to integrate your custom DataPub.js based dataset and resource editor

Enable Datapub apps in your CKAN instance.

Find out about Datapub here - https://github.com/datopian/datapub.

Installation

Just follow standard procedure for setting up an extension for the CKAN and add datapub into list of the plugins. Note that this extension supports CKAN 2.8 onwards.

Building and publishing the datapub app

To build the datapub React app we use the sync.sh script. This script will

1. Clone (or copy from a local folder) the datapub app source
2. Install its dependencies and create the production build
3. Update the asset references in the CKAN templates so they include the built application

The sync.sh accepts the following parameters:

sync.sh [<datapub-repo-or-folder>] [<git-tag>] [<template-tag-name>]

• <datapub-repo-or-folder> can be a Git URL or the path to a local datapub repository. Defaults to https://github.com/datopian/datapub
• <git-tag> is the git tag or branch that you want to build. Defaults to master
• <template-tag-name> is used to target CKAN<=2.8 (using resources) or CKAN>=2.9 (using webassets). Defaults to webassets.

For example, to build a released tag of a custom datapub repo:

./sync.sh https://github.com/datopian/datapub v0.1.2

Same, but targetting CKAN 2.8:

./sync.sh https://github.com/datopian/datapub v0.1.2 resources

To build a local development clone of datapub:

./sync.sh ../datapub-gdx

Template changes

This extension extends the templates/blob_storage/snippets/upload_module.html template:

{{ASSETS}}

<div id="ResourceEditor"
     data-dataset-id="{{ pkg_id }}"
     data-api="{{ base_url }}"
     data-lfs="{{ h.blob_storage_server_url() }}"
     data-auth-token="{{ api_key }}"
     data-organization-id="{{ h.blob_storage_storage_namespace() }}"
     data-resource-id="{{ resource_id }}">
</div>

When running sync.sh, the {{ASSETS}} placeholder will be replaced by the appropiate template tags.

On CKAN>=2.9, {{ASSETS}} gets replaced by these two webassets references:

{% asset "datapub/datapub-js" %}
{% asset "datapub/datapub-css" %}

On CKAN<=2.8, individual fanstatic resources need to be referenced, eg:

{% resource 'datapub/css/main.1eb11ae4.chunk.css' %}
{% resource 'datapub/js/runtime-main.c155da57.js' %}
{% resource 'datapub/js/2.2599189b.chunk.js' %}
{% resource 'datapub/js/main.c4752953.chunk.js' %}

ckanext-datapub's Issues

Duplicate templates between ckanext-blob-storage and ckanext-datapub

There are a few templates that are the same or very close between ckanext-blob-storage and this extension, and it's just confusing.

If ckanext-blob-storage will always depend on datapub, and has no UI at all, the templates should be removed from there. But I don't like this solution cause it creates cyclical dependencies between these two extensions, and I think ckanext-blob-storage should have some kind of basic UI. So I suggest assuming the base templates exist in ckanext-blob-storage (as this extension has a dependency on it anyway) and removing the duplicates from here.

You can then override only templates/blob_storage/snippets/upload_module.html to point to the right JS files, no need to also override templates/package/*.

Enable cloning react app from a private git repo

At the moment the datapub app we clone from using DATAPUB_APP env variable must be public. Is there a way to be able to set this up so we can access private git repos (using a token or username and password of a read-only user?)

https://github.com/datopian/ckanext-datapub/blob/master/sync.sh#L1

DATAPUB_APP=${1:='https://github.com/datopian/datapub'}

Use parameters in sync.sh as per datapub project, e.g. nhs, gdx.

We need sync.sh file to be parametrized as per datapub project, for example for NHS it should look like this, while we are fetching datapub-nhs or just nhs from config.

(for datapub-nhs - https://github.com/datopian/datapub-nhs)

git clone [email protected]:datopian/datapub-nhs.git
cd datapub-nhs
yarn
yarn build
cd -
rm -r ckanext/datapub/fanstatic/**/*
cp -r datapub-nhs/build/static/* ckanext/datapub/fanstatic/

Acceptance

• sync.sh is parametrized with what

Tasks

• Set up variable in config to point to specific datapub project
• sync.sh uses datapub specific project name as config variable

[BUG] Missing js, css files included from appropriate datapub-*** repo

While testing in NHS getting the following runtime error:

Error - <type 'exceptions.AttributeError'>: 'module' object has no attribute 'datapub/css/main.b4d771cb.chunk.css

as one of the files, we are pointing in upload_module.html

{% resource 'datapub/css/main.b4d771cb.chunk.css' %}

{% resource 'datapub/js/runtime-main.c155da57.js' %}
{% resource 'datapub/js/2.2599189b.chunk.js' %}
{% resource 'datapub/js/main.c4752953.chunk.js' %}

<div id="ResourceEditor"
     data-dataset-id="{{ pkg_name }}"
     data-api="{{ base_url }}"
     data-lfs="{{ h.extstorage_lfs_url() }}"
     data-auth-token="{{ api_key }}"
     data-organization-id="{{ h.extstorage_organization_name(pkg_name) }}"
     data-resource-id="{{ resource_id }}">
</div>

We need to have these files included in this repo here:
https://github.com/datopian/ckanext-datapub/tree/master/ckanext/datapub/fanstatic
currently excluded with gitignore