datopian / datapub Goto Github PK
View Code? Open in Web Editor NEW๐ React-based framework for building data publishing workflows (esp for CKAN)
Home Page: https://tech.datopian.com/publish/
License: MIT License
๐ React-based framework for building data publishing workflows (esp for CKAN)
Home Page: https://tech.datopian.com/publish/
License: MIT License
Follow up (broken out of) #17
When a linked file is chosen we infer core meta from url and try to load data (if CORs issue then just give up (maybe flag an error)
A Data Dictionary (or Table Schema) provides information about the data in a resource such as the names, types and description of the fields.
https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource
For more, see https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource
Working table schema editor as react component
UX is along lines of https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource (but feel free to adapt/innovate)
Basic
sample
attribute as array of dictsInfer
Bonus / future
Research editors for json schema to see if we could reuse for our purposes ...
Here's a rough code sketch of how this component could work:
<TableSchema
// props for initialization
// http://frictionlessdata.io/table-schema
// https://specs.frictionlessdata.io/table-schema/
schema= Frictionless (f11s) Table Schema object
// sample of the data for displaying etc
sample= [{ a: 1, b: 2}, {a: 5, b: 10}]
/>
Schema example for this case:
schema = {
fields: [
{
name: 'a',
type: 'string',
description: 'column a is about X',
format: '...'
},
{
name: 'b',
type: 'integer'
}
]
}
Inside TableSchema component:
TableSchema.props.schema = make a copy of passed in schema
As you edit in UI the internal schema object is updated
resource = open(HTML5 file object)
TableSchema(resource.schema, getSample(resource))
Use https://github.com/datopian/data.js for opening files ... (why? gives you all the metadata)
When trying to upload a file, if an error occurs somewhere in the process that is not originally expected, the error is suppressed and will not appear in logs. This makes debugging needlessly harder.
We should just add console.error()
call to src/components/Upload/index.jsx
onClickHandler
catch
call.
In portal.js we agreed to move to typescript and I think we probably want this here.
If a file is selected we want to infer a frictionless style object that we can use for the metadata. Suggest using data.js and fixing this issue frictionlessdata/frictionless-js#49
sample
attribute.It seems that there are cases in which we might calculate the hash of the same type for the same file more than once. I suggest caching the hash on the File object by type once it has been calculated, as this is a fairly expensive operation. Once a hash has been cached, and File.hash(<type>)
is called again, it should just be returned (assuming we have hash of <type>
cached).
Stub the react app for the data publishing app focused on adding a resource.
When the upload is in progress the metadata fields (Title, description, encode, format) the metadata information should be editable at the same moment of the upload.
I expect to be able to cleanly upload, with no errors in the console, files of any type whether they can be parsed by datapub or not.
When trying to upload a binary file (extension is .bin
, file is actually a random binary file but I suspect this will happen for many types of files that are not tabular data...), I get the following error in the js console:
Error: We do not have a parser for that format: bin
_rows file-base.js:137
rows file-base.js:116
onChangeHandler index.jsx:33
React 17
unstable_runWithPriority scheduler.development.js:653
React 4
index.js:1
e index.js:1
onChangeHandler index.jsx:37
React 17
unstable_runWithPriority scheduler.development.js:653
React 4
This doesn't block the upload process, but this should not be an error level message.
When the user selects a file and starts uploading they are presented with key metadata fields (e.g.title, description, encode, and format) that they can edit. These metadata fields should be auto-populated where possible. The user can then explicitly save (once the file has finished uploading).
Bonus would be extracting the table schema info: I want to extract the table schema info from the file and save it in a JSON object. See #13
Given a file by a user
data.js
https://github.com/datopian/data.js to load the file and infer informationBonus
Storybook is a tool that prepares a development environment for UI components. It allows you to develop and design your graphical interfaces quickly, isolated, and independently. Making it possible to define different states for components, thus documenting their states.
yarn add
Done See updated ReadmeWhen the user uploads an open upload app, the user wants to choose it is a file or link.
* [ ] The metadata will infer the user to choose a link (?)
* [ ] JS/UI - After the user chooses an upload link and insert a link.
- Does the metadata should be auto-populate like already happens with the upload a file?
Bonus:
At the moment, we have default setup created by CRA tool. This auto initializes the app without any arguments being passed:
// src/index.js
import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';
ReactDOM.render(
<React.StrictMode>
<App />
</React.StrictMode>,
document.getElementById('root')
);
What do we want is to be able to choose components that is needed for a project and construct the UI:
For example, in a specific project app, you may use only FileUploader
and MetadataEditor
:
import React from 'react';
import { FileUploader, MetadataEditor } from 'datapub';
function MyApp() {
return (
< FileUploader />
<MetadataEditor />
);
}
export default MyApp;
or you can just load entire app with all default components:
import React from 'react';
import { ResourceManager } from 'datapub';
function MyApp() {
return (
< ResourceManager resource={resource} />
);
}
export default MyApp;
same for multi resource manager:
import React from 'react';
import { MultiResourceManager } from 'datapub';
function MyApp() {
return (
<MultiResourceManager resources={resources} />
);
}
export default MyApp;
/src/index.js
according to analysisHow to refactor /src/index.js
:
As we need to componentize this app, we need to export each component so that it can be imported and used from the client app:
// src/index.js
export { ResourceEditor, MultiResourceEditor } from './App';
export { FileUploader } from './components/FileUploader';
export { MetadataEditor } from './components/MetadataEditor';
export { SchemaEditor } from './components/SchemaEditor';
export { ProgressBar } from './components/ProgressBar';
We also can have an option for users who just wants to mount this app to the HTML:
// src/index.js
import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';
function mountResourceEditorApp([elementId, datasetId] = ['root', undefined]) {
if (!datasetId) {
throw error
}
ReactDOM.render(
<React.StrictMode>
<App ... />
</React.StrictMode>,
document.getElementById(elementId)
);
}
Have DataPub work for resource edit as well as resource create
This issue will probably require changes both here and in ckanext-blob-storage to pass in the details of the existing resource (either just the resource id (and dataset id?) and then DataPub code looks up the resource object from the API or by passing in complete JSON object).
Resource edit page works correctly
resource_update
rather than resource createIf I don't provide any config values the default values are not being used.
Use the following config values:
{
authToken: "be270cae-1c77-4853-b8c1-30b6cf5e9878",
api: "http://localhost:5000",
lfs: "http://localhost:5001", // Feel free to modify this
organizationId: "myorg",
datasetId: "data-test-2",
}
{
api: null,
authToken: null,
datasetId: "test-id",
lfs: null,
organizationId: null,
}
The table schema component renders without scroll(x) and scroll(y).
When the user uploads a file to the storage the upload progress expects to be tracked showing a progress bar to the user with percentage done, time remaining and kb/s sec.
When creating a new resource in a Dataset I want to be able to copy a table schema from an existing resource (if there is an existing resource) so that I don't have to re-enter the whole table schema again
In terms of flow i imagine something like:
schema
attribute!)
NB: we don't need this feature to show up on resource editing (though no harm in it being there i suppose so could have it)
New React-based app/component for creating and editing one or more resources add adding to a Dataset. This replaces CKAN classic resource create and edit but ONLY for ckan inistances using ckanext-external-storage
(which we hope will soon be most or all of our CKAN instances).
NB: we can assume dataset already exists. In later evolution we will extend to include dataset creation etc.
Design features:
schema
on a resource (as per Frictionless pattern) rather than in Data Dictionary and associated with DataStore.TODO - see https://tech.datopian.com/publish/
In a CKAN classic instance using ckanext-external-storage the resource editor is a React app and:
schema
attribute)Leftovers
ckanext-external-storage
also upgrade to AirCan there is no need. => Probably no needgraph TD
choose[Select File or Link File]
choose --> select[Select file to upload]
choose --> link[Link file to upload]
select --> openfile(Open File => data.js Resource/File object)
openfile --> upload[Upload in Background + Upload UI]
openfile --> metaeditor[Metadata Editor]
openfile -.if tabular.-> schemaeditor[Schema Editor]
upload --completes--> save
metaeditor --no errors--> save
schemaeditor --no errors--> save
link -. if you can open or just create default.-> openfile
cancel[Cancel this resource]
classDef done fill:#21bf73,stroke:#333,stroke-width:1px;
classDef nearlydone fill:lightgreen,stroke:#333,stroke-width:1px;
classDef inprogress fill:orange,stroke:#333,stroke-width:1px;
classDef next fill:lightblue,stroke:#333,stroke-width:1px;
class choose,upload,metaeditor,select done;
class select,nearlydone nearlydone;
class schemaeditor,inprogress inprogress;
class openfile next;
v0.1 JS app (macro component) for adding a single resource to a dataset with file upload
v0.2
v0.3
v0.4: Integrate this all together into a Resource editor
v0.5 Add multiple resources at once
v0.6
v??
GsaApp = DatasetManager
GatesResourcEditorApp = Multi
DatasetManager
dataset
ui
...
ResourceManager(resource={}, dataset_id=None)
state: {
// only needed if this component saves the resource
// (usually parent component e.g. DatasetManager will deal with saving ...)
dataset_id: ...
// follows frictionless Data Resource
// if existing this will have lot of content o/w empty
resource: {...}
ui
fileOrLink: file | link
uploadComplete
...
}
MultiResourceManager(dataset)
DatasetMetadataManager
open(HTML File object or URL) => if you can't open the url just give { path: url }
resource
{
name
title
format
path
sample: // first 10/20 rows ...
schema
}
Adding JS module - https://docs.ckan.org/en/2.8/theming/fanstatic.html - but this is for adding custom JS module. What we need is to replace current "resource editor" UI/UX with our React app.
Current flow:
sequenceDiagram
Client->>CKAN: request `/datast/new`
CKAN->>Client: HTML form + JS modules (jquery etc.)
Client->>CKAN: Post form data to `/dataset/new`
CKAN-->>CKAN: Create dataset in the db with state as draft
CKAN->>Client: 302 to `/dataset/new_resource/{dataset-name}`
Client->>CKAN: Request `/dataset/new_resource/{dataset-name}`
CKAN->>Client: HTML form + jquery module for file uploader
Client->>CKAN: Post form data to `/dataset/new_resource/{dataset-name}`
CKAN-->>CKAN: Create a resource in the db + update dataset's state to active
CKAN->>Client: 302 to `/dataset/{dataset-name}`
Client->>CKAN: Requst `/dataset/{dataset-name}`
CKAN->>Client: HTML of dataset page
Note that when uploading file, you should post multipart/form-data.
What do we need to replace "Resource Editor":
{% resource 'my_extension/bundle.js' }
React App combining several lower level components into an app / flow: Upload, Edit metadata, Edit table schema, Save
Need to mock this up a bit.
Could be stepped or all in one page (maybe hide show for some of it so it would work with multiple resources).
This issue is related datopian/datapub-nhs#14
After some analysis and tests to upload large files, the root cause to upload large files happens because of the code below.
Note: After removed this part, I tested with 600Mb and 1.gb and it works very well.
try {
const rowStream = await resource.rows({ size: 20, keyed: true })
resource.descriptor.sample = await toArray(rowStream)
await resource.addSchema()
} catch (e) {
console.error(e)
}
Why exist the try-catch above? The code above is necessary to be able to render data into the table schema component, but for some reason is consuming a lot of memory, and sometimes crash the browser with no errors on the console dev tools.
Another error: in one of our tests we also try to upload a different format and it failed, and showed a parser error, as you can see on the screenshot below.
When the user drags and drops multiple files, the user wants to see all the files with each status independently and when the user clicks on card status, in metadata editor expects to show the file information.
e.g. prob something like ...
We need to insert DataPub resource editor into the pre-existing flow of CKAN resource editing (using https://github.com/datopian/ckanext-blob-storage to override CKAN templates).
graph TD
dnew[Click Dataset New]
dmeta[Dataset Metadata]
dnew --> dmeta
rnew[New Resource]
dmeta --saves dataset as draft--> rnew
rnew --finish--> dpage[Dataset Page]
rnew --add another--> rnew
rnew --previous-->dmeta
dedit[Click Edit Dataset] --> dupdate[Update Dataset Metadata]
dupdate --save/update--> dpage
dedit --resources--> res[Resources page]
res --add new--> rnew
res --click on existing resource --> redit[Resource Edit]
redit --delete--> dpage
redit --update--> resview[Resource view page]
graph TD
dnew[Click Dataset New]
dmeta[Dataset Metadata]
dnew --> dmeta
rnew[New Resource]
dmeta --saves dataset as draft--> rnew
rnew --save and publish + redirect--> dpage[Dataset Page]
dedit[Click Edit Dataset] --> dupdate[Update Dataset Metadata]
dupdate --save/update--> dpage
dedit --resources--> res[Resources page]
res --add new--> rnew
res --click on existing resource --> redit[Resource Edit]
redit --delete--> dpage
redit --update--> resview[Resource view page]
Resource editor
graph TD
start --> remove
remove --> showupload[Show Upload/Link options]
Have a component for "Link a file" in Resource add
Refactor due to recent updates in ckan-client-js so we can create/update resources + push to blob.
Acceptance
Tasks
Save resource metadata (including sha-256 field which indicates this is stored via giftless) into backend (CKAN).
Preliminaries in external libs
Work here
createResource(f11sResourceMeta, datasetIdOrName)
/dataset/{dataset.name}
active
from draft
and call push
(package_update)When the user Drag & drop multiple files, the uploading process expects to start for all files automatically and start to track the progress independently for each file.
See #1 for background.
Questions:
Data Publishing workflows are by their nature highly custom - though with many common components. For example, one site the resource editor may have one set of fields, and in another they have a different set of fields (as per the discussion in
http://tech.datopian.com/publish/)
Our approach to this is for DataPub to provide a rich set of common components for building those flows and a rich set of template applications that show how to combine those components.
Finally, to integrate a DataPub based app into CKAN v2 one needs a CKAN extension that would insert the React app into CKAN templates in right locations. For that we are creating https://github.com/datopian/ckanext-datapub
Right now the only app we have is a "demo" app in DataPub that shows off most of the components. That app is then embedded into ckanext-blob-storage
Where we want to go:
datapub-xxx
for each use casecreate-react-app
2) Change your App.js in this way etc). Also has demo (storybooked?) example apps that users of DataPub can use as a basis for their apps)ckanext-datapub
that can be customized to integrate a given custom uploader (ckanext-blob-storage
can probably revert to its original simplest state of just having different upload)ckanext-datapub
exists and is functional for integrating datapub based apps into CKAN v2. https://github.com/datopian/ckanext-datapubckanext-blob-storage
is back in its original state (? - we may want to always use DataPub going forward and not worth maintaining blob storage resource editor )
ckanext-datapub
approachThere are two areas of integration:
DataPub based app could replace one or both of these. (Note DataPub based app may have other steps that CKAN does not currently have but we are jus focusing on current CKAN v2 end).
We need a CKAN extension that can replace one or both of Dataset editing and Resource editing by handing off to a React app. The React app source code will likely be located elsewhere (need to specific its build state for integration).
ckanext-datapub
exists - https://github.com/datopian/ckanext-datapubIntegrate the ckan3-js-sdk with this code and refactor so that we can upload a single file to the storage.
We want to research if we can use uppy for our file upload. https://uppy.io/ is a fully featured client library for managing uploads with many adapters. Features include:
Answers to questions like:
This a discovery ticket to research how the pause/resume works and what is necessary to implement this functionality.
We have an integration test with actual file uploading ...
TODO
package.json
import { Upload } from 'datapub';
in my app and this will not throw error.We already have an example where we need to customize the publish flow:
Also, a resource can be restricted (eg, public vs private) but it is something custom and it is missing in DataPub. Eg: "restricted": "{"level": "public"}"
We need custom publish flows per client. e.g. this feature is specific to a given ckan deployment. Rather than modify ckanext-blob-storage we basically need to be able to install a custom React app.
ckanext-blob-storage
should have NO dependency on datapub.js. What we want is to pass a React app into CKAN or rather ckanext-blob-storage
.
You create
ckanext-publish-{client-name}
And inside of this is a React app using components from DataPub.js.
OR we have a way inside ckanext-blob-storage
to point to a react app.
Set up tests with Jest in the React application to test the upload flow. We use Jest because it is now default for React and has nice documentation, support, and tutorials.
2 kinds of tests:
New React-based app for creating and editing resources. This replaces CKAN classic resource create and edit but ONLY for ckan inistances using ckanext-external-storage
(which we hope will soon be most or all of our CKAN instances).
Design features:
schema
on a resource (as per Frictionless pattern) rather than in Data Dictionary and associated with DataStore.In a CKAN classic instance using ckanext-external-storage the resource editor is a React app and:
schema
attribute)Leftovers
ckanext-external-storage
also upgrade to AirCan there is no need. => Probably no needEst (remaining): 9d
Adding JS module - https://docs.ckan.org/en/2.8/theming/fanstatic.html - but this is for adding custom JS module. What we need is to replace current "resource editor" UI/UX with our React app.
Current flow:
sequenceDiagram
Client->>CKAN: request `/datast/new`
CKAN->>Client: HTML form + JS modules (jquery etc.)
Client->>CKAN: Post form data to `/dataset/new`
CKAN-->>CKAN: Create dataset in the db with state as draft
CKAN->>Client: 302 to `/dataset/new_resource/{dataset-name}`
Client->>CKAN: Request `/dataset/new_resource/{dataset-name}`
CKAN->>Client: HTML form + jquery module for file uploader
Client->>CKAN: Post form data to `/dataset/new_resource/{dataset-name}`
CKAN-->>CKAN: Create a resource in the db + update dataset's state to active
CKAN->>Client: 302 to `/dataset/{dataset-name}`
Client->>CKAN: Requst `/dataset/{dataset-name}`
CKAN->>Client: HTML of dataset page
Note that when uploading file, you should post multipart/form-data.
What do we need to replace "Resource Editor":
{% resource 'my_extension/bundle.js' }
"restricted": "{\"level\": \"public\"}"
{
"help": ".../api/3/action/help_show?name=resource_show",
"success": true,
"result": {
"cache_last_updated": null,
"package_id": "433aa98f-9dec-4052-a4b4-0b2590a9e5e8",
"datastore_active": false,
"id": "8ef6662d-5052-437c-9f5d-7704f8b5d88a",
"size": null,
"restricted": "",
"encoding": "utf-8",
"state": "active",
"schema": "{u'fields': [{u'type': u'yearmonth', u'name': u'YEAR_MONTH', u'format': u'default'}, {u'type': u'string', u'name': u'ODS_CODE', u'format': u'default'}, {u'type': u'integer', u'name': u'VMP_SNOMED_CODE', u'format': u'default'}, {u'type': u'string', u'name': u'VMP_PRODUCT_NAME', u'format': u'default'}, {u'type': u'integer', u'name': u'UNIT_OF_MEASURE_IDENTIFIER', u'format': u'default'}, {u'type': u'string', u'name': u'UNIT_OF_MEASURE_NAME', u'format': u'default'}, {u'type': u'integer', u'name': u'TOTAL_QUANITY_IN_VMP_UNIT', u'format': u'default'}], u'missingValues': [u'']}",
"dialect": "{u'quoteChar': u'\"', u'delimiter': u','}",
"hash": "c252dc3247713d7bd83c9b7a7729168273092231f736422c1a599e8dbf58a014",
"description": "",
"format": "CSV",
"last_modified": null,
"url_type": null,
"mimetype": null,
"cache_url": null,
"created": "2020-09-17T06:41:28.013104",
"url": "",
"mimetype_inner": null,
"position": 19,
"revision_id": "d6716aed-38e0-4126-9bde-ff6c244346e0",
"resource_type": null
}
}
First, i'm assuming we have a demo app using our component that we can view (or maybe we can use storybook ... https://storybook.js.org/docs/basics/exporting-storybook/)
Either way we want to have a simple demo app running at datopian.github.io/datapub/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.