noi-techpark/it.bz.opendatahub.databrowser

Explore and navigate through Open Data you need to build your next service.

Data Browser for the Open Data Hub

This is the repository for the Open Data Hub Data Browser.

Table of Contents

• Getting started
• Build for production
• Docker environment
• Information

Getting started

These instructions will get you a copy of this repository and prepare it for development of the Data Browser.

Prerequisites

To build the projects in the repository, the following prerequisites must be met:

• Node.js, at least v14.16 or v16 LTS (see https://nodejs.org/en/about/releases/)

Installing

Get a copy of the repository, e.g. by cloning it from the following location:

git clone https://github.com/noi-techpark/it.bz.opendatahub.databrowser

Change into the databrowser directory:

cd it.bz.opendatahub.databrowser/databrowser

Install the dependencies:

npm install

Copy the file .env.example to .env and adjust the configuration parameters if required.

Start the development server:

npm run dev

On successful start, the Data Browser application can be found at http://localhost:3000.

Build for production

Before proceeding, please ensure that you are in the databrowser directory.

Build the production version:

npm run build

The result, found in the ./dist folder, can be deployed to a web server.

Launch local production server:

npm run start

Docker environment

Before proceeding, please ensure that you are in the databrowser directory.

For the project a Docker environment is already prepared and ready to use with all necessary prerequisites.

These Docker containers are the same as used by the continuous integration servers.

In order to use the Docker environment, you must have Docker installed locally on your machine, together with Docker Compose.

Start and stop the containers

Before start working you have to start the Docker containers:

docker-compose up --build --detach

After finished working you can stop the Docker containers:

docker-compose stop

Running commands inside the container

When the containers are running, you can execute any command inside the environment. Just replace the dots ... in the following example with the command you wish to execute:

docker-compose exec node /bin/bash -c "..."

Some examples are:

docker-compose exec node /bin/bash -c "npm install"

# or

docker-compose exec node /bin/bash -c "npm run dev"

# or

docker-compose exec node /bin/bash -c "npm run build"

Information

Support

For support, please contact Christian Gapp or NOI Techpark Südtirol - Alto Adige.

Contributing

Please take a look at the Contributor Guidelines.

You can find an introduction on how to start developing in the contribute-to-development and guideline files.

Versioning

This project uses SemVer for versioning. For the versions available, see the tags on this repository.

License

The code in this project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE license, Version 3. See the LICENSE file for more information.

Reuse

This project aims to be REUSE compliant. All files contained in the repository either have to be mentioned in the dep5 file or preferably contain a header containing SPDX copyright and licensing information.

The project is checked for licensing and copyright information before every commit by a pre-commit hook if the reuse-helper-tool is installed as well as in the github CI which will fail if REUSE standards are not satisfied. Instructions for the installation of the helper tool here. If you want to use the helper tool via pipx run consider setting an alias for the command reuse to enable the pre commit hook to execute the lint.

alias reuse="pipx run reuse"

For more information on how to make new files REUSE compliant check the project guidline.

Authors

• Christian Gapp - Initial work - gappc
• AboutBits - Initial work - AboutBits
• FlashBeing - Lots of contributions FlashBeing
• NOI Support Team - Configurations and tests

Acknowledgements

• NOI Techpark Südtirol - Alto Adige

it.bz.opendatahub.databrowser's Issues

Developer view / tools - Generic proposals

• talk about how the developer view could be improved
• what features should be implemented / would be nice to have (e.g. link to developer docs, ODH docs, ...)?

Mismatches on how GPS Info is stored on ODH Objects

As we discussed via mail yesterday the ODH tourism has for historical reasons three different Types of GPS Info stored

1. flat (only one GpsInfo possible, embedded directly in the object)

Gpstype, 
Latitude, 
Longitude, 
Altitude, 
AltitudeUnitofMeasure

Used on Regions, Accommodations etc...

2. GpsInfo as Object

GpsInfo :
{
Gpstype, 
Latitude, 
Longitude, 
Altitude, 
AltitudeUnitofMeasure
}

only used on Webcams (have to see maybe i can eliminate this case)

3. GpsInfo as List

GpsInfo :
{
[
Gpstype, 
Latitude, 
Longitude, 
Altitude, 
AltitudeUnitofMeasure
],
}

Used on ODHActivityPois etc...

all object should have the standard

4. GpsPoints

GpsPoints: {
position: {
Gpstype: "position",
Altitude: null,
Latitude: 47.0496988,
Longitude: 12.1266379,
AltitudeUnitofMeasure: "m"
}
},

I think the best solution would be to mark all Except 4 as obsolete.......
The Databrowser should have a GpsPoints Component for visualizing......

When it comes to Edit, here i have to think more about it, i suppose the Component for visualization is used also for Edit, or are there other Edit Components used?

@gappc PR is now ready for review. Ready to hear some feedback, if any.

    @gappc PR is now ready for review. Ready to hear some feedback, if any.

One thing I would consider to change is the small bar at the bottom of the component, but I'm not sure if and how to proceed. Is that part of CSS?

Originally posted by @BlueRyse in #167 (comment)

Detail View: switch between Configured and Non Configured view

We should give the possibility to switch between the configured detail view and the auto-generated view

Example use-case: If there is a dataset already configured and a software architect adds another 10 attributes to the API they won't show up in the configured view. So one could fall back to the auto-generated view and compare the two. Or just look at the new fields there.

EPIC: 2nd version of the Raw, Table and Detail view

The data view provided in the first MVP is a standard view that now needs to be optimized considering that the target user of this visualization, in the most cases, will be a user that isn't familiar with the data and that want to:

• have an overview about the data provided;
• understand which data are most interesting for his business.

Table View - Sorting

ODH seems to support "rawsort", at least for some endpoints (see https://github.com/noi-techpark/odh-docs/wiki/Using-rawfilter-and-rawsort-on-the-Tourism-Api#rawsort) => talk to Rudi

• is it feasible to use it for sort, e.g. because of performance considerations?
• is rawsort applicable onto each data property / column?
• are there other things to take into consideration for sort?

EPIC: Provide default configurations for all endpoints that are shown in the Databrowser

EPIC: Through configuration it is possible to define which properties the Databrowser shows for a certain endpoint.

• we should try to complete the configuration for all endpoints that are presented in the Databrowser
• at the moment, the configuration is defined just for the properties specified in the UI designs => should we define a broader configuration, from which e.g. a user may select the properties to show?

Tourism API inconsistencies

Message box

In order to show messages, warnings and errors we need a place to put such information.

This could be the toolbox with a dedicated tab.

Use cases would be:

• Error messages from the API i.e. validation errors, etc
• Information on endpoints, i.e. "this endpoint is deprecated"
• Additional information on realtime data i.e. latency, etc.

EPIC: Table View - Filters

In the next issue it should be handled the following points concerning the filters:

• what kind of filters are missing (at the moment we have fixed-value filters, e.g. for the language, but there are other filter types that need e.g. to load data from remote)
• at the moment, filters are stateless and created / rendered each time anew when they are opened. It could be necessary for more advanced filters to not have that behavior (e.g. for complicated filters that need to fetch data before they can be used). We need to discuss if we should we make the behavior configurable
• do filters need the "Cancel" / "Save" buttons or wouldn't it be nice if the filter settings are applied immediately as the user changes its settings?

Table View - Table data load view

Table data load view (Studio Hug proposal: https://app.zeplin.io/project/619297bb195e0113987fbd5c/screen/61a0fc9a6bd1e2896d16150f); note, that the proposal may lead to a lot of page movement, because on each table load (e.g. pagination), the whole table is replaced first by the "loading" info, which is then replaced by a table after successful data load - maybe thats ok, but maybe there exist better solutions, e.g. loading indicators inside table

EPIC: 1st version of the Edit View Implementation

As a Open Data Hub content manager I want an edit view in order to be able to see, manage, change and delete my data.

As a Data Browser 2.0 i want the deployment pipeline online in order to see the new functionalities in the testing environment.

Tracing / feedback

There are many advantages to trace at least errors that happen in the Databrowser application and to report them to the ODH team in an automatic way. For example, stability problems could be detected early on, before they affect to many users.

In addition, user feedback is often very valuable. We should make it as simple as possible for a user to give feedback

• should we add some tracing mechanism? If so, we need to discuss how that would work, what conditions to trace, what data to send back and many more
• should we add some feedback mechanism for users, e.g. feedback form, "give feedback" menu entry, ...

Markdown for Table/Detail/Raw View

The Table / Detail / Raw View header show information about a dataset. In the first MVP, we support only text for these texts.

This issue has been open, in order to understand if we should support a markup language (e.g. Markdown, HTML) in order to be able to add formatting / links to the header texts.

EPIC: User settings setup

As a data editor and data viewer logged in the Data Browser I want to configure my settings in order to adapt as much as possible the Data Browser to my needs.

As a Open Data Hub PM I want that all Table View configuration are configured in order to provide a better experience to the Data Browser users.

deadline: 31st August 2022

Update dependencies

For example vue, vite, ...

ODH data structure consistency

• most endpoints in ODH tourism API provide an Id property (uppercase I, followed by lowercase d) for each data entry, but Rudi told us that there is an endpoint that uses id (both chars lowercase) as property, if I (Gapp Christian) remember correctly it was for an AlpineBits endpoint and the reason was, that Id was already used => exceptions like this complicate the whole implementation
  • are there more exceptions like this? => talk to Rudi
• The tourism API uses a different pagination type than the mobility API (page based vs. offset based)

#7756: Bedeutung von einigen Attributen der Wetter API

We asked the province about some Attributes in the weather API that aren't described in their documentation.

• Update the issue with the link ( https://meteo.provincia.bz.it/stazioni-meteo-valle.asp ) @BlueRyse
• we received the full name of the attributes we requested
• Create documentation (https://docs.google.com/spreadsheets/d/1Gryq2eyWa36mkkuiayLzIT7mMouxALMReudAj2mYg-E/edit#gid=924508651) @BlueRyse
• Open PR to describe these attributes in the swagger (noi-techpark/odh-api-core#325) @BlueRyse
• Check if the descriptions in the docs are valid (ticket 7756) @BlueRyse
• Open a PR to improve current descriptions and add some more (noi-techpark/odh-api-core#326)

Datasets not rendering when logged in.

After making a brand new account (Registered and verified E-mail), when entering any dataset it suddenly doesn't render anymore but instead only shows: ERROR {}. The console logs:

--------ERROR CAPTURED TypeError: Cannot read properties of undefined (reading 'includes')
    at Proxy.<anonymous> (index.a2eed0a1.js:48:55729)
    at index.a2eed0a1.js:48:55794
    at Array.find (<anonymous>)
    at Proxy.<anonymous> (index.a2eed0a1.js:48:55781)
    at Proxy.hasCreatePermission (index.a2eed0a1.js:105:55771)
    at Bf.fn (index.a2eed0a1.js:9:1311)
    at Bf.run (index.a2eed0a1.js:2:16276)
    at get value [as value] (index.a2eed0a1.js:2:24761)
    at Object.get (index.a2eed0a1.js:2:18627)
    at Bf.fn (DatasetPage.dd077f2d.js:3:17612)

TypeError: Cannot read properties of undefined (reading 'includes')
    at Proxy.<anonymous> (index.a2eed0a1.js:48:55729)
    at index.a2eed0a1.js:48:55794
    at Array.find (<anonymous>)
    at Proxy.<anonymous> (index.a2eed0a1.js:48:55781)
    at Proxy.hasCreatePermission (index.a2eed0a1.js:105:55771)
    at Bf.fn (index.a2eed0a1.js:9:1311)
    at Bf.run (index.a2eed0a1.js:2:16276)
    at get value [as value] (index.a2eed0a1.js:2:24761)
    at Object.get (index.a2eed0a1.js:2:18627)
    at Bf.fn (DatasetPage.dd077f2d.js:3:17612)

As a Data Browser 2.0 developer I want to have the Open Data Hub Keycloak properly configured in order to integrate it in the Data Browser authentication.

Table View - Search

• what data / properties can be and should be searched for in ODH tourism and mobility?
• search performance

Implement tree-traversal rendering for (auto) generated DetailView

For DetailViews with (auto) generated DetailView (i.e. where the ViewConfig is generated from an OpenAPI document), the data should be shown as tree.

Each node is either rendered according to a matching renderer, or it is further traversed. This happens recursively on the whole tree, starting from the root node.

Detail View: User Friedly Detail View

• default-attributes to be shown on detail view are defined per dataset
• maybe we need a Overview Tab

As a Data Browser data consumer I want to change the width of the single colums, to better fit the content in the colums.

• discuss how the feature should work on desktop and especially on mobile
• should column resizes be stored somewhere, such when a user reopens a Table View it is presented with the prior column sizes?

Table View - Empty table visualization

Empty table visualization, e.g. if filter settings lead to zero results (Studio Hug proposal: https://app.zeplin.io/project/619297bb195e0113987fbd5c/dashboard?sid=61929830770bc21914081664) => if we implement it as proposed, there is no way to e.g. reset / modify filters, because there is no table visible at all

As a Data Browser 2.0 developer I want to have the repository in order to commit the code.

Tasks:

• create the repository
• send the repository to Gapp

EPIC: 1st version of the No Config View

As a data provider I want to see the data that I share as soon as they are exposed from the open data hub without the need to wait for any configuration from the Open Data Hub Team.

• #90 The data should be listed by using default visualization components for the following data types:
  • String
  • Numbers
  • Boolean
  • GPS coordinates (Point on a map) --> Optional
  • Image --> Optional
• #91 Table automatically generated from API schemas shown in swagger
• #92 Detail view automatically generated from API schemas shown in swagger

Detail View: Possibility to “show/hide empty fields”

Raw view - Generic proposals

The raw view shows the JSON output for one specific dataset

• talk about current solution and possible improvements
  • proposal: "show all properties" / "hide all properties"
  • proposal: "copy JSON to clipboard" button (in addition to "Download JSON" on the bottom)
• Raw data load view (similar discussion as in "Table data load view")

Detail View: Optimization of tab-navigation and back-navigation to table-view

The navigation between Detail, Raw and Table view should be optimized accordingly to the feedback that Studio Hug will provide.

Table View - Show max 2 lines of data on each table cell

• how can one see what the full cell content is (column resize? popup?, ...)?
• how should this work on mobile?

Component to render GPS coordinates on a Map

As a data browser developer I want a map component to be used to show GPS coordinates on a map.

The web component should render:

• render one or more GPS point accordingly to the number of points that is passed to the web component;
• render one or more GPS tracks (e.g. lifts, bike tour, ski slopes, etc.) accordingly to the number of points that is passed to the web component (dedicated issue has been created: #361 );

Once the Web Component is developed We should remember to properly configure it in all Detail Views

Web Component usage

At the very early stages of the project there was the idea to use / support WebComponents. To simplify development, no WebComponent was developed or used in the Databrowser MVP 1

• should we re-introduce WebComponent support (adds complexity to the project and may be some effort)?

Table View: Full width view

As a data viewer I want to avoid empty spaces on the screen while I'm looking at the Table View.

Note: Implementing this issue we have to consider that in the future we may need some space to add the filters.

NoConfig View: Detail view automatically generated from API schemas shown in swagger

Functionality to be included in the first iteration

• Field order
  • Show attributes in order as they appear in the JSON
  • If there is an attribute called "Id" show it as first field
  • If there is an attribute called "Name" or "Title", show it as second field
  • If an attribute has an "unknown type", go iteratively to the next level until finding one of the known types (string, number, boolean)

Note: For MVP2 the detail view will have 1 column only, knowing that the hierarchical view of sub-attributes can/might invoke a horizontal scrolling

As a Open Data Hub PM I want that all Detail View configuration are configured in order to provide a better experience to the Data Browser users.

Deadline: 30th September 2022

Table View: sticky header and vertical scrolling

At the moment, the table headers are not always visible, e.g. if the user scrolls down. This is not optimal because a user scrolling down in the Table View is not able to determine which column shows which value. A better solution (sticky headers? => hard to implement) should be discussed and implemented

NoConfig View: Show data provided by the Open Data Hub endpoints by using only default components

In the first iteration of this functionality the following components should be included.

• The data should be listed by using default visualization components for the following data types:
  • String
  • Numbers
  • Boolean
  • GPS coordinates (Point on a map) --> Optional
  • Image list (Image Gallery) --> Optional
• [ ]

NoConfig View: Table View automatically generated from API schema shown in swagger

The automatically generated Table View will include the following functionalities:

• Default column order
  • Show attributes in order as they appear in the JSON
  • If there is an attribute called "Id" show it as first column
  • If there is an attribute called "Name" or "Title", show it as second column
  • If an attribute has an "unknown type" only show three points [...] or any other placeholder
• There will be a horizontal scrolling when showing all attributes in the table.

Improve Documentation

Introducing new users / developers to the Databrowser should be as easy as possible, proper documentation and examples should be created

• think about user documentation e.g. in Databrowser application (may also just be a link to an external page)
• think about how users could get help as easy as possible
• think about how to help developers to dive into the project and become productive (docs, demos, example code, best practices, guidance for pull requests, ...)

Detail View - Generic proposals

The detail view presents more details to one specific dataset

• discuss proposal by Alex to show a nice overview of the dataset as entry point to the Detail View (e.g. first category on the left could be "Overview") (see issue #182)
• discuss if to show empty fields:
  • if empty fields are not visible, how does one know that data for a certain field is missing? => maybe add toggle to "show all" / "show fields with data only"?
• discuss if to show categories on the left, if all of their properties are empty (same argument as for empty fields)
• discuss how to display further properties
• Detail data load view (similar discussion as in "Table data load view")

Hidden Paging on ODH Calls

Hi all

For easier integration on the Databrowser i added a hidden paging on several ODH Endpoints.
The Paging is used when the parameter pagenumber=1 is passed, then the result data is wrapped into a Result object under the Items List. I had to be careful because of not breaking the current api responses,

example
https://api.tourism.testingmachine.eu/v1/Region vs
https://api.tourism.testingmachine.eu/v1/Region?pagenumber=1

On this endpoint now there is paging available
https://api.tourism.testingmachine.eu/v1/MetaRegion?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Region?pagenumber=1
https://api.tourism.testingmachine.eu/v1/ExperienceArea?pagenumber=1
https://api.tourism.testingmachine.eu/v1/TourismAssociation?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Municipality?pagenumber=1
https://api.tourism.testingmachine.eu/v1/District?pagenumber=1
https://api.tourism.testingmachine.eu/v1/SkiArea?pagenumber=1
https://api.tourism.testingmachine.eu/v1/SkiRegion?pagenumber=1
https://api.tourism.testingmachine.eu/v1/WineAward?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Weather?pagenumber=1
https://api.tourism.testingmachine.eu/v1/WeatherHistory?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Weather/District?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Weather/Measuringpoint?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Weather/SnowReport?pagenumber=1
https://api.tourism.testingmachine.eu/v1/ODHTag?pagenumber=1
https://api.tourism.testingmachine.eu/v1/Tag?pagenumber=1

@gappc what do you think?

Mobile and responsive improvements

• improve mobile / responsive visualization
• talk about some challenges in mobile / responsive (e.g. table handling, optimize space to show data, ...)

Build broke due to PR #131 merge

@BlueRyse, @lukmay: unfortunately the merge of PR #131 broke the build (see https://github.com/noi-techpark/it.bz.opendatahub.databrowser/runs/8232015443?check_suite_focus=true).
The issue seems to be that number / boolean is not assignable to the href of the link element.

It would be nice if you could provide a fix: just remove the types boolean | number from the UrlCell component (see here:

it.bz.opendatahub.databrowser/databrowser/src/domain/cellComponents/components/cells/UrlCell/UrlCell.vue

Line 14 in 3ab9439

text?: string | boolean | number;

)

Detail View: Possibility to switch between records

Warning we need to define what does previous and next means. The first iteration of this functionality should be as simple as possible and then optimized on top of the feedback of real users.

Databrowser: Support on Deployment

https://github.com/noi-techpark/it.bz.opendatahub.databrowser/tree/development

Table view - Horizontal Scroll

The current scroll behavior is not optimal (need to scroll at the bottom to be able to scroll horizontally) => need to discuss alternatives and find a better solution

Problem in Table view language EN

Looking at the table view in language EN it seems that the column Location and Languages are overlapping.

See image belo

Can you please have a look and solve this small bug?