namanmanish/iudx-catalogue-server

IUDX Data Discovery portal and Metadata store

iudx-catalogue-server

The catalogue is IUDXs data discovery and dataset metadata publishing portal. It allows data providers to publish their data resources by making an IUDX vocabulary annotated meta-data document describing their datasource and affiliated terminologies. The datasources publish their data to the IUDX Resource Server. It allows consumers of such data to easily discover such resources by means of powerful queries and consume the data from Resource Servers in an automated and machine interpretable way.

Features

• Search and discovery of data resources hosted on IUDX platform
• Support for text, geo-spatial, relationship, list and attributes searches
• Upload, delete and modify operations on catalogue objects (meta-information corresponding to resources)
• Stores meta-information as JSON-LD documents using published vocabulary and attributes
• Scalable, service mesh architecture based implementation using open source components: Vert.X API framework, Elasticsearch for data-base
• Hazelcast and Zookeeper based cluster management and service discovery

Live

The live running instance of the IUDX catalogue can be found here.

API Docs

The api docs can be found here.

Get Started

Prerequisite - Make configuration

Make a config file based on the template in ./configs/config-example.json

• Generate a certificate using Lets Encrypt or other methods
• Make a Java Keystore File and mention its path and password in the appropriate sections
• Modify the database url and associated credentials in the appropriate sections

Docker based

1. Install docker and docker-compose
2. Clone this repo
3. Build the images ./docker/build.sh
4. Modify the docker-compose.yml file to map the config file you just created
5. Start the server in production (prod) or development (dev) mode using docker-compose docker-compose up prod

Maven based

1. Install java 13 and maven
2. Use the maven exec plugin based starter to start the server mvn clean compile exec:java@catalogue-server

Redeployer

A hot-swappable redeployer is provided for quick development ./redeploy.sh

Keystore

The server requires certificates to be stored in Java keystore format.

1. Obtain certs for your domain using Letsencrypt. Note: Self signed certificates using openssl will also work.
2. Concat all pems into one file sudo cat /etc/letsencrypt/live/demo.example.com/*.pem > fullcert.pem
3. Convert to pkcs format openssl pkcs12 -export -out fullcert.pkcs12 -in fullcert.pem
4. Create new temporary keystore using JDK keytool, will prompt for password keytool -genkey -keyalg RSA -alias mykeystore -keystore mykeystore.ks
keytool -delete -alias mykeystore -keystore mykeystore.ks
5. Make JKS, will prompt for password keytool -v -importkeystore -srckeystore fullcert.pkcs12 -destkeystore mykeystore.ks -deststoretype JKS
6. Store JKS in config directory and edit the keyfile name and password entered in previous step
7. Mention keystore mount path (w.r.t docker-compose) in config.json

Testing

Unit tests

1. Run the server through either docker, maven or redeployer
2. Run the unit tests and generate a surefire report mvn clean test-compile surefire:test surefire-report:report
3. Reports are stored in ./target/

Integration tests

Integration tests are through Postman/Newman whose script can be found from here.

1. Install prerequisites
   • postman + newman
   • newman reporter-htmlextra
2. Example Postman environment can be found here
3. Run the server through either docker, maven or redeployer
4. Run the integration tests and generate the newman report newman run <postman-collection-path> -e <postman-environment> --insecure -r htmlextra --reporter-htmlextra-export .
5. Reports are stored in ./target/

JUnit Integration tests

Run tests in this order for JUnit based integration tests

1. Cleanup any previous test artifacts from the db mvn clean test -Dtest=ServerVerticleDeboardTest
2. Prepare the database with items for testing mvn clean test -Dtest=ApiServerVerticlePreprareTest
3. Test the APIServer mvn clean test -Dtest=ApiServerVerticleTest
4. Repeat step 1 to clean db of all test artifacts

Contributing

We follow Git Merge based workflow

1. Fork this repository
2. Create a new feature branch in your fork. Multiple features must have a hyphen separated name, or refer to a milestone name as mentioned in Github -> Projects
3. Commit to your fork and raise a Pull Request with upstream

License

MIT