delving/hub3

RDF discovery and publication platform

Hub3

Hub3 is an RDF publication and discovery platform written in Golang. Before the 1.0 release packages that can be of individual use will be split into stand-alone packages.

The goal of Hub3 is to provide an API Framework that makes it easy and predictable for webdevelopers to work with arbitrarily structured RDF and leverage semantic network technology.

The core functionality that it aims to provide can be summarised by the acronym SILAS:

• SPARQL proxy
• Index RDF
• Linked Open Data Resolver
• Aggregate and transform RDF
• Search RDF

Part of the design is to require as little external dependencies outside the compiled Golang binary as possible.

NOTE: this is currently a work in progress and APIs can change between releases.

Install

Hub3 is written in Golang, so you have to setup your Golang environment first, see Golang Installation.

After that you can glone it from github:

$ git clone [email protected]:delving/hub3.git $GOPATH/src/github.com/delving

Or use go get

$ go get github.com/delving/hub3

Start the server with the default configuration.

$ hub3 http

For development setup, see Develop.

For deployment instructions, see Deployment.

Changelog

Master

0.1

• First fully-functional public version

License

Copyright (c) 2017-present Delving B.V.

Licensed under Apache 2.0

hub3's Issues

quering an invalid inventoryID with paging=true causes a redirect loop

Querying /api/tree/<spec>/<invalid inventoryID>?paging=true causes a HTTP 303 response which redirects to the same url.

Proposed fix:
Add && searchRequest.Tree.Page[0] != 1 to hub3/hub3/server/http/handlers/search.go line 419
This way the redirect does not happen if the page already is page 1.

CVE-2024-37298

A vulnerability (CVE-2024-37298) has been identified in gorilla/schema. The version (v1.2.0) hub3 is using has been affected by it:

hub3/go.mod

Line 30 in ceae0ad

github.com/gorilla/schema v1.2.0

The vulnerability can be fixed by upgrading to version 1.4.1: https://github.com/gorilla/schema/releases/tag/v1.4.1

Add multi-tenant support

Currently, all data is already stored with an orgID but routing, searching and filtering do not take this value into account.

• update viper configuration to support multiple organizations/tenants
• update ikuzo services to accept orgID as well when datasetID is used
• add default orgID support to viper configuration for backwards compatibility

CVE-2024-6104

A vulnerability (CVE-2024-6104) has been identified in hashicorp/go-retryablehttp. The version (v0.7.1) hub3 is using has been affected by it:

hub3/go.mod

Line 32 in ceae0ad

github.com/hashicorp/go-retryablehttp v0.7.1

The vulnerability can be fixed by upgrading to version 0.7.7: https://github.com/hashicorp/go-retryablehttp/blob/main/CHANGELOG.md#077-may-30-2024

Extend webresource service and store interface

The ikuzo/service/x/webresource package is now a base implementation. The service needs to be extended to support multiple storage backends through the webresource.Store interface.

Add dataset.Store implementation for `asdine/storm`

Currently, the dataset information is stored in (storm)[https://github.com/asdine/storm]. The implementation details need to be extracted from hub3.models and added to the ikuzo.storage.x.bbolt package.

Use v2 resource index to support LDF queries

The v2 resource index can be used to answer Linked Data Fragment queries. This index should be used instead of the dedicated fragment index that is now deprecated.

CVE-2020-14040 (High) detected in github.com/microsoft/hcsshim-fc27c5026e6ff001dc1b171b99bda7bb3dcf6e78

CVE-2020-14040 - High Severity Vulnerability

Vulnerable Library - github.com/microsoft/hcsshim-fc27c5026e6ff001dc1b171b99bda7bb3dcf6e78

Windows - Host Compute Service Shim

Dependency Hierarchy:

• github.com/testcontainers/testcontainers-go-df72de99f67fc54d1b3c02fbbb9a8a68758af13e (Root Library)
  • github.com/docker/docker/pkg/archive-a770dc191eea0a88236b4fb5575fe92efd356800
    • github.com/containerd/containerd/sys-d184a0a3430dc4a17a47cce37fb36126ac0c699a
      • ❌ github.com/microsoft/hcsshim-fc27c5026e6ff001dc1b171b99bda7bb3dcf6e78 (Vulnerable Library)

Found in HEAD commit: 58412d6eb876957e97397b033f9e18e546fb7b3d

Vulnerability Details

The x/text package before 0.3.3 for Go has a vulnerability in encoding/unicode that could lead to the UTF-16 decoder entering an infinite loop, causing the program to crash or run out of memory. An attacker could provide a single byte to a UTF16 decoder instantiated with UseBOM or ExpectBOM to trigger an infinite loop if the String function on the Decoder is called, or the Decoder is passed to golang.org/x/text/transform.String.

Publish Date: 2020-06-17

URL: CVE-2020-14040

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14040

Release Date: 2020-06-17

Fix Resolution: v0.3.3

---

Step up your Open Source Security Game with WhiteSource here

Move RAML api-console to dedicated repository

Including the RAML api-console should be optional to be included in the static resources. Therefore, it is best to move it to a dedicated repository.

• remove ./ikuzo/static/api-console
• remove api-console command from Makefile
• remove ikuzo/internal/assets/filesystem_vfsdata.go

Investigate compliance with EAD3

Currently, we are using EAD 2002, but the recommendation is now that we should use EAD3 whenever possible. This issue should investigate what is needed to support both.

For more information on EAD3, see https://www.loc.gov/ead/EAD3taglib/EAD3.html

Migrate Linked Data Fragments support to resource service

Linked Data Fragments support must be migrated from hub3.fragments package to the resource service.

• #17: remove dedicated fragment index support
• #18: use v2 resource index to support LDF queries
• #19: use v2 resource index to support Linked Open Data resolving

Support direct RDF uploads

Through a path that specifies the orgID and the datasetID it should be possible to directly upload RDF into a dataset. The dataset needs to be created with the correct workflow transformer.

This RDF upload handler can be taken as a starting point: https://raw.githubusercontent.com/delving/hub3/v0.1.11/hub3/server/http/handlers/rdf.go

Use revision service for orphan control during indexing

Using the revision service during indexing will bring the following benefits:

• only changed records can be submitted for indexing
• orphan control can delete specific records instead of using an delete query (which causes unwanted side-effects in clustered setup)
• a full reindex can happen without having to reprocess the source data

Extend namespace service and store interface

The ikuzo/service/x/namespace package is now a base implementation. The service needs to be extended to support multiple storage backends through the namespace.Store interface.

Use errgroup to retrieve remote webresource RDF graphs

Migrate `hub3/models` to dataset service

Currently, hub3/models models mixes various underlying storages for datasets. In particular, 'ElasticSearch', 'Triple-Stores', and the dataset storage.

This issue splits this functionality over the relevant ikuzo.services and moves implementation details into ikuzo.storage packages.

The dataset storage should be defined in `ikuzo.service.datasets.Store' interfaces.

• #14: Add dataset.Store implementation for asdine/storm
• #15: Add dataset.Store implementation for go-pg/pg

Use v2 resource index to support Linked Open Data resolving

Currently, the dedicated fragments index is used for resolving of Linked Open Data resources. Because the dedicated fragment index is deprecated this functionality should be supported by the v2 resource index.

• migrate lodKey from fragment mapping to v2 mapping
• update lod service query to use v2 index
• add lodKey to fragmentBuilder and resource model

Require `context.Context` for service store interface

We should be able to interrupt calls to remote services. Therefore we will update all method signatures of ikuzo/service packages that define Store interfaces with ctx context.Context.

This is in preparation of GRPC client interfaces to remote services.

Remove dedicated fragment index support

The functionality of the v2 resource index has almost come to feature parity with the dedicated Linked Data Fragment index. In order to reduce resources, the dedicated fragment index should be removed and no longer be used during index.

Import elements to remove:

• fragment configuration object in viper config
• fragment switches from dataset service and bulk service
• fragment elasticsearch mapping and index creation during setup

Extend OAI-PMH service and store interface

The ikuzo/service/x/oaipmh package is now a base implementation. The service needs to be extended to support multiple storage backends through the oaipmh.Store interface.

Add support for multi-tenant routing

Add support '/org/{orgID}' based routing. For backwards-compatibility, when this route prefix is not used it will revert back to the default organization.

• add organization middleware to inject default organisation into the request when non-multi-tenant paths are requested
• add tenant-aware routes to all services that use ikuzo.RouterFunc to register http Handlers.

Add dataset.Store implementation for `go-pg/pg`

Add dataset.Store implementation for (go-pg/pg)[https://github.com/go-pg/pg].

Use RAML for search REST API documentation

(RAML)[https://raml.org] will be used for API documentation.

This issue will add RAML documentation for the v1 and v2 search APIs.

Add OAI-PMH harvester to service

The harvester should harvest OAI-PMH endpoints and store the records in the "pockets" source format that is accepted by Narthex.

The harvester from 'hub3ctl' can be used as a starting point: https://raw.githubusercontent.com/delving/hub3/v0.1.11/hub3/hub3ctl/cmd/oaipmh.go.

Add support for running hub3 in DataNode mode.

This issue introduces support to run the hub3 in stateless mode behind a load-balancer. It still depends on a DataNode that can be configured via the viper configuration.

By default the ikuzoctl serve is started in DataNode mode. If you want to
run the ikuzo server in stateless mode you can add a dataNodeURL
config value with the URL to the datanode to the configuration. All
statefull routes are transparently handled via an internal reverse
proxy.