janwielemaker/hdt

Interface to HDT

hdt's Issues

Predicate `hdt_string_id/4' does not deal with Unicode correctly

Predicate hdt_string_id/4 does not deal with Unicode characters correctly. As an MWE take the following N-Triples file:

<x:x> <y:y> "宇宙"@ja .

When used with the current version on master, we get the following result:

?- test.
"��"@ja

Predicate tets/0 is implemented as follows:

:- use_module(library(hdt)).

test :-
  hdt_create_from_file('test.hdt', 'test.nt', []),
  setup_call_cleanup(
    hdt_open(Hdt, 'test.hdt'),
    (
      hdt_string_id(Hdt, object, String, 1),
      format(user_output, "~s\n", [String])
    ),
    hdt_close(Hdt)
  ).

Should HDT be used as of 2022?

For non mission-critical stuff, is this pack usable in practice?
How likely is it that it will be further enhanced in the future?

Error when opening an HDT file whose name contains a non-ASCII character

Opening an HDT file with non-ASCII characters in the file name results in an error (notice the SUPERSCRIPT TWO Unicode character in the example):

?- [library(hdt)].
?- hdt_open(X, 'SP²B-10000.hdt', []).
ERROR: HDT: Error opening HDT file for mapping.
ERROR: In:
ERROR:    [8] hdt_open(_6454,'SP²B-10000.hdt',[])
ERROR:    [7] <user>

After renaming the same file to test.hdt it works:

?- [library(hdt)].
?- hdt_open(X, 'test.hdt').
Predicate Bitmap in 322 us
Count predicates in 1 ms 141 us
Count Objects in 375 us Max was: 949
Bitmap in 120 us
Bitmap bits: 10303 Ones: 5468
Object references in 2 ms 784 us
Sort lists in 3 ms 492 us
Index generated in 8 ms 307 us
X = <hdt>(0x564949f11320).

Bug in hdt_search

I have a reproducible fatal error in the hdt interface. For some reason searching with a predicate and object, returns more results than simply searching for the object (in this case), and eventually gives you bounds check error.

The turtle file to reproduce this is:

@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .                                            
@prefix dcog: <https://datachemist.net/ontology/dcog#> .
@prefix dcogrel: <https://datachemist.net/ontology/dcogrel#> .
@prefix prov: <http://www.w3.org/ns/prov#> .                                                       
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .                                       
@prefix ipg: <https://datachemist.net/ontology/ipg#> .
@prefix note: <https://datachemist.net/uks/annotation/> .
@prefix inst: <https://datachemist.net/uks/candidate/> .
@prefix uksupply: <https://datachemist.net/uks/ontology/uk_supply_chain#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .

uksupply:Supplier a owl:Class;
  rdfs:comment "A supplier relationship"@en ;
  rdfs:subClassOf dcogrel:Relationship.  

uksupply:IndustryType a owl:Class; 
  rdfs:subClassOf dcog:Entity ;
  rdfs:comment "A type of industry"@en .

uksupply:supplier_source a owl:ObjectProperty ; 
  rdfs:comment "Source of a supplier relationship"@en ;
  rdfs:domain uksupply:Supplier ;
  rdfs:range uksupply:IndustryType . 

uksupply:supplier_target a owl:ObjectProperty ; 
  rdfs:comment "Target of a supplier relationship"@en ;
  rdfs:domain uksupply:Supplier ;
  rdfs:range uksupply:IndustryType . 

uksupply:supplier_value a owl:DatatypeProperty ;
  rdfs:comment "Value of a supplier relationship"@en ;
  rdfs:range  xsd:float ;
  rdfs:domain uksupply:Supplier .

And the following interaction demonstrates the problem:

?- hdt_open(HDT,'/home/me/mo.hdt',[]), hdt_search_id(HDT,X,Y,1).
HDT = <hdt>(0x28d5860),
X = Y, Y = 4 ;
HDT = <hdt>(0x28d5860),
X = 5,
Y = 4 ;
false.

?- hdt_open(HDT,'/home/me/mo.hdt',[]), hdt_search_id(HDT,X,3,1).
HDT = <hdt>(0x2969930),
X = 4 ;
HDT = <hdt>(0x2969930),
X = 5 ;
HDT = <hdt>(0x2969930),
X = 2 ;
HDT = <hdt>(0x2969930),
X = 4 ;
HDT = <hdt>(0x2969930),
X = 5 ;
HDT = <hdt>(0x2969930),
X = 6 ;
HDT = <hdt>(0x2969930),
X = 3 ;
HDT = <hdt>(0x2969930),
X = 1 ;
HDT = <hdt>(0x2969930),
X = 4 ;
HDT = <hdt>(0x2969930),
X = 5 ;
HDT = <hdt>(0x2969930),
X = 6 ;
HDT = <hdt>(0x2969930),
X = 6 ;
HDT = <hdt>(0x2969930),
X = 1 ;
HDT = <hdt>(0x2969930),
X = 3 ;
HDT = <hdt>(0x2969930),
X = 6 ;
HDT = <hdt>(0x2969930),
X = 4 ;
HDT = <hdt>(0x2969930),
X = 5 ;
HDT = <hdt>(0x2969930),
X = 1 ;
HDT = <hdt>(0x2969930),
X = 3 ;
terminate called after throwing an instance of 'std::runtime_error'
  what():  Trying to get an element bigger than the array.

Process prolog aborted (core dumped)

The bug can also be tickled using hdt_search instead of searching by id:

?- hdt_open(HDT,'/home/me/mo.hdt',[]), , hdt:hdt_search(HDT,X, 'http://www.w3.org/2000/01/rdf-schema#domain','https://datachemist.net/uks/ontology/uk_supply_chain#IndustryType').
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_source' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_target' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#Nonsense' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_source' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_target' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_value' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#Supplier' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#IndustryType' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_source' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_target' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_value' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_value' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#IndustryType' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#Supplier' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_value' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_source' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#supplier_target' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#IndustryType' ;
HDT = <hdt>(0x2103250),
X = 'https://datachemist.net/uks/ontology/uk_supply_chain#Supplier' ;
terminate called after throwing an instance of 'std::runtime_error'
  what():  Trying to get an element bigger than the array.

Process prolog aborted (core dumped)

I've used the prolog hdt library on multi-gig databases including millions of nodes with no problem, so this one has me stumped and quite surprised to find a failure on such a small example.

Can HDT suggestions be returned incrementally?

Is it possible to change hdt_suggestions/5 so that it non-deterministically enumerates all results? Alternatively, could it return a lazy list?

Sometimes there are very many terms with the same prefix and then the Length has to be set to increasingly large values.

Non-deterministic implementation of `hdt_suggestions`

In commit 8a3ea90 I tried to use the new, iterator-based implementation for retrieving suggestions. When I compile I get the following error, but handle is used at other places in the same file (including earlier positions) and then the compiler does not complain:

ERROR: c/hdt4pl.cpp: In function 'foreign_t pl_hdt_suggestions__4(PlTermv)':
ERROR: c/hdt4pl.cpp:434:29: error: 'handle' was not declared in this scope
ERROR:    switch(PL_foreign_control(handle))

How does handle work in this file?

PS: This commit assumes that the develop branch of hdt-cpp is used.

BitSequence build failure on arm32v7

I'm attempting to build the swipl Docker image and plugins on the arm32v7 architecture, but I encountered a build error with the hdt plugin.

The plugin build fails on arm32v7 (Raspberry Pi) with the following error:

%  [HDT] Compiling src/bitsequence/BitSequence375.cpp
ERROR: In file included from src/bitsequence/BitSequence375.cpp:30:0:
ERROR: src/bitsequence/../util/bitutil.h: In function 'bool hdt::bitget(uint32_t*, size_t)':
ERROR: src/bitsequence/../util/bitutil.h:86:13: error: redefinition of 'bool hdt::bitget(uint32_t*, size_t)'
ERROR:  inline bool bitget(uint32_t *e, size_t p) {
ERROR:              ^~~~~~
ERROR: src/bitsequence/../util/bitutil.h:69:13: note: 'bool hdt::bitget(size_t*, size_t)' previously defined here
ERROR:  inline bool bitget(size_t *e, size_t p) {
ERROR:              ^~~~~~
ERROR: src/bitsequence/../util/bitutil.h: In function 'void hdt::bitset(uint32_t*, size_t)':
ERROR: src/bitsequence/../util/bitutil.h:91:13: error: redefinition of 'void hdt::bitset(uint32_t*, size_t)'
ERROR:  inline void bitset(uint32_t * e, size_t p) {
ERROR:              ^~~~~~
ERROR: src/bitsequence/../util/bitutil.h:74:13: note: 'void hdt::bitset(size_t*, size_t)' previously defined here
ERROR:  inline void bitset(size_t * e, size_t p) {
ERROR:              ^~~~~~
ERROR: src/bitsequence/../util/bitutil.h: In function 'void hdt::bitclean(uint32_t*, size_t)':
ERROR: src/bitsequence/../util/bitutil.h:96:13: error: redefinition of 'void hdt::bitclean(uint32_t*, size_t)'
ERROR:  inline void bitclean(uint32_t * e, size_t p) {
ERROR:              ^~~~~~~~
ERROR: src/bitsequence/../util/bitutil.h:79:13: note: 'void hdt::bitclean(size_t*, size_t)' previously defined here
ERROR:  inline void bitclean(size_t * e, size_t p) {
ERROR:              ^~~~~~~~
Warning: At global scope:
Warning: cc1plus: warning: unrecognized command line option '-Wno-unknown-warning-option'
% Makefile:100: recipe for target 'src/bitsequence/BitSequence375.o' failed
% make[2]: Leaving directory '/usr/lib/swipl/pack/hdt/hdt-cpp/hdt-lib'
% make[2]: *** [src/bitsequence/BitSequence375.o] Error 1
% make[1]: *** [all] Error 2
% Makefile:63: recipe for target 'all' failed
% make[1]: Leaving directory '/usr/lib/swipl/pack/hdt/hdt-cpp/hdt-lib'
% Makefile:26: recipe for target 'hdt-cpp/hdt-lib/libhdt.a' failed
% make: *** [hdt-cpp/hdt-lib/libhdt.a] Error 2
ERROR: -g pack_rebuild(hdt): Process "process(path(make),[all])": exit status: 2