opensemanticsearch/open-semantic-entity-search-api

Open Source REST API for named entity extraction, named entity linking, named entity disambiguation, recommendation & reconciliation of entities like persons, organizations and places for (semi)automatic semantic tagging & analysis of documents by linked data knowledge graph like SKOS thesaurus, RDF ontology, database(s) or list(s) of names

open-semantic-entity-search-api's Issues

Add options/parameter for stemming to Entity Manager

Automated tests

Code quality: Add automated tests by Python unittest.

Score by knowledge graph / connected entities

Score by count of links to other entities connected in linked data knowledge graph which occur in the document, too.

Geonames importer

Command line tool and UI for easier import of custom geonames for end users.

Synonyms to bidirectional

Synonym graph config to bidirectional, so search with alias(es) will find preferred label, too.

Import entities from CSV files

Add documentation how to import entities from CSV files

Language parameter

Optional parameter / autodetection of (con)text language to limit Entity Extraction by language for usage of very general multilingual thesaurus with many false friends.

Support same IDs in multiple thesauri

If same ID/URI in different thesaurus, only last import for the URI/ID is used.

Merge entities index entries from all thesauri, if they describe same URI by different/additional labels.

Generate OCR dictionary

Move code for generation of OCR dictionary from Open Semantic Search Apps to Entity API module and provide the word list via REST API.

Running in docker

Hallo

I want to use docker to run the search engine on windows. I did run docker pull opensemanticsearch/solr but do not know how to do "Call shell script build-deb" to build the dependencies.

When run docker run -p 8983:8983 opensemanticsearch/solr i can access teh service thought "http://localhost:8983/" but I expected a search interface but see the solr Admin console.

How do run semantic search using docker instead of using virtual box?

How do you use API call with python

I have attempted to understand how to use solr's api calls but have not managed to due to I am not able to find the collection name that open semantic seach uses in solr

User interface for recommendation / disambiguation of named entities in document(s)

UI for that will be available as recommender user interface for disambiguation in (semi)automatic tagging in Django based Open Semantic Search Apps.

Optimize performance: Merge index segments after bulk entity import

Call optimize on Solr entities core after bulk import of entities.

Migrate entity extraction from KeepWordFilterFactory to Solr Tagger

With the new https://lucene.apache.org/solr/guide/7_4/the-tagger-handler.html we can integrate entity extraction with stemming at ETL time and need not to temporary index the document and to manage an additional dictionary file anymore with need of core reloads, since all in Solr index and updateable by Solr API in near realtime.

Docker container

Docker container for easier installation, separation & distribution.

Score by named entity recognition class

Score by named entity recognition class in compare with RDF class of the named entity.

Import named entities from plain text list

Outsource the plain text list of entities importer from Open Semantic Search Apps to generic Open Semantic Entity Search API module & command line tool.

Import from Triplestore by SPARQL

Import entities by SPARQL query.

Evaluate Python libraries and open API standards

For scoring by named entity recognition class use Spacy, which is integrated with Open Semantic ETL.
For scoring by "more like this" and different scoring by fields like name or description use Solr/Elasticsearch index / API

For other scoring methods in issues evaluate existing Python libraries.

For API standards/parameters inspiration from similar Open Source software:

• NERD named entity recognition and disambiguation: http://nerd.eurecom.fr
• Open Refine Reconciliation Service API: https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation-Service-API
• Nordlys: https://github.com/iai-group/nordlys
• Spacy: Named Entity Recognition API by spacy-services: https://github.com/explosion/spacy-services
• Europeana Entity API: https://pro.europeana.eu/resources/apis/entity#suggest
• Neonion Semantic Annotations recommends entities by wikidata: https://github.com/FUB-HCC/neonion
• Apache Stanbol disambiguation by Solr "more like this": https://stanbol.apache.org/docs/trunk/components/enhancer/engines/list.html

Check Solrs new TaggerRequestHandler

Check if the new TaggerRequestHandler (AKA SolrTextTagger) for tagging text in Solr 7.4 https://lucene.apache.org/solr/guide/7_4/the-tagger-handler.html can be used for dictionary extraction without have to add/index the text temproary to the Solr core like done now and using Solr index for labels to be extracted instead of plain text lists for a filter.

Add entity type to index and results

Add Type / Facet / Dictionary / RDF class to results

Import named entities from database

Import named entities from database so we can do normalization by additional data base fields or links, not only like now export lists of names to plain text lists or by converting tables to SKOS manually.

Locations: Score by distance to other locations

Score locations by distance to other locations in same document, which mostly are the near to the other.

Preconfigured Solr schema/core

Add preconfigured Solr 7 core/schema to ./solr

Dictionary based entity extraction

Extract entities in dictionary

Score by edit distance / levensthein distance

Score by similarity of name by levensthein distance.

Stemming for entity extraction

(Optional) integration of stemming for dictionary/ontology/thesaurus based entity extraction

Import entities from SQL database

Add documentation how to import named entities from SQL databases

Normalization to SKOS preferredLabel

Return SKOS preferredLabel (fallback to label, fallback to alternate label, fallback to ID/URI) for normalization for faceted search.

Automatic setup of not yet existent dictionaries

Automatic setup of not yet existent dictionaries by adding entities to dictionaries so we do not have to setup new dictionaries by Open Semantic Search Apps anymore and API users have not to add dictionaries before use manually by dictionary manager.

Score by classification by machine learning of linked/tagged documents

Build machine learning model for scores/probabilities based on classification by machine learning from documents, which are tagged by human editors by this entities by Open Semantic Tagger UI or other annotations (f.e. in Drupal).

Switch aliases/synonyms to case insensitive by default

Switch generated Solr aliases/synonyms config to case insensitive and bidirectional by default

Debian / Ubuntu packages

Packages with all dependencies to run out of the box.

Option for Entity Linking not to be case sensitive

For example john doe should be linked to ID/URI of John Doe, too.

REST-API

Russian stemming

Add Russian grammar config.

Import entities from Excel spreadsheet

REST-API: JSONP

Support of JSONP, so the REST-API can be used by Open Refine.

Import named entities from RDF ontologies and SKOS thesaurus

Documentation for import/index named entities from RDF ontologies and SKOS thesauri and lists of names.

Move aggragation of all different labels from Python lib to Solr schema

Move aggragation of all different labels for FST index field from Python lib to Solr schema, so entities can be added directly by Solr APIs like HTTP-REST-API, CSV importer or from SQL dbs without preprocessing by Entity Manager Python library.

Names in scanned legacy documents

Names of Entities in old/legacy documents are often written with whitespaces like
John D O E
which should be recognized as John Doe, too.

Module entity manager

Python module for easy adding of single entities to index for easy custom entity importers and to move code from Open Semantic Search Apps UIs to API.

Increase performance of bulk entities import

Solr commit after import of last entity or by autocommit with latency, not on import of every entity.

Score by context

Enrich entities in index with context like description or wikipedia page and score the posted (con)text by matching such additional informations