bjarnigithub/tokenizerapi

Tokenizer API

This is an API for Tokenizer using the ELG specification. The API is wrapped in a docker container and is implemented using fastapi.

Getting started

Running ./run.sh will build and run the docker container

API calls

All the API calls use post and input/outputs are in a json format, following the specification from elg.

HTTP METHOD	Description
/tokenizer	Takes in text and splits them up into tokens

Testing

In the test folder, basic scripts for testing the functionality of the API can be found .

Testing ELG

For testing if the ELG specifications are being met, you must specify which API you want to test in the .env file. Then you run docker-compose up and submit the API calls to localhost:8080/process/service.

Funding

This ELG API was developed by the Language and Voice Lab at Reykjavík University in EU's CEF project: Microservices at your service.

European Language Grid

The docker image is hosted here and is running on the European Language grid as ELG API for Icelandic Tokenizer.

Underlying tool

The underlying tokenizer is Tokenizer by Miðeind, which is licensed under this MIT license. The ELG API implementation imports the corresponding PyPi package.