yahoojapan/ngtd

Serving NGT over HTTP or gRPC ※This project is not maintained. We have moved to a new product, [Vald](https://vald.vdaas.org) .

[Attention] This project is not maintained.

We have moved to a new product called Vald

NGTD: Serving NGT over HTTP or gRPC.

Description

NGTD provides serving function for NGT.

NGTD supports gRPC and HTTP protocol, so you can implement applications with your favorite programming language.

You can set any labels for each vectors, and enable to search with the label.

Install

You must install NGT before installing ngtd.

The easiest way to install is the following command:

$ go get -u github.com/yahoojapan/ngtd/cmd/ngtd

Docker

You can get ngtd docker image with the following command.

$ docker pull yahoojapan/ngtd

The image include only NGTD single binary, you enable to run the docker container.

Usage

$ ngtd --help
NAME:
   ngtd - NGT Daemonize

USAGE:
   ngtd [global options] command [command options] [arguments...]

VERSION:
   0.0.1-first

COMMANDS:
     http, H   serve ngtd index by http
     grpc, g   serve ngtd index by grpc
     build, b  build ngtd index
     help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Serve

ngtd supports http and gRPC protocol.

HTTP

$ ngtd http --help
NAME:
   ngtd http - serve ngtd index by http

USAGE:
   ngtd http [command options] [arguments...]

OPTIONS:
   --index value, -i value                 path to index (default: "/usr/share/ngtd/index")
   --dimension value, -d value             vector dimension size.(Must set if create new index) (default: -1)
   --database-type value, -t value         ngtd inner kvs type(redis, golevel, bolt or sqlite)
   --database-path value, -p value         ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
   --redis-host value                      redis running host (default: "localhost")
   --redis-port value                      redis running port (default: "6379")
   --redis-password value                  redis password
   --redis-database-index value, -I value  list up 2 redis database indexes (default: 0, 1)
   --port value, -P value                  listening port (default: 8200)

Request

$ curl -H 'Content-Type: application/json' -X POST http://localhost:8200/search -d '{"vector":[...], "size": 10, "epsilon": 0.01}'
$ curl -H 'Content-Type: application/json' -X POST http://localhost:8200/searchbyid -d '{"id":"<id>", "size": 10, "epsilon": 0.01}'

If you want more information, please read model.go

gRPC

$ ngtd grpc --help
NAME:
   ngtd grpc - serve ngtd index by grpc

USAGE:
   ngtd grpc [command options] [arguments...]

OPTIONS:
   --index value, -i value                 path to index (default: "/usr/share/ngtd/index")
   --dimension value, -d value             vector dimension size.(Must set if create new index) (default: -1)
   --database-type value, -t value         ngtd inner kvs type(redis, golevel, bolt or sqlite)
   --database-path value, -p value         ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
   --redis-host value                      redis running host (default: "localhost")
   --redis-port value                      redis running port (default: "6379")
   --redis-password value                  redis password
   --redis-database-index value, -I value  list up 2 redis database indexes (default: 0, 1)
   --port value, -P value                  listening port (default: 8200)

Client

If you use language except golang, compile proto file for the language.

Go examples are in example/.

Build

Build will construct the NGT database by importing the Vector data, please check the Input Format section at the bottom of the Build chapter for the file format to import.

$ ngtd build --help
NAME:
   ngtd build - build ngtd index

USAGE:
   ngtd build [command options] [arguments...]

OPTIONS:
   --index value, -i value                 path to index (default: "/usr/share/ngtd/index")
   --dimension value, -d value             vector dimension size.(Must set if create new index) (default: -1)
   --database-type value, -t value         ngtd inner kvs type(redis, golevel, bolt or sqlite)
   --database-path value, -p value         ngtd inner kvs path(for golevel, bolt and sqlite) (default: "/usr/share/ngtd/db/kvs.db")
   --redis-host value                      redis running host (default: "localhost")
   --redis-port value                      redis running port (default: "6379")
   --redis-password value                  redis password
   --redis-database-index value, -I value  list up 2 redis database indexes (default: 0, 1)
   --text-delimiter value, -D value        delimiter for text input (default: "\t", " ")
   --pool value                            number of CPU using NGT indexing (default: 8)
   --parallel-parse value                  number of CPU using input parser (default: 8)

Input format

Now, we support only text format.

It enables for id to use any character without delimiter1 and for vector elements to use decimal/hex format.

If you choice hex format, you must begin the value with "0x".

<id1><delimiter1><v11><delimiter2><v12><delimiter2>...<delimiter2><v1d>\n
<id2><delimiter1><v21><delimiter2><v22><delimiter2>...<delimiter2><v2d>\n
...
<id1><delimiter1><vn1><delimiter2><vn2><delimiter2>...<delimiter2><vnd>\n

License

Copyright (C) 2018 Yahoo Japan Corporation

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contributor License Agreement

This project requires contributors to agree to a Contributor License Agreement (CLA).

Note that only for contributions to the ngtd repository on the GitHub (https://github.com/yahoojapan/ngtd), the contributors of them shall be deemed to have agreed to the CLA without individual written agreements.

Authors

Kosuke Morimoto
kpango

ngtd's Issues

keys not found

Hi,

I created an NGT server using docker and inserted many (~1.5million) vectors into it while logging their IDs to a separate file. When all of the insertions had complete, I uploaded the server files to AWS and downloaded them from a different device.

Now, when I try to look up my vectors by the IDs from the separate file, NGT is unable to locate any of them. Is there any sort of encoding/decoding issue I should be aware of when launching my server on the new machine?

I can see familiar IDs in the index file... I just am unable to retrieve them via GRPC calls.

Any help would be much appreciated. Thank you!

Benchmark Code

Can we access pprof api?

pprof is imported in main.go L22.
So I want to access pprof api via http request.
But I cannot access it now.

In my knowledge we need call http.ListenAndServe, such that

 func main() {
+	go func() {
+		log.Println(http.ListenAndServe("0.0.0.0:PORT", nil))
+	}()

Is it right ? Or ngtd already expose pprof http api ?

Do more test

404 Not Found flow for searchbyid and getobjects.

I want to know which id cause db nil error from log.
And I think it is better that db nil error returns 404 Not Found.

pprof option makes some error

After merged #23 , I tried to run it.
But I can not start ngtd when pass --pprof option.

I fixed comma-less Name problem

cli.BoolFlag{
	Name:  "pprof, pp", // HERE
	Usage: "enable pprof server",
},

but ngtd does not start.

This problem happens ONLY when with pprof option, so it is not critical now.

How to generate ngtd.pb.go?

I red https://github.com/gogo/protobuf and I knew protoc-gen-gogo has many options.
And I can't generate same ngtd.pb.go on master branch.

Please let me know the way to generate ngtd.pb.go.