rtradeltd/lens

🔍 Lens is an opt-in search engine and data collection tool to aid content discovery of the distributed web

🔍 Lens

Search engine for the distributed web

Lens is an opt-in search engine and data collection tool to aid content discovery of the distributed web. It exposes a simple, minimal API for intelligently indexing and querying content on IPFS.

Multi-Language

Features and Usage

Initially integrated with Temporal, Lens will allow users to optionally have the data they upload be searched and indexed and be awarded with RTC for participating in the data collection process. Users can then search for content using a simple-to-use API.

Searching through Lens will be facilitated through Temporal web. Optionally, we will have a service independent from Temporal which users can submit content to have it be indexed. This however, is not compensated with RTC. In order to receive the RTC, you must participate through Lens indexing within the Temporal web interface.

API

Lens exposes a simple API via gRPC. The definitions are in RTradeLtd/grpc.

The Lens API, summarized, currently consists of three core RPCs:

service LensV2 {
  rpc Index(IndexReq)   returns (IndexResp)  {}
  rpc Search(SearchReq) returns (SearchResp) {}
  rpc Remove(RemoveReq) returns (RemoveResp) {}
}

Golang bindings for the Lens API can be found in RTradeLtd/grpc.

Supported Formats

Only IPFS CIDs are supported, and must be either images, text files, or pdfs. We attempt to determine the content type via mime type sniffing, and use that to determine whether or not we can analyze the content.

Please see the following table for supported content types that we can index. Note if the type is listed as <type>/* it means that any "sub type" of that mime type is supported.

Mime Type	Support Level	Tested Types
text/*	Beta	text/plain, text/html
image/*	Beta	image/jpeg
application/pdf	Beta	application/pdf

Deployment

The recommended way to deploy a Lens instance is via the rtradetech/lens Docker image.

$> docker pull rtradetech/lens:latest

A docker-compose configuration is available that also starts up other prerequisites:

$> wget -O lens.yml https://raw.githubusercontent.com/RTradeLtd/Lens/master/lens.yml
$> LENS=latest BASE=/my/dir docker-compose -f lens.yml up

Development

This project requires:

• Go 1.11+
• dep
• Tesseract
• Tensorflow
• go-fitz

To fetch the codebase, use go get:

$> go get github.com/RTradeLtd/Lens

A rudimentary Makefile target make dep is available for installing the required dependencies.

lens's Issues

Look Into Bayesian Classification

Another possible method of filtering data would be using bayesian classification such as https://github.com/lytics/multibayes.

To complete this issue research should be done to see if this provides any notable benefits, and if so provide a small prototype of the functionality.

Existing security issues

⚠️ Here are the existing security issues we detected on your default branch:

Insecure File Management (4)

• /analyzer/images/tensorflow.go:49
• /analyzer/images/tensorflow.go:103
• /mocks/stubs.go:14
• /analyzer/images/tensorflow.go:246

More info on how to fix Insecure File Management in Go.

GPU Enable Docker Images

Docker images already support CPU based tensorflow processing, but do not support GPU processing.

potentially useful: https://hub.docker.com/r/nvidia/cuda/

how tensorflow does it themselves: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/docker/Dockerfile.devel-gpu

Can I try Lens online?

Hi,

Can I try using the Lens search engine online somewhere?

BM

Add Category Update to Store Lens::Store

Lens::Store does not update the category, which makes advanced search less efficient

Index Gnosis Research Papers

https://github.com/gnosis/dex-research

Integrate OCR for image processing

We should use OCR as a fallback for image processing. You can upload images of words which tensorflow will mis-categorize as its not meant for that. The fallback to this should be running the image through OCR.