shotit/shotit-worker

Four core workers of shotit: watcher, hasher, loader and searcher.

Shotit-worker

Backend workers for shotit. Four core workers of shotit: watcher, hasher, loader and searcher.

Features

• watch file system changes and upload hash to shotit-media
• download video from shotit-media, hash, and upload to shotit-api
• download hash from shotit-api, deduplicate, and upload to milvus

Prerequisites

• Node.js 16.x, 18.x
• ffmpeg 4.x
• java (openjdk 1.8.0)
• git
• pm2

Local Development Guide

Install Prerequisites first, then:

git clone https://github.com/shotit/shotit-worker.git
cd shotit-worker
yarn install

Configuration

• Copy .env.example to .env
• Edit .env as appropriate for your setup, as is for the first time.

Start workers

You can use pm2 to run this in background in cluster mode.

Use below commands to start / restart / stop server.

yarn start
yarn stop
yarn reload
yarn restart
yarn delete
yarn logs

To change the number of nodejs instances, edit ecosystem.config.json

shotit-worker's Issues

Low CPU usage when `shotit-worker-loader` loads data to the vector database

This JSON polishment operation is just single-threaded and slow.

Proposal:

Need to make use of js workers to speed up the index loading process.

Loader Bug: grpc error - Error: Channel has been shut down

Context:
Update milvus to the latest v2.2.11, and milvus-sdk-node to v2.2.18, then when running the shotit docker-compose at ubuntu, the loader worker is not working properly.
From sudo docker logs -f -n 100 shotit-worker-loader, got

Unzipping hash
Parsing xml
Polish JSON data
Pause for 5 seconds
/app/node_modules/@grpc/grpc-js/build/src/internal-channel.js:452
            throw new Error('Channel has been shut down');
            ^

Error: Channel has been shut down
    at InternalChannel.createCall (/app/node_modules/@grpc/grpc-js/build/src/internal-channel.js:452:19)
    at ChannelImplementation.createCall (/app/node_modules/@grpc/grpc-js/build/src/channel.js:64:37)
    at getCall (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:253:26)
    at getBottomInterceptingCall (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:376:18)
    at /app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:424:26
    at retry (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Grpc.js:157:39)
    at Timeout._onTimeout (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Grpc.js:136:45)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)

Node.js v18.16.1

repeatedly every time a new episode is getting loaded, and then shotit-worker-loader crashes and reboots.

Where the error message of Error: Channel has been shut down is coming from is grpc/grpc-node/

Feature Request: add duration information to shotit-loader

The ongoing Shotit paper places an impressive demo image on the first page. However, the image is not consistent with the demo image previously placed on shoitit GitHub profile and shotit/shotit

In the result section, there is no duration information currently at the production demo site, but it is proposed in the shotit paper to make the title Shotit: compute-efficient image-to-video search engine for the cloud more intuitive.

Please implement it in the shotit source code to make it consistent with the paper.