sayedalesawy/videra-storage

The Storage Module Implementation for the Videra Video Indexer Engine.

Videra-Storage

The Storage Module Implementation for the Videra Video Indexer Engine.

Frontend Contract

Tags endpoint

GET: /tags

[
  "tag1",
  "tag2",
  "tag3",
  ...
]

Search endpoint

GET /search?tag=tag1&start=1&end=6

Note: start and end are optional params

[
  {
    "name": "video1",
    "token": "token1",
    "thumbnail": "link/to/thumbnail1",
  },
  {
    "name": "video2",
    "token": "token2",
    "thumbnail": "link/to/thumbnail2",
  },
  ...
]

Stream endpoint

GET /stream?token=token1&tag=tag1&start=1&end=6

Notes:

• tag same as the one sent in /search.
• start and end are optional params (same as the ones sent in /search).
• token is the video token sent in the /search response.

{
  "src_link": "link/to/stream/src", //To be passed to the hls.js library
  "clips": [
    {
      "start": 1, //clip start
      "end": 10 //clip end
    },
    {
      "start": 15, //clip start
      "end": 20 //clip end
    },
    ...
  ]
}

videra-storage's Issues

Connection loss between name node and data node after some time

Description

• After some time, the connection between name node and data node is lost. The name node can't send any more health check messages to data node, so after it misses some messages (which it didn't receive) it considers it as dead.
• This bugs happens when a data node is up for some minutes (usually 15, 20, 30) and the problem persists even if data node was restarted.
• It gets fixed after name node is restarted.

Expected

The connection shouldn't be lost as long as both nodes are up.

Actual

Connection is lost after some time (usually 15, 20, 30).

Load concentrated on new nodes

Description

• When a new nodes is connected (or reconnected) to the name node, it will redirect all requests to it.
• This happens because we use number of requests redirected to the node, without updating when a node finished work, so the new node will have zero requests, so the name node will redirect to it until it reaches the same number of requests as others.
• It can be fixed by using "last time redirect", to maintain when was last time a request was redirected to this node, and pick the one with earliest redirect time.

Expected

Number of requests is uniformly distributed over data nodes.

Actual

New nodes take all the new requests for some time.