akshatsh/videosearchengine

Semantically be able to search through a database of videos (using generated summaries)

videosearchengine's Issues

Group Video Frames into semantically Different Components

Frame Grouping

Given a video, break apart sequences of frames into different components, and parallelize/distribute the work.

Need to implement

• video_utils.py

• VideoDistributer.py

• any other necessary files

• Basic Version

• More complex things

Create a database for Videos

Store video name and summary in a no SQL store

Probably following an API with

 { 
    "name" : "the name of the video",
    "summary": "the summary of the video",
    "url" : "video_url"
}

Language Generation for a single Frame

Generate an image caption for each frame

Obj2Text is probably helpful

Be able to upload videos to the database through the UI

Create Poster

Adding more information about how to train and evaluate the captioning network

Hi,
Thanks a lot for this amazing work. Can you please add more details on the readme file about the training and evaluation steps of your network with and without YOLO ?
The current readme file lacks of information on how to do that.
It will be very useful if you can provide the trained model in order to test it using some raw videos.
many thanks again

Filter Noisy Frames

GIven a set of object detections, drop the noisy frames to help language generation

• Look into descrimintory networks

Extract Useful Frames

Given a set of frames, select a subset that are meaningful and have no repititions.

Basic Heuristic: select every 10th frame

Use TaCos Dataset

This dataset is incredibly large, and incredibly helpful (30GB in size). We want to use this, but attu has a file limit and this is way too big for it. We need some solution to be able to train on this end to end.

• Sent email to TA asking for help

Educational credits for different cloud services (AWS, Google Could Compute, Azure)

We have access to one of the research servers for running experiments.

Implement Bounding Box object detection for a frame

• Possible extend to multiple frames at once

Detect objects in each frame using YOLO

Add tensor board connection

Be able to monitor status of models as they train using tensor board

Given a video, be able to estimate time to generate a summary

[Errno 61] Connection refused on mac

Connection is being refused when all the proper permissions are granted.

Error:
Sending cluster to worker: localhost:24448
[Errno 61] Connection refused

---

24449 1

---

Sending cluster to worker: localhost:24449
[Errno 61] Connection refused

Update README for contributors

On the README update it to include a link to something about you (Github, Linkedin, etc.)

• Akshat Shrivastava
• Kevin Bi
• Sarah Yu
• Abby Gray

Write Report

Create Search Engine UI

Some Google like UI for the video database

Combine generated summaries of frames to one large summary

The current system, can/will split up work to different workers if the videos are too large and generate summaries.

Each worker would generate a summary for its portion.

When all finished, each worker would return a summary for its portion, all the summaries need to be blended together for one coherent summary of the video.

Implement Page Rank for summaries

A basic version would be what we did for 333