voxel51/player51

Player51 is a client-side JavaScript media player that can render metadata overlays on images and video

player51's Issues

Support for Video Attributes

Not blocking anything, but it would be nice to have soon.

Single frame looping support

This is a rather strange case, but when player51 is attempting to play a single frame in a loop (like in scoop clips) it does not, and simply plays the full video.

Data that causes this issue:

{
  "data": {
    "clips": {
      "matches": [{
...
      }, {
        "labels_uuid": "af7c7ecd-de33-4b6c-a8ba-4bdbf4c53bce",
        "video_url": "https://storage.googleapis.com/scoop-dev/bits%2Fc%2F3%2F1%2F4%2F0%2Fc3140020-b015-409f-87b8-9e1c4e6d2fee.mp4?Expires=1556295758&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=dxW5S37WrQZg%2FKUFUyPGLwgblr0NpKoA%2B%2B4yAz6%2BiKnCTBw21TDPWXp4bLlxZmIWXBQ%2BSWznXOTCYReGUV952DsbUSmTxX4PvKsPEDj23eCQaA7NziOfe9EFDNP%2B35IHWFYWbJOEDaIte%2B%2F%2BqNEr%2BoKKDpqBHfgEBIRNc9gwzxEsbACew8zVAzVyQCp2U7uh%2BJxJP5RrmocpLa1M0G%2BiGosBBFJRBvI8cZx1zju5qbe%2FyJz5JpDoiqRFKyxo3zSROSsVZSd0QjEUBq0XuO%2B6DO4%2BIhBLYLQWcd%2Fz0GGfsJW1XHCimBpXuAGV2eZpI59XX6s8OQhyCQiGN6wwRO8ZSw%3D%3D",
        "added_by_email": "[email protected]",
        "time_range": [5.376542048293088, 5.376542048293088],
        "added_by_last": "",
        "date_added": "2019-04-25T16:22:29",
        "frame_range": [162, 162],
        "sample_uuid": "63576b08-9a9b-4dc9-ad08-ede6cf988482",
        "extension": "mp4",
        "clip_length": 0.0,
        "added_by_first": "",
        "thumbnail_url": "https://storage.googleapis.com/scoop-dev/bits%2Fc%2F3%2F1%2F4%2F0%2Fc3140020-b015-409f-87b8-9e1c4e6d2fee-thumbnail.mp4?Expires=1556299221&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=2FRtMS2sxI3PX8zdvqAYb9kBK1ClyWNjw2IuLLjetEOsxyvB%2B3B1S6h9NNpO4izNtfPEtVpuYhIz%2FKmEeL5%2FzH%2BUT3jDZb81zqN06tWr31HLA2RizIWyenusuCtqd2cxkLU68pkVshKnM9wDkoQkSErEKI%2BpOUFC2hlJMN8BJ2VaZhW4YrDh1Hw%2FGGyrgb1r%2BOwt1Bc%2F%2BrtD8wTDthJlWqV214CooxxDhLTzXxwOXRr%2BqhI5mV1gEfMFpzsMcTkysqjiHT7gFtXrqGps3qfNnPlkrpnStk9NbPr9uivgbxNB1zN%2FbXVdQ%2FhDKSVbOVgHxkcGdk51LjMN9tlNKT1GmA%3D%3D",
        "video_metadata": {
          "frame_size": [1280, 720],
          "frame_rate": 29.97002997002997,
          "total_frame_count": 1202,
          "duration": 40.107,
          "size_bytes": 27065708,
          "encoding_str": "avc1"
        },
        "labels_url": "https://storage.googleapis.com/scoop-dev/bits%2Fa%2Ff%2F7%2Fc%2F7%2Faf7c7ecd-de33-4b6c-a8ba-4bdbf4c53bce.json?Expires=1556299221&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=MO%2BLI76%2BI7UQn%2Br%2BgFzc9sQL%2F3cGS3jNrLnRwcRgOtt4douerPfUlvRuGAsdymQKdyTrw0GFd7uRwW10dr74qEOwIqtt%2BMzswNxTNbNboN6hofpXZTP5C2YQkg6LTVkqPtgSs%2BnHdt35CZ4hdWktkWHC1Sr9VgDQRo%2BNQvto9oX2lYQ%2B1LwfJ%2FXZ9KPNGb0DXMQpt%2Fp%2B8KcfmZiYjrfGQ%2Fu0wGgtolC9%2Bwz7O1YY72exhCHyCP4T1yLeef2mKCw7te4LKPQGr%2F74sG7JG9wFE%2B%2BTt7uyjtQXn7lgbUL5lJYr1nrN8IAuojT3t%2FqPFM3Eu%2Bclv4BiEC40DrPtamAdPw%3D%3D",
        "video_uuid": "c3140020-b015-409f-87b8-9e1c4e6d2fee"
      }, {
      ...
      }]
    },
    }
  }
}

Where clip length is 0 and the frame_range is [162,162]

Add support for rendering video-level attributes

eta.core.video.VideoLabels now has video-level attributes, which are the same as frame-level attributes but are set once and apply to all frames of the video. We need to add support for rendering these attributes via Player51.

My proposal is that video-level attributes are rendered in the same box as the existing frame attributes. Always on top of the list, say.

This proposal aligns with how video-level attributes are rendered by ETA: https://github.com/voxel51/eta/blob/develop/eta/core/annotations.py

See the example JSON on the right-hand side of the docs below to see a minimal example of a VideoLabels with video-level, frame-level, and object-level attributes:

https://voxel51.com/docs/api/#types-videolabels

Fire event when everything is loaded and player is really ready

User request to have an event fired when all data is loaded from the server and video can really begin playing (with annotations).

Scrubber always resets to beginning of video

Attempting to seek to a position always resets to the beginning of the video.

Example with a roughly 2 second video (can't attach videos directly, unfortunately): p51-scrubber.mp4.gz

Need ability to play video in slow motion

Player51 cleanup

Now that I've had the chance to see and use the new Player51 in production for myself, I have some requests:

Invalid options

• we cannot currently allow users to select Show frame number when previewing data (as opposed to a job output) because we don't have access to frame information. So, this needs to be disabled for data preview

Usability

• We should only allow users to click on objects that have an index. Otherwise, the object will instantly disappear, which is never what one wants. Hovering on objects with no index is fine and good

Keyboard shortcuts

• didn't we use to have these? we should continue to support single frame navigation via left/right arrow keys, and play/pause toggle via the space bar

Look-and-feel

• text formatting is inconsistent/unclear. I see the following things:
  • (menu bar) Show Frame Number -> Show frame number
  • (menu bar) Select mode: -> Object selection mode (note: no colon)
  • (menu bar) Attribute rendering mode: -> Object attribute mode (note: no colon)
  • (menu bar) Attribute background -> Show attribute background
  • (menu bar) attr -> Attribute
  • (menu bar) attr-value -> Attribute: value
  • (menu bar) checkboxes/radio buttons need to be vertically centered. They are too close to the upper boundary of their divs currently
  • (menu bar) DISPLAY OPTIONS needs to be bolded, like to how such headings are bolded elsewhere on the Platform (e.g., dashboard cards)
  • (play bar) ideally the play button and the time/frame text next to it would have the same vertical spacing. Currently, the time/frame is lower than the play button
  • (play bar) need space before/after / when showing frame numbers. Like is done with timestamps
  • (play bar) use a monospace font for rendering frame numbers/timestamps in the play bar. The idea with the zero padding was to make it so that the numbers don't shift around as time progresses. In order to achieve that, we need a monospace font
• when the HUD loses focus, close the menu bar. This provides a convenient way to close the menu without having to select the gear icon. Like YouTube
• can we color the seek bar orange behind the current time indicator? this makes it visually much easier to see where we are in the video. Like YouTube

Race condition causing labels not to render sometimes

Splitting this off into a separate issue

Labels are not rendering for single frame media fragments. These asynchronous calls I've linked below are likely part of the problem. I will look into this more when time permits. Labels are being flagged as processed after issuing the asynchronous functions. There is no guarantee that the labels are actually loaded with the current approach.

player51/src/js/renderer.js

Line 317 in 05fc2a1

waitUntil(() => (typeof(this.canvasWidth) != 'undefined' &&

player51/src/js/renderer.js

Line 349 in 05fc2a1

waitUntil(() => (typeof(this.canvasWidth) != 'undefined' &&

The following labels will not be rendered when given the clip range [0, 0].

{
    "filename": "sample.mp4",
    "frames": {
        "1": {
            "frame_number": 1,
            "objects": {
                "objects": [
                    {
                        "label": "vehicle",
                        "bounding_box": {
                            "top_left": {
                                "x": 0.5090277777777777,
                                "y": 0.49027777777777776
                            },
                            "bottom_right": {
                                "x": 0.5506944444444445,
                                "y": 0.5666666666666667
                            }
                        },
                        "index": 1,
                        "frame_number": 1,
                        "attrs": {
                            "attrs": [
                                {
                                    "type": "eta.core.data.CategoricalAttribute",
                                    "name": "color",
                                    "value": "silver",
                                    "confidence": 1
                                }
                            ]
                        }
                    }
                ]
            }
        }
    }
}

Originally posted by @benjaminpkane in #41 (comment)

Questions about the sample data (broken links)

Hello,

I am working on a project which attemps to perform client side rendering of frame meta data (generated by a deep neural network) on the original video footage, without having to re-encode the labels on the video. I came across this repository, and it seems to have something that I have been looking for. I wanted to test your sample applications first, but came across some issues accessing the sample data.

I understand that this repository has been deprecated and that things have been migrated over to Looker, but I am more interested in the contents of this repository since I have very little idea how I would integrate Looker to my custom app (since they seem to be very tightly coupled with Fiftyone platform). Unfortunately, I do not have access to your sample data as the link seemed to have expired. I tried your code for download_data.py, but it seems to have the same issue.

There are no documentation in terms of how to format the data exactly, so i was wondering if you could share this data if you still have access to them.

Thank you very much,
Philip

Support for boolean attributes

Adding support for boolean attributes would be of great use to team scoop and the new ontology.

Mask index support

If a mask index is specified, 0 could be a valid mask - draw everything in the mask index

Responsive Sizing

Player51 currently implements a fixed size. This should be extended to allow for resizing and reaction to browser size changes.

Only show labels of interest when user is querying/filtering while Exploring

Want to only be able to see what is being queried/filtered. In the following case, all vehicles should not have bounding boxes around it, only the stop sign:

This is a simple case but if a user wants to only see red, ford, sedans then only those labeled detections should be shown in the clip.

React indicates outdated syntax in minified files

Compiled with warnings.

../player51/build/css/player51.min.css (./node_modules/css-loader??ref--6-oneOf-3-1!./node_modules/postcss-loader/src??postcss!../player51/build/css/player51.min.css)
Warning

(1:305) Gradient has outdated direction syntax. New syntax is like `to left` instead of `right`.

Preview error when output has no labels

I ran vehicle-sense v0.3 on a video for which vehicle-sense is expected to produce an empty labels output. (Job number c8489f13-ccf7-4f28-aaf4-4dd4473bad16) Indeed, when I download the output json, it correctly contains an empty dictionary {}. However, when I try to preview the video output, I sometimes get the error message:

"Cannot read property 'objects' of null"

However, sometimes the preview displays correctly (with no annotations on the video as expected).

Render floating point numeric attributes with two decimal places of accuracy

Showing many significant digits for floating point numeric attributes is not useful and takes up a lot of visual space.

This is how ETA renders numeric attributes, which I request we adopt in Player51:

https://github.com/voxel51/eta/blob/ba108043fa9afb324915a7c2419c12e6a1768bed/eta/core/annotations.py#L1188

Need additional spacing to the left of object attributes

We need a bit of space to the left of object attributes, both in list,of,attrs mode and attr:value mode.

The goal is to have equal top/left spacing within the object attribute panel, similar to what we have with the frame attribute panel.

Examples below:

Options menu closes when "Show attributes" is toggled

The options menu automatically closes when the Show attributes checkbox is clicked. I'm guessing this is because this particular checkbox changes the dimensions of the menu.

The goal is that the options menu never closes in response to a click within the menu, regardless of how the size of the menu changes in response to the click.

Need ability to hide uninteresting labels in player51 modal

A common user workflow is to want to inspect a single object in a video. The ask is that clicking on a detection in the player modal will toggle between showing all detections and only showing that particular object (and its attributes).

This is not high priority.

Ability to hide the frame/video attribute box

Frame/video attributes quickly fill up the screen hide the video and boxes.

Can't view zip files that don't have a zip extension

This is an issue identified by Brian at https://www.pivotaltracker.com/n/projects/2184985/stories/165914600
The platform console can preview zips locally, but can't in dev/staging/production because the signed URLs we're using don't have extensions.

player51/src/js/renderer.js

Lines 498 to 501 in f8ff0f7

	if (!this.checkMediaFormat(zipPath)) {
	/* eslint-disable-next-line no-console */
	console.log('WARN: media is not a zip file.');
	return;

is the problematic check, which can't even be handled by code that uses player51 since it's just a log message. Since the mime type is also available when this check takes place, the check should just be for the mime type, if anything. (Working on a fix now but want this to be tracked.)

Player controls are (sometimes?) invisible if player is created under mouse cursor

For example, when previewing a job's output on the platform console, on certain window sizes, the player will be created above the "preview" option (although this issue can also be reproduced by moving the mouse quickly too). In some cases, the video controls will be invisible, but clicking on where the play/pause button would be still plays/pauses the video, and the cursor changes to a pointer when hovering over that area.
This issue only seems to occur sometimes, and may depend on the load order of certain things (maybe the video and labels).

Arrow key frame navigation sometimes shows blank frame at end of video

At the end of the attached video, pressing the right arrow key alternates between displaying the overlay and displaying nothing. May be related to the fractional FPS.
cars-2.3s-zipped.zip

Issue when trying to render bounding boxes on frames that do not start from beginning of video

This is very strange but I think I'm able to rule this out as a console issue.
It seems that this video has frame objects that start at 122 or so. The json is loading correctly (which is why I think I can rule out a console issue), and the video, when loaded, buffers to about 3/4 of the way through the video (I'm assuming this is so that it starts at the first detected object?). The first frame of the video has no objects, though - and this may be the culprit. @kevinqi34 @MikeJeffers do you guys have any insight into what's going on here?

MediaFragments is not supported when launching player51 using REACT.

There is a problem with MediaFragments.js when trying to follow the README, and launching player51 through REACT.

Does not compile when running NPM start.

Add a way to detect unhandled media types

Currently this situation is only handled by logging a message to the console, as far as I can tell:

player51/src/js/player51.js

Line 122 in fa6965a

console.log('WARN: Player51 doesn\'t support this media type yet.');

On the console, this results in a loading page being shown indefinitely to the user:

Either an event that gets triggered in this case or some other way to render a fallback message would be better.

Clip looping fails when clip extends to end of video

I performed a search in Scoop that matched the last 3 seconds of a 40 second video. However, when I view the clip, it does not loop 00:37 - 00:40 as expected, it loops back to the beginning of the video.

Clip looping fails when clip is too short

This search in Scoop returns a match with a time of 0s. Maybe the match is only 1 frame or something like that. In any case, the clip looping does not work and just lets the entire video play beyond the match.

For example, I searched for Black Panther and clicked on the matching clip. But there is no Black Panther in this frame, which should never happen:

Cache Player51 options state

Currently I have to reconfigure my Player51 options each time I preview a video.

A more typical experience would be to cache my settings so that I get the same behavior each time I preview an output

Support for passing in overlay as an object

Currently Player51 takes a URL pointing to the overlay JSON and then tries to download it. We want to continue to support this if desired, but we also want to support the user passing in the overlay directly as an object (pre-downloaded, if you like).

/**
 * Player51 Class Definition
 *
 * INHERITS:  None
 * F-MIXINS:  None
 * @constructor
 * @param media is an object that has 'src' and 'type' attributes.
 * @param overlay can be either an object describing the overlay or a URL
 *   from which to download the overlay JSON information
 * @param fps is the frame-rate of the media.  If it is not provided then it
 * will be guessed.
 */
function Player51(media, overlay, fps) {
  // if overlay is an object, use it
  // if overlay is a string, download the JSON and parse it
}