voxel51 / player51 Goto Github PK
View Code? Open in Web Editor NEWPlayer51 is a client-side JavaScript media player that can render metadata overlays on images and video
License: Apache License 2.0
Player51 is a client-side JavaScript media player that can render metadata overlays on images and video
License: Apache License 2.0
Not blocking anything, but it would be nice to have soon.
This is a rather strange case, but when player51 is attempting to play a single frame in a loop (like in scoop clips) it does not, and simply plays the full video.
Data that causes this issue:
{
"data": {
"clips": {
"matches": [{
...
}, {
"labels_uuid": "af7c7ecd-de33-4b6c-a8ba-4bdbf4c53bce",
"video_url": "https://storage.googleapis.com/scoop-dev/bits%2Fc%2F3%2F1%2F4%2F0%2Fc3140020-b015-409f-87b8-9e1c4e6d2fee.mp4?Expires=1556295758&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=dxW5S37WrQZg%2FKUFUyPGLwgblr0NpKoA%2B%2B4yAz6%2BiKnCTBw21TDPWXp4bLlxZmIWXBQ%2BSWznXOTCYReGUV952DsbUSmTxX4PvKsPEDj23eCQaA7NziOfe9EFDNP%2B35IHWFYWbJOEDaIte%2B%2F%2BqNEr%2BoKKDpqBHfgEBIRNc9gwzxEsbACew8zVAzVyQCp2U7uh%2BJxJP5RrmocpLa1M0G%2BiGosBBFJRBvI8cZx1zju5qbe%2FyJz5JpDoiqRFKyxo3zSROSsVZSd0QjEUBq0XuO%2B6DO4%2BIhBLYLQWcd%2Fz0GGfsJW1XHCimBpXuAGV2eZpI59XX6s8OQhyCQiGN6wwRO8ZSw%3D%3D",
"added_by_email": "[email protected]",
"time_range": [5.376542048293088, 5.376542048293088],
"added_by_last": "",
"date_added": "2019-04-25T16:22:29",
"frame_range": [162, 162],
"sample_uuid": "63576b08-9a9b-4dc9-ad08-ede6cf988482",
"extension": "mp4",
"clip_length": 0.0,
"added_by_first": "",
"thumbnail_url": "https://storage.googleapis.com/scoop-dev/bits%2Fc%2F3%2F1%2F4%2F0%2Fc3140020-b015-409f-87b8-9e1c4e6d2fee-thumbnail.mp4?Expires=1556299221&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=2FRtMS2sxI3PX8zdvqAYb9kBK1ClyWNjw2IuLLjetEOsxyvB%2B3B1S6h9NNpO4izNtfPEtVpuYhIz%2FKmEeL5%2FzH%2BUT3jDZb81zqN06tWr31HLA2RizIWyenusuCtqd2cxkLU68pkVshKnM9wDkoQkSErEKI%2BpOUFC2hlJMN8BJ2VaZhW4YrDh1Hw%2FGGyrgb1r%2BOwt1Bc%2F%2BrtD8wTDthJlWqV214CooxxDhLTzXxwOXRr%2BqhI5mV1gEfMFpzsMcTkysqjiHT7gFtXrqGps3qfNnPlkrpnStk9NbPr9uivgbxNB1zN%2FbXVdQ%2FhDKSVbOVgHxkcGdk51LjMN9tlNKT1GmA%3D%3D",
"video_metadata": {
"frame_size": [1280, 720],
"frame_rate": 29.97002997002997,
"total_frame_count": 1202,
"duration": 40.107,
"size_bytes": 27065708,
"encoding_str": "avc1"
},
"labels_url": "https://storage.googleapis.com/scoop-dev/bits%2Fa%2Ff%2F7%2Fc%2F7%2Faf7c7ecd-de33-4b6c-a8ba-4bdbf4c53bce.json?Expires=1556299221&GoogleAccessId=scoop-dev-storage-admin%40scoop-234920.iam.gserviceaccount.com&Signature=MO%2BLI76%2BI7UQn%2Br%2BgFzc9sQL%2F3cGS3jNrLnRwcRgOtt4douerPfUlvRuGAsdymQKdyTrw0GFd7uRwW10dr74qEOwIqtt%2BMzswNxTNbNboN6hofpXZTP5C2YQkg6LTVkqPtgSs%2BnHdt35CZ4hdWktkWHC1Sr9VgDQRo%2BNQvto9oX2lYQ%2B1LwfJ%2FXZ9KPNGb0DXMQpt%2Fp%2B8KcfmZiYjrfGQ%2Fu0wGgtolC9%2Bwz7O1YY72exhCHyCP4T1yLeef2mKCw7te4LKPQGr%2F74sG7JG9wFE%2B%2BTt7uyjtQXn7lgbUL5lJYr1nrN8IAuojT3t%2FqPFM3Eu%2Bclv4BiEC40DrPtamAdPw%3D%3D",
"video_uuid": "c3140020-b015-409f-87b8-9e1c4e6d2fee"
}, {
...
}]
},
}
}
}
Where clip length is 0 and the frame_range is [162,162]
eta.core.video.VideoLabels
now has video-level attributes, which are the same as frame-level attributes but are set once and apply to all frames of the video. We need to add support for rendering these attributes via Player51.
My proposal is that video-level attributes are rendered in the same box as the existing frame attributes. Always on top of the list, say.
This proposal aligns with how video-level attributes are rendered by ETA: https://github.com/voxel51/eta/blob/develop/eta/core/annotations.py
See the example JSON on the right-hand side of the docs below to see a minimal example of a VideoLabels
with video-level, frame-level, and object-level attributes:
User request to have an event fired when all data is loaded from the server and video can really begin playing (with annotations).
Attempting to seek to a position always resets to the beginning of the video.
Example with a roughly 2 second video (can't attach videos directly, unfortunately): p51-scrubber.mp4.gz
Now that I've had the chance to see and use the new Player51 in production for myself, I have some requests:
Invalid options
Show frame number
when previewing data (as opposed to a job output) because we don't have access to frame information. So, this needs to be disabled for data previewUsability
index
. Otherwise, the object will instantly disappear, which is never what one wants. Hovering on objects with no index
is fine and goodKeyboard shortcuts
Look-and-feel
Show Frame Number
-> Show frame number
Select mode:
-> Object selection mode
(note: no colon)Attribute rendering mode:
-> Object attribute mode
(note: no colon)Attribute background
-> Show attribute background
attr
-> Attribute
attr-value
-> Attribute: value
DISPLAY OPTIONS
needs to be bolded, like to how such headings are bolded elsewhere on the Platform (e.g., dashboard cards)/
when showing frame numbers. Like is done with timestampsSplitting this off into a separate issue
Labels are not rendering for single frame media fragments. These asynchronous calls I've linked below are likely part of the problem. I will look into this more when time permits. Labels are being flagged as processed after issuing the asynchronous functions. There is no guarantee that the labels are actually loaded with the current approach.
Line 317 in 05fc2a1
Line 349 in 05fc2a1
The following labels will not be rendered when given the clip range
[0, 0]
.
{
"filename": "sample.mp4",
"frames": {
"1": {
"frame_number": 1,
"objects": {
"objects": [
{
"label": "vehicle",
"bounding_box": {
"top_left": {
"x": 0.5090277777777777,
"y": 0.49027777777777776
},
"bottom_right": {
"x": 0.5506944444444445,
"y": 0.5666666666666667
}
},
"index": 1,
"frame_number": 1,
"attrs": {
"attrs": [
{
"type": "eta.core.data.CategoricalAttribute",
"name": "color",
"value": "silver",
"confidence": 1
}
]
}
}
]
}
}
}
}
Originally posted by @benjaminpkane in #41 (comment)
Hello,
I am working on a project which attemps to perform client side rendering of frame meta data (generated by a deep neural network) on the original video footage, without having to re-encode the labels on the video. I came across this repository, and it seems to have something that I have been looking for. I wanted to test your sample applications first, but came across some issues accessing the sample data.
I understand that this repository has been deprecated and that things have been migrated over to Looker, but I am more interested in the contents of this repository since I have very little idea how I would integrate Looker to my custom app (since they seem to be very tightly coupled with Fiftyone platform). Unfortunately, I do not have access to your sample data as the link seemed to have expired. I tried your code for download_data.py, but it seems to have the same issue.
There are no documentation in terms of how to format the data exactly, so i was wondering if you could share this data if you still have access to them.
Thank you very much,
Philip
Adding support for boolean attributes would be of great use to team scoop and the new ontology.
If a mask index is specified, 0 could be a valid mask - draw everything in the mask index
Player51 currently implements a fixed size. This should be extended to allow for resizing and reaction to browser size changes.
Compiled with warnings.
../player51/build/css/player51.min.css (./node_modules/css-loader??ref--6-oneOf-3-1!./node_modules/postcss-loader/src??postcss!../player51/build/css/player51.min.css)
Warning
(1:305) Gradient has outdated direction syntax. New syntax is like `to left` instead of `right`.
I ran vehicle-sense v0.3
on a video for which vehicle-sense
is expected to produce an empty labels output. (Job number c8489f13-ccf7-4f28-aaf4-4dd4473bad16) Indeed, when I download the output json, it correctly contains an empty dictionary {}
. However, when I try to preview the video output, I sometimes get the error message:
"Cannot read property 'objects' of null"
However, sometimes the preview displays correctly (with no annotations on the video as expected).
Showing many significant digits for floating point numeric attributes is not useful and takes up a lot of visual space.
This is how ETA renders numeric attributes, which I request we adopt in Player51:
The options menu automatically closes when the Show attributes
checkbox is clicked. I'm guessing this is because this particular checkbox changes the dimensions of the menu.
The goal is that the options menu never closes in response to a click within the menu, regardless of how the size of the menu changes in response to the click.
Frame/video attributes quickly fill up the screen hide the video and boxes.
This is an issue identified by Brian at https://www.pivotaltracker.com/n/projects/2184985/stories/165914600
The platform console can preview zips locally, but can't in dev/staging/production because the signed URLs we're using don't have extensions.
Lines 498 to 501 in f8ff0f7
For example, when previewing a job's output on the platform console, on certain window sizes, the player will be created above the "preview" option (although this issue can also be reproduced by moving the mouse quickly too). In some cases, the video controls will be invisible, but clicking on where the play/pause button would be still plays/pauses the video, and the cursor changes to a pointer
when hovering over that area.
This issue only seems to occur sometimes, and may depend on the load order of certain things (maybe the video and labels).
At the end of the attached video, pressing the right
arrow key alternates between displaying the overlay and displaying nothing. May be related to the fractional FPS.
cars-2.3s-zipped.zip
This is very strange but I think I'm able to rule this out as a console issue.
It seems that this video has frame objects that start at 122
or so. The json is loading correctly (which is why I think I can rule out a console issue), and the video, when loaded, buffers to about 3/4 of the way through the video (I'm assuming this is so that it starts at the first detected object?). The first frame of the video has no objects, though - and this may be the culprit. @kevinqi34 @MikeJeffers do you guys have any insight into what's going on here?
There is a problem with MediaFragments.js when trying to follow the README, and launching player51 through REACT.
Does not compile when running NPM start.
Currently this situation is only handled by logging a message to the console, as far as I can tell:
Line 122 in fa6965a
On the console, this results in a loading page being shown indefinitely to the user:
Either an event that gets triggered in this case or some other way to render a fallback message would be better.
I performed a search in Scoop that matched the last 3 seconds of a 40 second video. However, when I view the clip, it does not loop 00:37 - 00:40 as expected, it loops back to the beginning of the video.
This search in Scoop returns a match with a time of 0s
. Maybe the match is only 1 frame or something like that. In any case, the clip looping does not work and just lets the entire video play beyond the match.
For example, I searched for Black Panther
and clicked on the matching clip. But there is no Black Panther
in this frame, which should never happen:
Currently I have to reconfigure my Player51 options each time I preview a video.
A more typical experience would be to cache my settings so that I get the same behavior each time I preview an output
Currently Player51 takes a URL pointing to the overlay JSON and then tries to download it. We want to continue to support this if desired, but we also want to support the user passing in the overlay directly as an object (pre-downloaded, if you like).
/**
* Player51 Class Definition
*
* INHERITS: None
* F-MIXINS: None
* @constructor
* @param media is an object that has 'src' and 'type' attributes.
* @param overlay can be either an object describing the overlay or a URL
* from which to download the overlay JSON information
* @param fps is the frame-rate of the media. If it is not provided then it
* will be guessed.
*/
function Player51(media, overlay, fps) {
// if overlay is an object, use it
// if overlay is a string, download the JSON and parse it
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.