evercam/ex_nvr

Video recording and computer vision for edge devices

ExNVR

NVR (Network Video Recorder) for Elixir using Membrane Framework

Contents

• ExNVR
  • Contents
  • Installation
    • Docker
      • Arm/v7
    • Releases
  • Environment Variables
  • WebRTC
    • WebRTC Configuration
    • Page URL
    • Embedding
  • Features

Installation

Docker

To get started with ex_nvr it's preferable and easy to run a docker image:

docker run --rm -it -p 4000:4000 --env-file .env ghcr.io/evercam/ex_nvr:latest

Or create a new image using the Dockerfile. Run the following command from the root folder of the project

docker build -t ex_nvr:custom .

This will create an image ex_nvr with custom tag. To run it, issue this command:

docker run --rm -it -p 4000:4000 --env-file .env ex_nvr:custom

Arm/v7

There's currently no automated docker build for arm/v7 since building the image using buildx and Qemu take ages to complete, the workaround is to build the image on the target host itself

docker build -t ex_nvr:0.6.0 -f Dockerfile-armv7 .

Releases

Starting from v0.6.0, there are elixir releases for GNU/Linux and debian packages available as release assets.

You can download the tar file and uncompress it. cd to the decompressed directory and then run:

sudo ./run

The sudo is needed to create the database as the default location is /var/lib/ex_nvr which is not accessible to not-root users by default. If you want to run it as the current user, either:

• Update the DATABASE_PATH env variable in the env.sh file in releases/<version> to point to another location.
• Or create the /var/lib/ex_nvr folder and make it owned by the current user sudo chown $UID:$GID /var/lib/ex_nvr

For debian packages, just download the package and run:

sudo dpkg -i <package name>

This will install ex_nvr as a systemd service under the name ex_nvr. To run it issue the command

sudo systemctl start ex_nvr.service

To start it on boot

sudo systemctl enable ex_nvr.service

To delete the package, first stop the service and then run dpkg to delete it

sudo systemctl stop ex_nvr.service
sudo systemctl disable ex_nvr.service
sudo dpkg -P ex-nvr

Environment Variables

If you want to configure some aspects of ex_nvr, you can set the following environment variables:

Env variable	description
DATABASE_PATH	The path where Sqlite database will be created. Defaults to: /var/lib/ex_nvr/ex_nvr.db
EXNVR_HLS_DIRECTORY	The directory where hls playlists will be stored. Defaults to: /tmp/hls. It is not necessary to expose this folder via volumes since the playlists are deleted each time the user stop streaming.
EXNVR_ADMIN_USERNAME	The username(email) of the admin user to create on first startup. Defaults to: admin@localhost.
EXNVR_ADMIN_PASSWORD	The password of the admin user to create on first startup. Defaults to: P@ssw0rd.
SECRET_KEY_BASE	A 64 byte key that's used by Pheonix to encrypt cookies
EXNVR_URL	The url to use for generating URLs. The host is used as a default value for check_origin of the websocket. Defaults to: http://localhost:4000
EXNVR_CHECK_ORIGIN	if the transport should check the origin of requests when the origin header is present. May be true, false or a list of hosts that are allowed. Defaults to true.
EXNVR_HTTP_PORT	Http port, defaults to: 4000
EXNVR_CORS_ALLOWED_ORIGINS	A space separated allowed origins for CORS requests. defaults to: *
EXNVR_ENABLE_HTTPS	Enable https, default: false
EXNVR_HTTPS_PORT	Https port, defaults to: 443
EXNVR_SSL_KEY_PATH	The path to the SSL key.
EXNVR_SSL_CERT_PATH	The path to the SSL certificate.
EXNVR_JSON_LOGGER	Enable json logging, defaults to: true

WebRTC

WebRTC Configuration

Env variable	description
EXTERNAL_IP	The external IP address to which attach the turn server. Defaults to: 127.0.0.1
VIRTUAL_HOST	The TURN server domain name. Defaults to: localhost
INTEGRATED_TURN_PORT_RANGE	UDP port range to use for WebRTC communication. The number of ports in the range decides the total number of users that can stream at the same time. Defaults to: 30000-30100
INTEGRATED_TURN_TCP_PORT	TCP port of TURN server. Defaults to: 20000
INTEGRATED_TURN_PKEY	Private key to use for TURN server. If not provided, TURN will not handle TLS connections. Defaults to: nil
INTEGRATED_TURN_CERT	Certificate to use for TURN server. If not provided, TURN will not handle TLS connections. Defaults to: nil

Note

Actually even if private and certificate files are provided, TURN server will not handle TLS connections. We'll support this in the near future.

Page URL

You can access the webrtc page using the following url:

http://localhost:4000/webrtc/{device_id}

Embedding

A webrtc player can be embedded in web page by using iframe

<iframe width="640" height="480" src="http://localhost:4000/webrtc/device_id?access_token=token" title="ex_nvr" allowfullscreen></iframe>

The access_token will eventually expire and must be updated to ensure the proper functioning of the embedded page. We plan to enhance this by introducing the capability to make the page public or generate non-expiring tokens with view privileges.

Features

The main feature of this project is to store video streams retrieved from devices to local file system and allow users to stream back those recorded videos.

• Devices: read video streams from

  • IP Cameras
  • USB / Webcams
  • Raspberry Pi Cameras
  • Plain RTSP stream
  • File Upload (Helpful for debug, benchmarking & demos)

• Camera Streams: save and playback

  • Main stream
  • Sub-stream

• Device Discovery: discover devices

  • Onvif discovery

• Video Codecs: allow storing and streaming videos with following codecs

  • H264
  • H265

• Streaming: live view and playback

  • HLS
  • WebRTC (only mainstream, disabled for H265)
  • Webm
  • RTSP
  • RTMP

• Integration: Integrate with third party software

  • Web Hooks: Notify external services of new recorded videos
  • Unix-domain Socket: Send snapshots via unix-domain socket

• Other: other interesting features

  • Support audio
  • Support multiple devices in same instance
  • Get snapshot from live/recorded stream
  • Download videos of arbitrary durations
  • Add stats/info about streams (for devs)
  • API / API documentation
  • Gather metrics
  • Run machine learning models on video streams (live video / stored footages)
  • Sync recorded videos to cloud storage
  • Application for management of multiple NVRs
  • Support HTTPS

ex_nvr's Issues

NVR Device Management Features ( WiFi, VPN, Modem ... )

That could include management of and alerts for:

• #19
• Encryption
• Time
• Modem
• VPN
• WiFi
• Battery Voltage Reading #154
• Relay Control (on Pi) #154

Some of these might suit implementing via a Livebook #369

Support for multiple devices

Allow storing video footages from multiple devices.

Ability to fix Camera time to have correct recording time.

For hikvision, it's simple,

1. Setup NTP
   

2. Setup timezone (same as jetson)
   

3- REBOOT

GET Snapshot (Live)

Reference Architecture for Computer Vision produced Data (e.g. Motion/Brightness)

It would be very helpful to see a reference architecture for how we would like to gather and store timeseries data.

Motion Levels (full-image) or Brightness Levels (same) might be a decent start.

v0.1 Release Goals

• Live View on a separate tab (with stream selector if you wish). Showing HLS stream. (Not Transcoding).
• Add as much developer under the bonnet info as you can easily access. Sources may include ( rtsp stream info, calculated info (Explain the calculation), ffprobe, onvif & device).
• Add APIs on the actual pages (live view, recordings). Just text is fine.
• Add Device Method ( v0.1 is ok with just 1 device, 2 streams )

Ability to import footage from another NVR.

It is likely that we will sometimes want to be replacing an existing installation (with recordings) with ex-nvr.

In this case, it would sometimes be nice to replace the nvr, but not the hdd and that means importing old footage into ExNVR.

I wonder about a process that would import recordings from other sources. The 2 likely ones being Nx and Hik

[ ] Ansible script to install ExNVR and uninstall (NxWitness). @dhairyas17 ?
[ ] Script to import existing recordings (and cameras?) to ExNVR

Make project open source

Suggest this licence which allows us to control commercial use and support and provision as a service

https://www.tldrlegal.com/license/gnu-affero-general-public-license-v3-agpl-3-0

HDD / Storage Area Monitoring & Management

We want to know things like:

• Serial # and physical characteristics ( Capacity, model etc.) https://unix.stackexchange.com/questions/121757/harddisk-serial-number-from-terminal
• Remaining Space & Time to Full (Days).
• Allocation of Storage areas for different processes & cameras.
• Health (Bad sectors)
• Alerts for Bad Health or LImited Remaining Space.
• Formatting & FileSystem Management

I'd be ready to have a debate about only limiting scope to logical storage areas, but I'm currently leaning in favour of ExNVR also dealing with Physical Storage.

Seeing the Erlang docs, maybe the easiest implementaiton is just whatever we can get from this;
https://www.erlang.org/doc/man/disksup#get_disk_info-0

• [https://github.com//issues/331]

Show the build # / version # in top right nav

This is the beginning of a long journey. We want to plan for regular updates, CI style ... we want to make that easy & low friction.
I'm suggesting a very visible version number, probably in the top right corner.

Delete expired tokens

Add HLS playback streaming

Playback saved recordings via HLS. To achieve this we need to the following steps:

• Read the mp4 files
• De-mux the mp4 files
• Convert the de-muxed h264 data to Annex B format
• Transcode the stream (if necessary)
• Convert to HLS playlist

We need to fetch the optimal position on the files to start streaming by looking at the nearest keyframe from the specified date.

Note
Membrane Team currently created a depayloader for H264 from MP4 files, but it's still under review and not yet merged. An alternative solution is to use ffmpeg to read the files.

Switch between light/dark modes

Rename from NVR to VMS

Video Management Software is probably more appropriate.

Add user information fields

Add some fields to the user, like:

• First name & last name
• Username
• Profile picture
• language (useful when adding i18n to the NVR)

Live streaming with hls

Add live streaming with HLS.

Add timezone and state fields to device

Add two fields to the device:

• Timezone - when requesting a playback we should use the device timezone and not UTC
• State - defines the current state of the device: RECORDING, STOPPED, FAILED, ...etc.

API in Plurals

UserS
RecordingS
StreamS
DeviceS

Add an endpoint to get recorded videos

Support camera sub-stream streaming and storage

Add a configuration to allow the user to choose storing camera sub-streams.

Camera Passthrough / Proxy (esp. Snapshot).

We've found in the past that we sometimes want to talk directly to the camera. e.g. to get a snapshot (and maybe this is how onvif PTZ works?)

Today we do this by sending commands directly to the camera, but life could be (maybe?) a bit simpler if we only ever need to talk to the NVR and it would proxy the commands and responses.. as it knows where the camera is and how to auth.

I am not sure about this, but want the idea recorded somewhere so it can be discussed.

( ... related: I could imagine embedding an interactive swagger UI with the complete Hikvision ISAPI API there )

Make the Dashboard page fit to screensize

This page:

should fit to the available screensize and have a full-screen mode.

Add metrics from live_dashboard (?)

Offer some visibility to metrics using https://github.com/phoenixframework/phoenix_live_dashboard .

Thumbnails Mosaic for scrubbing in UI

Create mosaic thumbnails as in the following image

This will be useful to show when you hover over the progress bar of the player to allow fast seeking (like youtube).

A fullscreen button on the Liveview player.

H265 support

To support H265 we need the following elements:

• H265 depayloader - Extract video NAL units from RTP packets.
• H265 parser - parse the NAL units and create access units from them, each access unit represent a encoded frame.
• H265 encoder/decoder - Encode & decode the H265 stream (use ffmpeg for that)
• MP4 payloader - An MP4 payloader of the H265.
• Add h265 support for ex_nvr pipelines

RTSP source: buffer packets and handle parameter sets in SDP

Currently, when the packets arrive to the RTSP source and this latter is not in playing state, the packets are discarded. This packets may contain SPS and PPS which are needed for subsequent packets. So we'll buffer the packets until the source is linked and switched to playing state.

The second part of this issue is that parameter sets SPS and PPS may be signaled out of band (in SDP). We need to extract this params and pass them to the parser since this parameters may not be repeated in the bytestream.

Wishlist thinking from Evercam Dash perspective.

• Meta Data v1 (Motion & Lux). We want timeseries data and lots of it. Let's begin to capture it and save it. This serves the purpose of a reference architecture for other backend and for frontend. I expect the cost to be low and I think this would help us get out thinking straight.
• Server Side Zooming (I think Truelook do this very well). i.e. I zoom to a region and the VMS only streams that zoomed area.

Timelapse creator

Create timelapse video from recorded footages

Activity Logs

As a developer focussed NVR, I would like easy access to all device and stream activity.

• User Login/Logout
• Stream Requests
• Device creation/update
• Reboots
• ...

API: get recordings

Since the recorded footage is in small chunks, this endpoint will get the continuous start date and end date of the recorded footage.

Add authentication

Add authentication via web page (/login) or API (/api/login)

• Create a user schema
• Create a session schema (for API endpoints)
• Create an API endpoint
• Create the login UI

---

@halimb I would like you to try to create a login feature using Elixir/Heex templates. (you can work on it whenever you're free)

What's Special About This NVR/VMS? (And which is it?)

There are 3 people we are primarily considering here:

The Computer Vision Developer (e.g. Dhia).

He wants to run his code without having to worry about the nuances of different cameras. If he makes changes to streams etc. he'd like clarity on the consequences (there are always consequences). E.g. Dhia can consume different streams or request different transcoding configurations. Which ever selections he makes, the inputs, outputs and resources consumed should be visible. In future version we can add developer hints ( Hey, why don't you try using the 3rd stream here instead? ).

The Web Developer (e.g. Yazid).

Famously these people are not ready for the pain of video. We must protect them, they are gentle souls. Wherever possible, we let them continue in their Ctrl+C , Ctr+V utopia.

The TechOps Person. (e.g. Ramzi) .

This person cannot be expected to know all the nuances of different camera & NVR configurations. We will identify what really matters and help them to see that at a glance.

So who are we not supporting?

End Users.

This does not have to be friendly to anyone who doesn't know what Docker, an API or an IFRAME is.

What's special about this app?

• Docker Install (Certainly initially, until we understand this chap: https://www.nerves-hub.org/ )
• GPU Friendly (Jetsons et al). Our primary platform is The Jetson Orin Nano
• Developer Friendly (APIs for everything, easy to see). We will show the APIs on the relevant pages. e.g. Live View will show the inbound and outbound streams.
• System Info friendly> Great detail on memory consumption, CPU consumption etc.
• Visibility to stream details and to pipelines and to consequences (consumption). So I can see what the RTSP in is and I can also specify what I want it to be (untouched or custom) . If custom I want to see resources consumed. This is not common and is importnat to us.
• HDD Status https://unix.stackexchange.com/questions/121757/harddisk-serial-number-from-terminal
• VPN Config Built in. WebRTC.

Ability to download videos

Some suggestions:

PTZ

Implement OnVIF PTZ functionality.

I'm flagging this as a good first issue because I suspect a lot of the principles already exist in Evercam Backend (and that was done by an Elixir Newbie ... see video posted online).

Add UI PTZ dashboard

to ExNVR dashboard

NB: only display the buttons when camera supports PTZ

Show available recordings on calendar

A user would like to see whether recordings exist or not. (to use for download for incident requests)

For example, Hikvision shows blue dots with timeline

NXW shows green days with timeline

@magixus I've rephrased this issue to focus on the functionality that we do want (ability to see available recordings calendar) rather than what is not needed. I think your main point is good.

Wrong time in timeline

Clicking on the timeline at date 2023-08-01T00:56:10 with Europe/Dublin timezone send the following date to the back-end 2023-08-01T23:56:10Z.

GET Snapshot (Recorded)

OnVIF Discovery

Search network for available cameras, one-touch adding them as devices.

• Discover devices using WS-Discovery
• UI to add discovered cameras as devices

Anonymise Stream (Optionally After x Days)

The big idea here is:
a) An MVP for a feature that customers are asking for
b) An MVP for a Computer Vision Pipeline running on ExNVR Recordings

Not sure how useful these links are, but they seem relevant (if a bit old):
https://github.com/NVIDIA-AI-IOT/redaction_with_deepstream
https://github.com/NVIDIA-AI-IOT/retinanet_for_redaction_with_deepstream

Settings:
Choose Recordings Location.
Choose Days

The specific use case here is that we keep full recordings for 30 days and then anonymise.
We'll probably also need a scenario where the live stream is redacted before it is saved. My hunch is that that's a fundamentally different pipeline.

Add UI for HLS live view

Create a Raspberry Pi version.

This will allow us to roll out on more existing machines. Quite exciting.

Recordings Tab: Clips Segment Download

We agreed it would be ok to make a v1 where it is just a list of chunks from the filestorage.

v2 might allow user to specify a date range and get a produced file (Via API)

How far to go with the UI & other questions

Topic 1: PTZ Functionality
We dont need to implement any PTZ functionality, we could do it all direct to the camera via evercam dash. But, it seems to make sense that we would in order for this to be a "proper" NVR.

Topic 2: IFRAME / Microservices
A model that is working elsewhere in Evercam is layers of iframes e.g. drone/bim/camera view ... and, we could aim for a world where all the camera UI functionality is on ExNVR and simply iframed into Dash.

I've not thought this through, but wanted to write it somewhere.

Ability to edit camera settings

https://gitlab.com/MachinesAreUs/archeometer for project visibility

This could be an easy addition to give better visibility into the workings of the application.

Notes from call with Marco

• Keep updating the WG4 camera.
• Tell Marco when you do a new deploy.
• Put a version number in the top right (hardcoded is ok)
• Default to using the primary stream (I get no value from the secondary stream at all)
• I am ready to go open. Suggest AGPL 3.0 https://www.tldrlegal.com/license/gnu-affero-general-public-license-v3-agpl-3-0
  #88
• Add this https://gitlab.com/MachinesAreUs/archeometer #87
• Halim will work on the download clips UI from Dash.
• Any chance of a clips UI from ExNVR? #86
• Billlal to understand and help promote the Liveview frontend agenda (I'm relaxed about asking you to make beautiful UI ... but I need you to bring the technology to the other people in evercam)
• A fullscreen button. #85

What I dont want to work on
ONVIF Discovery > Only interesting for the Open Community, not for Evercam #89
WebRTC > If you mean Peer to Peer, it's not interesting for evercam as we always have a VPN tunnel.
Nerves > Buildroot story : Not interesting for us to go to this level now. Docker is much preferable. For now. This might be interesting to support Raspberry Pi.
h265 > Maybe, but in terms of difficulty versus reward it's not interesting. We might also skip and go straight to AV1.

What I do want to work on
Ability to download clips (Halim is working on this)
Raspberry Pi Deployment Method. (Docker, Direct, Nerves ... my vote is for docker (try not to rely on Dhia, he's occupied).
Some computer vision pipelines & local data storage
e.g. Readaction Camera implementation
Motion Detection/Object Detection (this produces data ! that's good). Then Halim can visualise this data in the timeline along with the recording runs.

** other **
IFRAME: This is not going to be our primary path. We build basic functionality in ExNVR and the rest in the Dash.

Timeline cleanup

I dont think it makes sense to show Mon Tue Wed (the days of the week)

or the time when it's 00:00:00

Design the layout of the website

1- Remove the extra Phoenix default stuff (clean the top bar)
2- The main page should have a login for or redirect the main connected page
3- Top bar:
-- Add "Live"
-- Add "Recordings"
-- Add "Phoenix dashboard"
-- Add "API docs"

Do not save mp4 files with a small duration

When there's connection problems, the segmenter may generate a lot of small segments, since it flushes the current segment if RTSP connection is lost.

We should check for small segments (say less than 10 seconds) and discard them.