Reference Implementation: Store Traffic Monitor in Python*
Details | |
---|---|
Target OS: | Ubuntu* 16.04 LTS |
Programming Language: | Python 3* |
Time to Complete: | 50-70min |
An application capable of detecting objects on any number of screens.
What it Does
This application is one of a series of IoT reference implementations aimed at instructing users on how to develop a working solution for a particular problem. It demonstrates how to create a smart video IoT solution using Intel® hardware and software tools. This reference implementation monitors people activity inside and outside a facility as well as counts product inventory.
How it Works
The counter uses the Inference Engine included in the Intel® Distribution of OpenVINO™ toolkit. A trained neural network detects objects within a designated area by displaying a green bounding box over them. This reference implementation identifies multiple intruding objects entering the frame and identifies their class, count, and time entered.
Requirements
Hardware
- 6th Generation Intel® Core™ processor with Intel® Iris® Pro graphics or Intel® HD Graphics
Software
- Ubuntu* 16.04 LTS Note: We recommend using a 4.14+ Linux* kernel with this software. Run the following command to determine your kernel version:
uname -a
- OpenCL™ Runtime Package
- Intel® Distribution of OpenVINO™ toolkit
Setup
Install the Intel® Distribution of OpenVINO™ toolkit
Refer to https://software.intel.com/en-us/articles/OpenVINO-Install-Linux on how to install and setup the Intel® Distribution of OpenVINO™ toolkit.
You will need the OpenCL™ Runtime Package if you plan to run inference on the GPU as shown by the instructions below. It is not mandatory for CPU inference.
ffmpeg
ffmpeg is installed separately from the Ubuntu repositories:
sudo apt update
sudo apt install ffmpeg
Configure the application
What model to use
The application works with any object-detection model, provided it has the same input and output format of the SSD model.
The model can be any object detection model:
- that is provided by the Intel® Distribution of OpenVINO™ toolkit.
These can be found in thedeployment_tools/intel_models
folder. - downloaded using the model downloader, provided by the Intel® Distribution of OpenVINO™ toolkit.
These can be found in thedeployment_tools/model_downloader/object_detection
folder. - created by the user
By default this application uses the mobilenet-ssd model, that can be accesed using the model downloader. The model downloader downloads the model as Caffe* model files. These need to be passed through the model optimizer to generate the IR (the .xml and .bin files), that will be used by the application.
Downloading the mobilenet-ssd Intel® Model
To download the mobilenet-ssd model you first need to go to the deployment_tools/model_downloader/ folder inside your Intel® Distribution of OpenVINO™ toolkit install folder.
If the Intel® Distribution of OpenVINO™ toolkit was installed using the default installation paths, you can go to the model downloader folder by running:
cd /opt/intel/computer_vision_sdk/deployment_tools/model_downloader/
You can specify which model to download with --name
and the output path with -o
(otherwise it will be downloaded to the current folder).
Run the model downloader with the following command:
./downloader.py --name mobilenet-ssd
Note: You may need to run the downloader.py
command with sudo
You will find the model inside the object_detection/common folder. To make it work with the Intel® Distribution of OpenVINO™ toolkit, the model needs to be passed through the model optimizer to generate the IR (the .xml and .bin files).
If you haven't configured the model optimizer yet, use this link to do so https://software.intel.com/en-us/articles/OpenVINO-ModelOptimizer.
After configuration go to the deployment_tools/model_optimizer/ folder inside your Intel® Distribution of OpenVINO™ toolkit install folder.
If the Intel® Distribution of OpenVINO™ toolkit was installed using the default installation paths, you can go to the model optimizer folder by running:
cd /opt/intel/computer_vision_sdk/deployment_tools/model_optimizer/
Assuming you've downloaded the reference-implementation to your home directory, run this command to optimize mobilenet-ssd:
./mo_caffe.py --input_model /opt/intel/computer_vision_sdk/deployment_tools/model_downloader/object_detection/common/mobilenet-ssd/caffe/mobilenet-ssd.caffemodel -o $HOME/store-traffic-monitor-python/resources
Note: Replace $HOME
in the command with the path to the reference-implementation's folder
If you want to optimize the model for FP16:
./mo_caffe.py --input_model /opt/intel/computer_vision_sdk/deployment_tools/model_downloader/object_detection/common/mobilenet-ssd/caffe/mobilenet-ssd.caffemodel -o $HOME/store-traffic-monitor-python/resources --data_type FP16
The labels file
In order to work, this application requires a labels file associated with the model being used for detection.
All detection models work with integer labels and not string labels (e.g. for the ssd300 and mobilenet-ssd models, the number 15 represents the class "person"), that is why each model must have a labels file, which associates an integer (the label the algorithm detects) with a string (denoting the human-readable label).
The labels file is a text file containing all the classes/labels that the model can recognize, in the order that it was trained to recognize them (one class per line). For the ssd300 and mobilenet-ssd models, we provide the class file labels.txt in the resources folder.
The config file
The resources/conf.txt contains the videos that will be used by the application, one video per line.
Each of the lines in the file is of the form path/to/video label
, e.g.:
videos/video1.mp4 person
The path/to/video
is the path, on the local system, to a video to use as input. Followed by the label
of the class (person, bottle, etc.) to be detected on that video. The labels used in the conf.txt file must coincide with the labels from the labels file.
The application can use any number of videos for detection (i.e. the conf.txt file can have any number of lines), but the more videos the application uses in parallel, the more the frame rate of each video scales down. This can be solved by adding more computation power to the machine the application is running on.
What input video to use
The application works with any input video. Sample videos for object detection are provided here.
For first-use, we recommend using the people-detection, one-by-one-person-detection, bottle-detection videos.
E.g.:
sample-videos/people-detection.mp4 person
sample-videos/one-by-one-person-detection.mp4 person
sample-videos/bottle-detection.mp4 bottle
These videos can be downloaded directly, via the video_downloader
python script provided in the top folder of this repository. The script works with both python2 and python3. Run the following command:
python video_downloader.py
The videos are automatically downloaded to the resources/
folder.
Using camera stream instead of video file
Replace path/to/video
with the camera ID, where the ID is taken from yout video device (the number X in /dev/videoX).
On Ubuntu, to list all available video devices use the following command:
ls /dev/video*
Setup the environment
To setup the environment variables required to run the Intel® Distribution of OpenVINO™ toolkit applications:
source /opt/intel/computer_vision_sdk/bin/setupvars.sh
Note: This command only needs to be executed once in the terminal where the application will be executed. If the terminal is closed, the command needs to be executed again.
Building and using the CPU extension library
When runnning Intel® Distribution of OpenVINO™ toolkit Python applications on the CPU, the libcpu_extension.so CPU extension library is required. This can be found in /opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ ubuntu_16.04/intel64/
after building the OpenVINO examples.
Run the application
To see a list of the various options:
./store-traffic-monitor.py -h
To run the application with the needed models:
./store-traffic-monitor.py -m resources/mobilenet-ssd.xml -l resources/labels.txt -e /opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64/libcpu_extension_avx2.so
Having the input video loop
By default, the application reads the input videos only once, and ends when the videos ends.
In order to not have the sample videos end, thereby ending the application, the option to continously loop the videos is provided.
This is done by running the application with the -lp true
command-line argument:
./store-traffic-monitor.py -lp true -m resources/mobilenet-ssd.xml -l resources/labels.txt -e /opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64/libcpu_extension_avx2.so
This looping does not affect live camera streams, as camera video streams are continuous and do not end.
Running on different hardware
A user can specify what target device to run on by using the device command-line argument -d
followed by one of the values CPU
, GPU
, or MYRIAD
.
If no target device is specified the application will default to running on the CPU.
Running on the CPU
Although the application runs on the CPU by default, this can also be explicitly specified through the -d CPU
command-line argument:
./store-traffic-monitor.py -d CPU -m resources/mobilenet-ssd.xml -l resources/labels.txt -e /opt/intel/computer_vision_sdk/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64/libcpu_extension_avx2.so
Running on the integrated GPU
To run on the integrated Intel® GPU, use the -d GPU
command-line argument:
./store-traffic-monitor.py -d GPU -m resources/mobilenet-ssd.xml -l resources/labels.txt
Running on the Intel® Neural Compute Stick
To run on the Intel® Neural Compute Stick, use the -d MYRIAD
command-line argument:
./store-traffic-monitor.py -d MYRIAD -m resources/mobilenet-ssd.xml -l resources/labels.txt
Note: The Intel® Neural Compute Stick can only run FP16 models. The model that is passed to the application, through the -m <path_to_model>
command-line argument, must be of data type FP16.
Using the browser UI
The default application uses a simple user interface created with OpenCV. A web based UI, with more features is also provided here. In order for the application to work with the browser UI, you have to set the existing UI_OUTPUT global variable to TRUE. E.g.:
UI_OUTPUT = True