siudya/orb_fpga

Introduction

This project implements a ORB feature extractor accelerator on FPGA (on PYNQ-Z2 board). ORB ( Oriented FAST and Rotated BRIEF) feature is a kind of image feature based on oriented FAST feature and BRIEF descriptor. It's easy to calculate and robust to rotating so it's widely used in embedded computer vision application. For example, it's used in feature matching of some SLAM (Simultaneous Localization And Mapping) system, such as ORB-SLAM and ORB-SLAM2. SLAM system help robot or other platform to locate themselves and build a map of their surroudings. Here is a brief explanation of how ORB Features are extracted.

The extractor will take a gray image and applys a FAST extractor on it. After feature points being found, it will calculate m01 and m10 moment of the feature points and figure out the angles of them. Calculating BRIEF descriptors needs those angles and a gray image which is the original gray image applied a 7x7 gaussian filter.

Result

1.Resource Utilizaton (on Pynq-Z2)

Resource	Utilization	Available	Utilization
LUT	35807	53200	67.31
LUTRAM	1412	17400	8.11
FF	54895	106400	51.59
BRAM18K	50	140	35.71
DSP	24	220	10.91

2.Performance (testing on images in 640*480)

Platform	Average Delay	Throughoutput
PS 650mHz -O	291.7 ms	3.43 FPS
PS 650mHz -O2	98.5 ms	10.20 FPS
PL 140mHz	17.56 ms	56.58 FPS

3. Result Picture

Feature points are drawn on the picture.Descriptors can be read in the buffer filled by DMA.

Quick Start

Run these command lines on your Pynq-Z2 Board (tested on v2.4):

sudo pip3 install git+https://github.com/Siudya/ORB_FPGA.git

Contents of each folder

ip

HLS sources files

pynq_arch

Vivado project

pynq_notebook

Notebook run in Jupyter Notebook

software_test

A .cpp file that test the same process on CPU

test_data

Images for testing

hw

.bit and .hwh files for PYNQ

How to rebuild Vivado project

vivado 2018.3 is required.

Step 1: rebuild HLS IP

Open Vivado HLS command terminal and run these commands :

cd <path-to-proj>/ip
vivado_hls -f build_ip.tcl

This should take about half an hour.

Step 2: rebuild top project

Open Vivado command terminal and run these commands :

cd <path-to-proj>/pynq_arch
source pynq_arch.tcl

orb_fpga's Issues

More resources are used when project is rebuilt and synthesised

Would you do me a favor? I need some help.
It costed 139115 luts when the file called comDescriptor.cpp was synthesised.
I followed your steps to rebuild vivado project， but i failed.
Thanks!

Generating the bitstream

Usually once the HLS synthesis is finished did you autoconnect the ip in vivado ? or did you do the place and route of the ip?

The descriptor calculated by Pynq is different from software_test

Hello, I am trying to implement this project into ORB-SLAM2. However, the descriptor calculated by Pynq is different from software_test.
I compared one same keypoint from one picture, and this is the result from Pynq.

And this is the descriptor calculated by software_test.
132
236
142
31
217
46
166
238
246
179
203
24
244
227
73
75
81
110
106
114
112
221
219
62
61
38
87
60
128
113
131
130

By calculation we can know that they are different. Is this normal? Thank you very much!

Change input image size

Thanks for your wonderful work. I am trying to change input image size from (640,480) to (2560,2560). I change WIDTH and HEIGHT in xf_config_params.h. I also modify src_buf in python as well. Is there any place I ignore or is there any suggestion for me? Thanks a lot.

Final design is overutilizing the resources when rebuilding

As mentioned by a few people, the HLS IPs are overutilizing the resources present on the PYNQ Z2 board, and should resolve when generating bitstream. However the final design is still overutilizing the resources resulting in failed implementation.

NO TOP Module

After following the given step of rebuild project in vivado, I try to generate bitstream. However, I received the following error,
[Common 17-70] Application Exception: Top module not set for fileset 'sources_1'. Please ensure that a valid value is provided for 'top'. The value for 'top' can be set/changed using the 'Top Module Name' field under 'Project Settings', or using the 'set_property top' Tcl command (e.g. set_property top <name> [current_fileset]).
Are there any help??

Does the code here support NPC >1?

I found several lines below, and it seems that the original code

469		if(NPC == XF_NPPC8)
470		{
471		}

If NPC equals XF_NPPC8, the module will have no output. And in the else branch, the code only implements one pixel's judgement. Similar operations could be seen in other module.