An approach to compute patch-based local feature descriptors efficiently in presence of pooling and striding layers for whole images at once.
A Pytorch and TF (WIP) implementation of the paper "Fast Dense Feature Extraction with CNNs with Pooling Layers" https://arxiv.org/abs/1805.03096
In recent years, many publications showed that convolutional neural network based features can have a superior performance to engineered features. However, not much effort was taken so far to extract local features efficiently for a whole image. In this paper, we present an approach to compute patch-based local feature descriptors efficiently in presence of pooling and striding layers for whole images at once. Our approach is generic and can be applied to nearly all existing network architectures. This includes networks for all local feature extraction tasks like camera calibration, Patchmatching, optical flow estimation and stereo matching. In addition, our approach can be applied to other patch-based approaches like sliding window object detection and recognition. We complete our paper with a speed benchmark of popular CNN based feature extraction approaches applied on a whole image, with and without our speedup, and example code (for Torch) that shows how an arbitrary CNN architecture can be easily converted by our approach.
These instructions will explain how to use the Fast Dense Feature Extraction (FDFE) project.
- Python>=3.5
- pytorch>=1.0
- tensorflow=2.0
- numpy
- matplotlib
- Install all prerequisites - there maybe some dependency collisions between TF-Pytorch for simplicity choose one for time being
- Clone the project
-
pytorch
-
FDFE.py
- implementation of the all approach layers and pre & post process methods as described in the paper , including:- MultiMaxPooling
- MultiConv
- multiPoolPrepare
- unwarpPrepare
- unwarpPool
-
BaseNet.py
- This refers to an implementation of a pre-trained CNNon training patches
.
-
sample_code.py
- test run
-
-
tf
-
FDFE.py
- implementation of the all approach layers and pre & post process methods as described in the paper , including:- MultiMaxPooling
- MultiConv
- multiPoolPrepare
- unwarpPrepare
- unwarpPool
-
BaseNet.py
- This refers to an implementation of a pre-trained CNNon training patches
.
-
sample_code.py
- test run -
tests
tf_tests.py
- unit tests to check the output shapes of the FDFE layers
-
Now you should sample_code.py
to make sure that FDFE project works correctly.
The test generates a random input image
of size
imH X imW
and evaluates it on both
and
.
The script continues and evaluates differences between both CNN's outputs and performs speed benchmarking.
There are two modes of operation for
:
-
singlePatch mode- run
over a single patch
pH x pW
that would get cropped from input imagearound
-
allPatches mode - run
over multiple patches at ones. here
batch_size
will determine how many patches would get evaluated at once.
In sample_code.py
there are initial parameters that could be adjusted:
- Tested Input Image dimensions:
- imH - Input image height - imW - Input image width - pW - patch Width - current implementation supports only odd width size - pH - patch Height - current implementation supports only odd width size - sL1 - First stride value - sL2 - Second stride value . . . - sLn - n-th stride value
- patch_i_center - patch row center - patch_j_center - patch column center
- batch_size - number of patches to be evaluated at the same time
Script outputs the following:
- aggregated difference between base_net (
) output and slim_net output (
)
- For
, an averaged evaluation per patch
- For
, Total evaluation per frame. i.e. the entire input image
Expected verbose would look like: (depends on running mode):
Total time for C_P: 0.017114248275756836 sec ------------------------------------------------------------ Averaged time for C_I per Patch without warm up: 0.0010887398617342114 sec ------- Comparison between a base_net over all patches output and slim_net ------- aggregated difference percentage = 0.0000000000 % maximal abs difference = 0.0000000000 at index i=0,j=0 ------------------------------------------------------------
In order to use your own pre-trained network that operates on patches you would need to:
- implemented your network in
BaseNet.net
- modify
SlimNet.py
accordingly:- Duplicate
BsetNet.py
model layers according to its order, e.g.
self.conv1 = list(base_net.modules())[change_this_index]
- For every
MaxPool2d
layer placemultiMaxPooling
instead with the decided stride value (sLn) - Deplicate unwrapPool layers according to the number of
multiMaxPooling
in your model - Do not remove the following layers - multiPoolPrepare, unwrapPrepare
- Duplicate
- Verify TF implementation
- Export model to IR language
Contributions are always welcome! Please read the contribution guidelines first.
- Erez P. ([email protected])
- Arnon K. ([email protected])
A big thanks to the following individuals for designing the approach:
- Christian Bailer ([email protected])
- Tewodros A. Habtegebrial ([email protected])
- Kiran Varanasi1 ([email protected])
- Didier Stricker ([email protected])