yangjirui/rrpn_fpn_tensorflow

Arbitrary-Oriented Scene Text Detection via Rotation Proposals(RRPN)

---

A Tensorflow implementation of RRPN based on FPN. You can refer to the papers RRPN paper here, and FPN paper here.

We(me and yang Xue) also implement the $R^2 CNN$ (link)based on FPN. And you can find the paper here. And we also proposed R-DFPN, but papers is under review so the complete code and instructions are will uploaded later. If useful to you, please star to support my work. Thanks.

---

##Configuration Environment: ubuntu+ python2 + tensorflow1.2 + cv2 + cuda8.0 + GeForce GTX 1080 If you want to use cpu, you need to modify the parameters of NMS and IOU functions use_gpu = False in cfgs.py You can also use docker environment.

docker push yangxue2docker/tensorflow3_gpu_cv2_sshd:v1.0

---

##Installation:

Clone the repository

---

##Make tfrecords: The data is VOC format, reference here. You can make tfrecords as following:

cd $RRPN_ROOT/data/io/  
python convert_data_to_tfrecord.py --VOC_dir='***/VOCdevkit/VOCdevkit_train/' --save_name='train' --img_format='.jpg' --dataset='ship'

If the format of your data is not VOC, you can follow my advices below and generate tfrecords by yourself.

• Features in TFrecords are as follow:

feature = tf.train.Features(feature={
            'img_name': _bytes_feature(img_name),
            'img_height': _int64_feature(img_height),
            'img_width': _int64_feature(img_width),
            'img': _bytes_feature(img.tostring()),
            'gtboxes_and_label': _bytes_feature(gtbox_label.tostring()),
            'num_objects': _int64_feature(gtbox_label.shape[0])
        })
# the format of gtboxes of label:
'''
gtboxes_and_label are gtboxes and labels in a img. It's a Matrix.
Shape:(num_objects, 9)
Contents:(x0, y0, x1, y1, x2, y2, x3, y3, category)

(x0, y0, x1, y1, x2, y2, x3, y3) are 4 vertices of inclined rectangle.
Note: they can be unorderd

(x0, y0)
+----------+(x1, y1)
|          |
+----------+(x2, y2)
(x3, y3)
'''

• You can refer data/io/convert_data_to_tfrecord.py and data/io/read_tfrecord.pyto to make yourself code.

---

##Demo This is a demo about detecting arbitrary-oriented buildings.(our dataset from SpaceNet and some modifications have been done)

1. Download Trained Weights: you can download trained weights here(the link will attach soon)
2. unzip them on ***/FPN_with_RRPN/output/trained_weights/FPN_RRPN_v1

cd tools/
python inference.py

4. Put your test imgs on FPN_with_RRPN/tools/inference_image. I have uploaded some imgs for demo test.
5. Detection results will show on folder: /FPN_with_RRPN/tools/inference_results

---

##Train

1. download pretrain weight(resnet_v1_101_2016_08_28.tar.gz or resnet_v1_50_2016_08_28.tar.gz) from here, then extract to folder data/pretrained_weights.
2. Change the name of pretrained weights to 'resnet_v1_101.ckpt'(for resnet101), or 'resnet_v1_50'(for resnet 50).(you can also use softlink)

cd tools/
python train.py

---

##Some Test Results

rrpn_fpn_tensorflow's Issues

OutOfRangeError (see above for traceback): PaddingFIFOQueue

Thanks for sharing your work. I have created my own tfrecord file which can be used for traning R2CNN_FPN_Tensorflow. However, when I train the RRPN model, the following error raised :

`2018-03-25 22:44:13: step1 image_name:0000000110.jpg |
rpn_loc_loss:0.297302305698 | rpn_cla_loss:1.0283062458 | rpn_total_loss:1.3256084919 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.43538749218 | fast_rcnn_total_loss:1.43538749218 |
total_loss:3.46225190163 | per_cost_time:3.72758412361s
2018-03-25 22:44:23: step11 image_name:0000000102.jpg |
rpn_loc_loss:0.205864414573 | rpn_cla_loss:0.336254030466 | rpn_total_loss:0.542118430138 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.04045414925 | fast_rcnn_total_loss:1.04045414925 |
total_loss:2.28382301331 | per_cost_time:0.68939781189s
2018-03-25 22:44:30: step21 image_name:0000000158.jpg |
rpn_loc_loss:0.27754047513 | rpn_cla_loss:0.293545395136 | rpn_total_loss:0.571085870266 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.0313565731 | fast_rcnn_total_loss:1.0313565731 |
total_loss:2.30369186401 | per_cost_time:0.698785066605s
2018-03-25 22:44:37: step31 image_name:0000000169.jpg |
rpn_loc_loss:0.126704588532 | rpn_cla_loss:0.266889303923 | rpn_total_loss:0.393593907356 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.21958398819 | fast_rcnn_total_loss:1.21958398819 |
total_loss:2.3144288063 | per_cost_time:0.69150686264s
2018-03-25 22:44:44: step41 image_name:0000000193.jpg |
rpn_loc_loss:0.6847884655 | rpn_cla_loss:0.277387470007 | rpn_total_loss:0.962175965309 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.05666255951 | fast_rcnn_total_loss:1.05666255951 |
total_loss:2.72009158134 | per_cost_time:0.70182800293s
2018-03-25 22:44:51: step51 image_name:0000000060.jpg |
rpn_loc_loss:0.375000298023 | rpn_cla_loss:0.243579685688 | rpn_total_loss:0.618579983711 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.01494431496 | fast_rcnn_total_loss:1.01494431496 |
total_loss:2.33477902412 | per_cost_time:0.69392490387s
2018-03-25 22:45:00: step61 image_name:0000000013.jpg |
rpn_loc_loss:0.184998586774 | rpn_cla_loss:0.218382105231 | rpn_total_loss:0.403380692005 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.16415882111 | fast_rcnn_total_loss:1.16415882111 |
total_loss:2.26879382133 | per_cost_time:0.687323093414s
2018-03-25 22:45:07: step71 image_name:0000000072.jpg |
rpn_loc_loss:0.396233230829 | rpn_cla_loss:0.345920056105 | rpn_total_loss:0.742153286934 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.13157987595 | fast_rcnn_total_loss:1.13157987595 |
total_loss:2.57498908043 | per_cost_time:0.748478889465s
2018-03-25 22:45:14: step81 image_name:0000000042.jpg |
rpn_loc_loss:0.635928988457 | rpn_cla_loss:0.273401498795 | rpn_total_loss:0.909330487251 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.09614527225 | fast_rcnn_total_loss:1.09614527225 |
total_loss:2.70673298836 | per_cost_time:0.71663403511s
2018-03-25 22:45:21: step91 image_name:0000000162.jpg |
rpn_loc_loss:0.111868612468 | rpn_cla_loss:0.167042255402 | rpn_total_loss:0.27891087532 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.0333006382 | fast_rcnn_total_loss:1.0333006382 |
total_loss:2.01346969604 | per_cost_time:0.712218046188s
2018-03-25 22:45:28: step101 image_name:0000000099.jpg |
rpn_loc_loss:0.639244914055 | rpn_cla_loss:0.138194292784 | rpn_total_loss:0.777439236641 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.2477004528 | fast_rcnn_total_loss:1.2477004528 |
total_loss:2.72639989853 | per_cost_time:0.690226078033s
2018-03-25 22:45:36: step111 image_name:0000000147.jpg |
rpn_loc_loss:0.2139043957 | rpn_cla_loss:0.126326650381 | rpn_total_loss:0.340231060982 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.14501595497 | fast_rcnn_total_loss:1.14501595497 |
total_loss:2.18650889397 | per_cost_time:0.711352109909s
2018-03-25 22:45:43: step121 image_name:0000000174.jpg |
rpn_loc_loss:0.138845905662 | rpn_cla_loss:0.144465446472 | rpn_total_loss:0.283311367035 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.48444437981 | fast_rcnn_total_loss:1.48444437981 |
total_loss:2.46901965141 | per_cost_time:0.706383943558s
2018-03-25 22:45:50: step131 image_name:0000000185.jpg |
rpn_loc_loss:0.27352809906 | rpn_cla_loss:0.133923202753 | rpn_total_loss:0.407451301813 |
fast_rcnn_loc_loss:0.0 | fast_rcnn_cla_loss:1.17738771439 | fast_rcnn_total_loss:1.17738771439 |
total_loss:2.28610467911 | per_cost_time:0.718617916107s
2018-03-25 22:45:57: step141 image_name:0000000116.jpg |
rpn_loc_loss:0.170325279236 | rpn_cla_loss:0.0914347320795 | rpn_total_loss:0.261759996414 |
fast_rcnn_loc_loss:0.281328618526 | fast_rcnn_cla_loss:1.30894041061 | fast_rcnn_total_loss:1.59026908875 |
total_loss:2.55329680443 | per_cost_time:0.699913978577s
2018-03-25 22:46:03.734151: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]
2018-03-25 22:46:03.734662: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]
2018-03-25 22:46:03.734703: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]
2018-03-25 22:46:03.734727: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]
2018-03-25 22:46:03.734747: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]
Traceback (most recent call last):
File "/home/choiyeren/Documents/dockerfiels/RRPN_FPN_Tensorflow/tools/train.py", line 274, in
train()
File "/home/choiyeren/Documents/dockerfiels/RRPN_FPN_Tensorflow/tools/train.py", line 234, in train
fast_rcnn_total_loss, total_loss, train_op])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]

Caused by op u'get_batch/batch', defined at:
File "/home/choiyeren/Documents/dockerfiels/RRPN_FPN_Tensorflow/tools/train.py", line 274, in
train()
File "/home/choiyeren/Documents/dockerfiels/RRPN_FPN_Tensorflow/tools/train.py", line 37, in train
is_training=True)
File "/home/choiyeren/Documents/dockerfiels/RRPN_FPN_Tensorflow/data/io/read_tfrecord.py", line 90, in next_batch
dynamic_pad=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py", line 927, in batch
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py", line 722, in _batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 464, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2418, in _queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): PaddingFIFOQueue '_2_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: get_batch/batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_INT32, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](get_batch/batch/padding_fifo_queue, get_batch/batch/n)]]`

Any suggestion will be appreciated.

License

There are quite a few files in https://github.com/yangJirui/RRPN_FPN_Tensorflow/search?q=license which have an Apache-2.0 license header.

Is the entire repo under Apache 2.0?

If so, a LICENSE file would be helpful.

If not, could the README provide some detail about the licenses used by this repository.

How to use text detection network to train a multiclass dataset?where are needed to modify?

meet this error:fast_rcnn_loc_loss:0.0

meet this error:fast_rcnn_loc_loss:0.0.can any one helpp me?
@yangJirui

no rotate_nms_kernel.cu and rbbox_overlaps_kernel.cu files in box_utils

There are no rotate_nms_kernel.cu and rbbox_overlaps_kernel.cu files in box_utils directory, so I can't compile nms_gpu and iou_gpu on Windows. Would you update these files?

OOM when allocating tensor with shape[14,256,128,128]

I meet this error :OOM when allocating tensor with shape[14,256,128,128],look forward to hearing from you

pretrained weights for text

Hi, 你的标题是"Text Detection", 但是readme中result是别的东西，貌似是建筑物。
所以你提供的pretrained_weights 能否用在Text Detection上？

The page of pretrained-weight is not existed

Issue with libs/rpn/build_rpn.py during inference

Hi there,

Thank you for providing this repo. When I execute the inference script, I end up with the error ValueError: Dimension 0 in both shapes must be equal, but are 5 and 4 for 'rpn_proposals/cond_1/concat' (op: 'ConcatV2') with input shapes: [?,5], [?,4], [] and with computed input tensors: input[2] = <0>.

Full traceback:
++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++-- /mnt/cirrus/models/RRPN_FPN_Tensorflow Traceback (most recent call last): File "inference.py", line 165, in <module> inference() File "inference.py", line 87, in inference rpn_proposals_boxes, rpn_proposals_scores = rpn.rpn_proposals() # rpn_score shape: [300, ] File "../libs/rpn/build_rpn.py", line 468, in rpn_proposals lambda: (valid_boxes, valid_scores)) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 289, in new_func return func(*args, **kwargs) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in cond orig_res_t, res_t = context_t.BuildCondBranch(true_fn) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1689, in BuildCondBranch original_result = fn() File "../libs/rpn/build_rpn.py", line 465, in <lambda> self.max_proposals_num), File "../libs/box_utils/boxes_utils.py", line 92, in padd_boxes_with_zeros final_boxes = tf.concat([boxes, zero_boxes], axis=0) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1048, in concat name=name) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 495, in _concat_v2 name=name) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2508, in create_op set_shapes_for_outputs(ret) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1873, in set_shapes_for_outputs shapes = shape_func(op) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1823, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn debug_python_shape_fn, require_shape_fn) File "/mnt/cirrus/venvs/python_27/local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Dimension 0 in both shapes must be equal, but are 5 and 4 for 'rpn_proposals/cond_1/concat' (op: 'ConcatV2') with input shapes: [?,5], [?,4], [] and with computed input tensors: input[2] = <0>.

I am running ubuntu 16.04, cuda 8.0.61, python 2.7, and tensorflow 1.2.0

If you could help me out here, that would be greatly appreciated! Thank you in advance.

Could you support your trained model for us to test?

No nms_rotate and iou_rotate files in box_utils

There are no nms_rotate and iou_rotate files in box_utils, so I can't eval the model, can you update these files?

about category number

Is this could train multi-labels datasets？when I train change the category number over 1 in cfg.py， the train.py has an error。I also change the NAME_LABEL_MAP in label_name_dict/remote_sensing_dict.py, and the tfrecord is generated sucessfully。