测试的时候速度很慢

[thinklab-sjtu]

测试的时候速度很慢

[yangxue0827]

你用的哪个脚本测试的？ @MTQ819

[MTQ819]

用的inference.py测试的

[yangxue0827]

首先你确定一下有没有调用到GPU，还有就是inference中画图也比较耗时，你最好吧cfgs里的内容给我看一下。 @MTQ819

[MTQ819]

我也感觉是不是我的cfg文件哪里写错了，麻烦看一下，谢谢。

-- coding: utf-8 --

from future import division, print_function, absolute_import
import os
import tensorflow as tf
import math

"""
anchor free

cls : ship|| Recall: 0.9535830618892508 || Precison: 0.2536278969027507|| AP: 0.8938264420182536
F1:0.9282455412691829 P:0.9459002535925612 R:0.9112377850162866
mAP is : 0.8938264420182536

"""

------------------------------------------------

VERSION = 'RetinaNet_HRSC2016_R3Det_2x_20200312'
NET_NAME = 'resnet152_v1d' # 'MobilenetV2'
ADD_BOX_IN_TENSORBOARD = True

---------------------------------------- System_config

ROOT_PATH = os.path.abspath('../')
print(20*"++--")
print(ROOT_PATH)
GPU_GROUP = "1"
NUM_GPU = len(GPU_GROUP.strip().split(','))
SHOW_TRAIN_INFO_INTE = 20
SMRY_ITER = 200
SAVE_WEIGHTS_INTE = 1000 * 2

SUMMARY_PATH = ROOT_PATH + '/output/summary'
TEST_SAVE_PATH = ROOT_PATH + '/tools/test_result'

if NET_NAME.startswith("resnet"):
weights_name = NET_NAME
elif NET_NAME.startswith("MobilenetV2"):
weights_name = "mobilenet/mobilenet_v2_1.0_224"
else:
raise Exception('net name must in [resnet_v1_101, resnet_v1_50, MobilenetV2]')

PRETRAINED_CKPT = ROOT_PATH + '/data/pretrained_weights/' + weights_name + '.ckpt'
TRAINED_CKPT = os.path.join(ROOT_PATH, 'output/trained_weights')
EVALUATE_R_DIR = ROOT_PATH + '/output/evaluate_result_pickle/'

------------------------------------------ Train config

RESTORE_FROM_RPN = False
FIXED_BLOCKS = 1 # allow 0~3
FREEZE_BLOCKS = [True, False, False, False, False] # for gluoncv backbone
USE_07_METRIC = True
EVAL_THRESHOLD = 0.5

MUTILPY_BIAS_GRADIENT = 2.0 # if None, will not multipy
GRADIENT_CLIPPING_BY_NORM = 10.0 # if None, will not clip

CLS_WEIGHT = 1.0
REG_WEIGHT = 1.0
USE_IOU_FACTOR = False

BATCH_SIZE = 1
EPSILON = 5e-5
MOMENTUM = 0.9
LR = 1e-4
DECAY_STEP = [SAVE_WEIGHTS_INTE12, SAVE_WEIGHTS_INTE16, SAVE_WEIGHTS_INTE20]
MAX_ITERATION = SAVE_WEIGHTS_INTE20
WARM_SETP = int(1.0 / 4.0 * SAVE_WEIGHTS_INTE)

-------------------------------------------- Data_preprocess_config

DATASET_NAME = 'tingchequyu' # 'pascal', 'coco'
PIXEL_MEAN = [123.68, 116.779, 103.939] # R, G, B. In tf, channel is RGB. In openCV, channel is BGR
PIXEL_MEAN_ = [0.485, 0.456, 0.406]
PIXEL_STD = [0.229, 0.224, 0.225] # R, G, B. In tf, channel is RGB. In openCV, channel is BGR
IMG_SHORT_SIDE_LEN = 800
IMG_MAX_LENGTH = 800
CLASS_NUM = 6

IMG_ROTATE = False
RGB2GRAY = True
VERTICAL_FLIP = True
HORIZONTAL_FLIP = True
IMAGE_PYRAMID = False

--------------------------------------------- Network_config

SUBNETS_WEIGHTS_INITIALIZER = tf.random_normal_initializer(mean=0.0, stddev=0.01, seed=None)
SUBNETS_BIAS_INITIALIZER = tf.constant_initializer(value=0.0)
PROBABILITY = 0.01
FINAL_CONV_BIAS_INITIALIZER = tf.constant_initializer(value=-math.log((1.0 - PROBABILITY) / PROBABILITY))
WEIGHT_DECAY = 1e-4
USE_GN = False
NUM_SUBNET_CONV = 4
NUM_REFINE_STAGE = 2
USE_RELU = False
FPN_CHANNEL = 256

---------------------------------------------Anchor config

LEVEL = ['P3', 'P4', 'P5', 'P6', 'P7']
BASE_ANCHOR_SIZE_LIST = [32, 64, 128, 256, 512]
ANCHOR_STRIDE = [8, 16, 32, 64, 128]
ANCHOR_SCALES = [1.]
ANCHOR_RATIOS = [1.]
ANCHOR_ANGLES = [-90, -75, -60, -45, -30, -15]
ANCHOR_SCALE_FACTORS = None
USE_CENTER_OFFSET = True
METHOD = 'H'
USE_ANGLE_COND = False
ANGLE_RANGE = 90

--------------------------------------------RPN config

SHARE_NET = True
USE_P5 = True
IOU_POSITIVE_THRESHOLD = 0.35
IOU_NEGATIVE_THRESHOLD = 0.25
REFINE_IOU_POSITIVE_THRESHOLD = [0.5, 0.6]
REFINE_IOU_NEGATIVE_THRESHOLD = [0.4, 0.5]

NMS = True
NMS_IOU_THRESHOLD = 0.1
MAXIMUM_DETECTIONS = 100
FILTERED_SCORE = 0.05
VIS_SCORE = 0.5

--------------------------------------------MASK config

USE_SUPERVISED_MASK = False
MASK_TYPE = 'r' # r or h
BINARY_MASK = False
SIGMOID_ON_DOT = False
MASK_ACT_FET = True # weather use mask generate 256 channels to dot feat.
GENERATE_MASK_LIST = ["P3", "P4", "P5", "P6", "P7"]
ADDITION_LAYERS = [4, 4, 3, 2, 2] # add 4 layer to generate P2_mask, 2 layer to generate P3_mask
ENLAEGE_RF_LIST = ["P3", "P4", "P5", "P6", "P7"]
SUPERVISED_MASK_LOSS_WEIGHT = 1.0

[yangxue0827]

你gpu调用到了吗？另外吧inference里画图部分注释了。把你运行的命令也发我一下 @MTQ819

[MTQ819]

GPU调用到了，占了7700MB的显存，但是利用率特别低，不知道什么原因。运行的命令是python3 inference.py --data_dir=/home/maotianqi/detect_network/tingchequyu/trainimages/ --save_dir=/home/maotianqi/detect_network/tingchequyu/results/ - -gpu=1

[MTQ819]

昨天是好的，今天突然就慢了，我再查一查吧

[yangxue0827]

--gpu=1那怎么调用的是0卡

[MTQ819]

打错了，是0来着

[MTQ819]

我仔细查了一下，用相同模型推理时，只有我自己数据集的图片推理时间很长，公开数据集的只要0.1s一张，我的却要20s一张。而且目标越多时间越长。

[MTQ819]

[MTQ819]

我把推理过程中的nms从cpu处理改成gpu处理就好了。

[LegendSun0]

@MTQ819 你好，最近我也在研究这个算法方便加157075073，探讨下么？

[piaoliuping123]

我把推理过程中的nms从cpu处理改成gpu处理就好了。

你好，我预测推理的时候改为gpu，会报错，
2020-04-01 15:07:52.174659: W tensorflow/core/framework/op_kernel.cc:1490] Unknown: IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):

File "/home/matrix/.local/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 209, in call
ret = func(*args)

File "rotate_polygon_nms.pyx", line 22, in rotate_polygon_nms.rotate_gpu_nms (rotate_polygon_nms.cpp:1874)

IndexError: Out of bounds on buffer access (axis 0)，
可以交流一下吗？谢谢

[piaoliuping123]

975966355

[MTQ819]

@piaoliuping123 我有的时候训练出来也会这样，还不知道原因。

[piaoliuping123]

@piaoliuping123 我有的时候训练出来也会这样，还不知道原因。
我就拿原作者的测试也是这样的，inference.py

[Wo-Ki]

我也有同样GPU利用率低的问题。之前采用CUDA10-tf1.14，直接显存使用155MB。采用CUDA9-tf1.12，显存使用了7kMB左右，但是利用率很低，cpu占用率很高，是不是版本问题？必须要cuda8？