文本识别指从图像中识别出文本,将图像中的文字区域转化为字符信息,通常采用CNN网络从图像中提取丰富的特征信息,然后根据提取的特征信息进行识别。这里采用ResNet作为特征提取网络,采用CTC方法进行识别。由于每张样本的字符数量不同,字符样式不同,字体大小也不同,导致每列的输出并不一定能与每个字符一一对应,CTC提出一种不需要对齐的Loss计算方法,用于训练网络。
对于固定长度的输出和可变长度的标签数据,CTC提供了一种计算loss的方法,用于训练网络参数。CTC的关键方法是预测每列的字符并通过删除重复的字符和空白将整个字符序列修改为非固定的字符流。
模型输出的序列H =
,其中$y^t_{\pi_t}$为在每个时间步t生成字符$\pi_t$的概率。
映射函数M,通过删除重复字符和空白,将$\pi$映射到Y。 例如,对于‘aaa-dd-d-c-d---d’,映射函数M将其映射为‘addcdd’,其中‘-’表示空白。
条件概率则定义为所有能够通过映射函数M映射到标签字符序列Y上的定长字符序列$\pi$的概率之和: $$ p(Y|H)= \displaystyle \sum^{ }_{\pi:M(\pi)=Y}{P(\pi|H)} $$
训练的目标是使得这个条件概率取得最大值,而且计算条件概率的过程只有加和乘操作,一定是可导的,故可以得到一个梯度来进行网络模型参数的更新 $$ p(Y|H)=- \displaystyle \sum^{ }_{X_i,Y_i\in TD}{logP(Y_i|X_i)} $$ TD表示训练数据集
在进行测试的时候,在每个时间步,取具有最大概率的字符,来作为预测的标签序列 $$ Y^*\approx M(arg max P(\pi|H)) $$
残差神经网络(ResNet)是由微软研究院的何恺明、张祥雨、任少卿、孙剑等人提出的。ResNet 在2015 年的ILSVRC(ImageNet Large Scale Visual Recognition Challenge)中取得了冠军。残差神经网络的主要贡献是发现了“退化现象(Degradation)”,并针对退化现象发明了 “快捷连接(Shortcut connection)”,极大的消除了深度过大的神经网络训练困难问题。
残差网络结构是ResNet的主要亮点,ResNet使用残差网络结构后可有效地减轻退化问题,实现更深的网络结构设计,提高网络的训练精度
残差网络结构由两个分支构成:一个主分支,一个shortcuts。主分支通过堆叠一系列卷积操作得到,shortcuts从输入直接到输出,主分支得到的输出加上shortcuts得到的输出,通过Relu激活函数后即为残差网络最后的输出。
这里使用的残差网络结构,主分支有两层卷积网络结构:
- 第一层网络,kernel_size为3的卷积层 + BatchNorm + Relu
- 第二层网络,kernel_size为3的卷积层 + BatchNorm
最后将主分支输出的特征矩阵与shortcuts输出的特征矩阵相加,通过Relu激活函数即为残差网络结构的最终输出。
这里的ResNet由5层网络结构组成:
- 第一层,kernel_size为3的卷积层 + BatchNorm + kernel_size为3的卷积层 + BatchNorm + Relu
- 第二层,最大池化 + 残差网络 + kernel_size为3的卷积层 + BatchNorm
- 第三层,最大池化 + 残差网络 + kernel_size为3的卷积层 + BatchNorm
- 第四层,最大池化 + 残差网络 + kernel_size为3的卷积层 + BatchNorm
- 第五层, 残差网络 + kernel_size为2的卷积层 + BatchNorm + kernel_size为2的卷积层 + BatchNorm
为了使得训练的结果更具有说明性,采用了多个数据集进行训练,在训练之前需要将多个数据集合并为一个数据集;同时数据集中每条数据的标签长度不一,为了达到更好的训练效果,希望每次训练的数据的标签长度都一致,在提取数据的时候需要按照一定的索引进行提取。
采用合成数据集进行训练,真实数据集用于测试
MJSynth(MJ)是一个为STR设计的合成数据集,包含8.9M个单词框图像。单词框生成过程如下:1)字体渲染,2)边框和阴影渲染,3)背景着色,4)字体、边框和背景的合成,5)应用投影失真,6)与真实图像混合,7)添加噪声。
SynthText(ST)是另一个合成产生的数据集,最初设计用于场景文本检测。尽管SynthText是为场景文本检测任务设计的,但它也可以通过裁剪单词框用于STR。通过裁剪单词框并过滤为非字母数字字符后,SynthText包含5.5 M个单词框图像。
IIIT5K Words(IIIT)是从Google图像搜索中抓取的数据集,其中包含可能返回文本图像的查询词,例如“广告牌”、“招牌”,“门牌号”、“门牌号”和“电影海报”。IIIT包括2000张用于训练的图像和3000张用于评估的图像。
#合并前
|--- CNNCTC/
|--- cnnctc_data/
|--- ST/
data.mdb
lock.mdb
|--- MJ/
data.mdb
lock.mdb
|--- IIIT/
data.mdb
lock.mdb
#合并后
|--- CNNCTC/
|--- cnnctc_data/
|--- ST/
data.mdb
lock.mdb
|--- MJ/
data.mdb
lock.mdb
|--- IIIT/
data.mdb
lock.mdb
|--- ST_MJ/
data.mdb
lock.mdb
st_mj_fixed_length_index_list.pkl
依次打开数据集的lmdb文件,读取图片和标签,并写入同一个新的lmdb文件中
在一个batch中所有数据的label总长度固定的情况下,得到batch中组成数据的label的长度的可能组合 1、首先得到每条数据中标签的长度,获取数据集中各种标签长度的分布情况,以及数据集中所有标签的总长度 2、根据数据计算每个batch中的标签的平均长度 3、根据batch中的平均长度以及各种标签长度的分布情况,获得可能的组合,使得每个batch的平均长度相同
根据获取的可能组合,形成训练时采集数据的索引列表 注意:这里的num_of_iters应该根据实际训练数据集的大小进行适当调整 在采样过程中,是根据标签长度组合随机选择的,故num_of_iters应该设置稍微大一点,才能基本将数据集中所有数据都采集一遍
实现代码如下:
import lmdb
from tqdm import tqdm
import numpy as np
import random
import pickle
def combine_lmdbs(lmdb_paths, lmdb_save_path):
max_len = int((26 + 1) // 2)
character = '0123456789abcdefghijklmnopqrstuvwxyz'
env_save = lmdb.open(lmdb_save_path,map_size=1099511627776)
cnt = 0
for lmdb_path in lmdb_paths:
env = lmdb.open(lmdb_path, max_readers=32, readonly=True, lock=False, readahead=False, meminit=False)
with env.begin(write=False) as txn:
nSamples = int(txn.get('num-samples'.encode()))
nSamples = nSamples
# Filtering
for index in tqdm(range(nSamples)):
index += 1 # lmdb starts with 1
label_key = 'label-%09d'.encode() % index
label = txn.get(label_key).decode('utf-8')
if len(label) > max_len:
continue
illegal_sample = False
for char_item in label.lower():
if char_item not in character:
illegal_sample = True
break
if illegal_sample:
continue
img_key = 'image-%09d'.encode() % index
imgbuf = txn.get(img_key)
with env_save.begin(write=True) as txn_save:
cnt += 1
label_key_save = 'label-%09d'.encode() % cnt
label_save = label.encode()
image_key_save = 'image-%09d'.encode() % cnt
image_save = imgbuf
txn_save.put(label_key_save, label_save)
txn_save.put(image_key_save, image_save)
nSamples = cnt
with env_save.begin(write=True) as txn_save:
txn_save.put('num-samples'.encode(), str(nSamples).encode())
#在一个batch中所有数据的label总长度固定的情况下,batch中组成数据的label的长度的可能组合
#首先得到每条数据中标签的长度,获取数据集中各种标签长度的分布情况,以及数据集中所有标签的总长度
#获取每个batch中的平均长度
#根据batch中的平均长度以及各种标签长度的分布情况,获得可能的组合,使得每个batch的平均长度相同
def analyze_lmdb_label_length(lmdb_path, batch_size=64, num_of_combinations=1000):
label_length_dict = {}
env = lmdb.open(lmdb_path, max_readers=32, readonly=True, lock=False, readahead=False, meminit=False)
with env.begin(write=False) as txn:
nSamples = int(txn.get('num-samples'.encode()))
nSamples = nSamples
for index in tqdm(range(nSamples)):
index += 1 # lmdb starts with 1
label_key = 'label-%09d'.encode() % index
label = txn.get(label_key).decode('utf-8')
label_length = len(label)
if label_length in label_length_dict:
label_length_dict[label_length] += 1
else:
label_length_dict[label_length] = 1
sorted_label_length = sorted(label_length_dict.items(), key=lambda x: x[1], reverse=True)
label_length_sum = 0
label_num = 0
lengths = []
p = []
for l, num in sorted_label_length:
label_length_sum += l * num
label_num += num
p.append(num)
lengths.append(l)
for i, _ in enumerate(p):
p[i] /= label_num
average_overall_length = int(label_length_sum / label_num * batch_size)
def get_combinations_of_fix_length(fix_length, items, p, batch_size):
ret = []
cur_sum = 0
ret = np.random.choice(items, batch_size - 1, True, p)
cur_sum = sum(ret)
ret = list(ret)
if fix_length - cur_sum in items:
ret.append(fix_length - cur_sum)
else:
return None
return ret
result = []
while len(result) < num_of_combinations:
ret = get_combinations_of_fix_length(average_overall_length, lengths, p, batch_size)
if ret is not None:
result.append(ret)
return result
#根据获取的可能组合,形成训练时采集数据的索引列表
#注意:这里的num_of_iters应该根据实际训练数据集的大小进行适当调整
#在采样过程中,是根据标签长度组合随机选择的,故num_of_iters应该设置稍微大一点,才能基本将数据集中所有数据都采集一遍
def generate_fix_shape_index_list(lmdb_path, combinations, pkl_save_path, num_of_iters=650):
length_index_dict = {}
env = lmdb.open(lmdb_path, max_readers=32, readonly=True, lock=False, readahead=False, meminit=False)
with env.begin(write=False) as txn:
nSamples = int(txn.get('num-samples'.encode()))
nSamples = nSamples
for index in tqdm(range(nSamples)):
index += 1 # lmdb starts with 1
label_key = 'label-%09d'.encode() % index
label = txn.get(label_key).decode('utf-8')
label_length = len(label)
if label_length in length_index_dict:
length_index_dict[label_length].append(index)
else:
length_index_dict[label_length] = [index]
ret = []
for _ in range(num_of_iters):
comb = random.choice(combinations)
for l in comb:
ret.append(random.choice(length_index_dict[l]))
with open(pkl_save_path, 'wb') as f:
pickle.dump(ret, f, -1)
combine_lmdbs(['cnnctc_data/ST/', 'cnnctc_data/MJ/'], 'cnnctc_data/ST_MJ')
combination = analyze_lmdb_label_length("cnnctc_data/ST_MJ")
generate_fix_shape_index_list('cnnctc_data/ST_MJ', combination,'cnnctc_data/st_mj_fixed_length_index_list.pkl')
100%|██████████| 19996/19996 [01:01<00:00, 322.90it/s]
100%|██████████| 19554/19554 [01:05<00:00, 297.23it/s]
100%|██████████| 39550/39550 [00:00<00:00, 118154.71it/s]
100%|██████████| 39550/39550 [00:00<00:00, 376512.71it/s]
经过上面数据集合并后,数据是以图片和文本标签的形式放在lmdb文件中的,在模型训练中,我们需要将图片和标签以tensor向量进行表示 首先,从lmdb文件中读取图片和标签;然后对图片进行归一化处理,resize为相同的size,再把图像转为tensor,把文本标签转为编码形式。
通过定义了 ST_MJ_Generator_batch_fixed_length类(Data_loader类)来提取数据,在该类的__init__函数中,设置图片归一化处理函数,文本标签编码函数,分布式处理设置,batch_size大小。在该类的__getitem__函数中,通过传入索引值读取训练集中的样本和标签图像,并对图像进行对应的数据处理操作,就完成了__getitem__函数对样本和标签图像的读取。在函数中实例化ST_MJ_Generator_batch_fixed_length类获取st_dataset,也就是训练集或验证集对应路径下的样本和标签元组对,再通过mindspore.dataset中的GeneratorDataset将元组转换为Tensor,最后通过设定好的batch_size将样本和标签按照batch_size大小分组,由此完成数据集的创建,上述流程对应代码如下:
import six
import lmdb
import numpy as np
from PIL import Image
import argparse
import random
import pickle
import lmdb
import sys
import math
import six
import os
import time
import numpy as np
from tqdm import tqdm
from PIL import Image
import mindspore
import mindspore.nn as nn
import mindspore.common.dtype as mstype
from mindspore.ops import operations as P
from mindspore.ops import composite as C
from mindspore.ops import functional as F
from mindspore.common.initializer import TruncatedNormal, initializer
from mindspore.common import set_seed
from mindspore.communication.management import init, get_rank, get_group_size
from mindspore.dataset import GeneratorDataset
from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, TimeMonitor, Callback
from mindspore.train.model import Model
from mindspore.train.serialization import load_checkpoint, load_param_into_net
from mindspore import Tensor, Parameter, ParameterTuple, context
from mindspore.context import ParallelMode
from mindspore.nn.wrap.grad_reducer import DistributedGradReducer
#从lmdb文件中读取图片和标签
def get_img_from_lmdb(env, index):
with env.begin(write=False) as txn:
label_key = 'label-%09d'.encode() % index
label = txn.get(label_key).decode('utf-8')
img_key = 'image-%09d'.encode() % index
imgbuf = txn.get(img_key)
buf = six.BytesIO()
buf.write(imgbuf)
buf.seek(0)
try:
img = Image.open(buf).convert('RGB') # for color image
except IOError:
print(f'Corrupted image for {index}')
# make dummy image and dummy label for corrupted image.
img = Image.new('RGB', (config.IMG_W, config.IMG_H))
label = '[dummy_label]'
label = label.lower()
return img, label
#对图片进行归一化
class NormalizePAD():
def __init__(self, max_size, PAD_type='right'):
self.max_size = max_size
self.PAD_type = PAD_type
def __call__(self, img):
# toTensor
img = np.array(img, dtype=np.float32)
# normalize
means = [121.58949, 123.93914, 123.418655]
stds = [65.70353, 65.142426, 68.61079]
img = np.subtract(img, means)
img = np.true_divide(img, stds)
img = img.transpose([2, 0, 1])
img = img.astype(float)
_, _, w = img.shape
Pad_img = np.zeros(shape=self.max_size, dtype=np.float32)
Pad_img[:, :, :w] = img # right pad
if self.max_size[2] != w: # add border Pad
Pad_img[:, :, w:] = np.tile(np.expand_dims(img[:, :, w - 1], 2), (1, 1, self.max_size[2] - w))
return Pad_img
#图片resize后转为统一shape的tensor
class AlignCollate():
def __init__(self, imgH=32, imgW=100):
self.imgH = imgH
self.imgW = imgW
def __call__(self, images):
resized_max_w = self.imgW
input_channel = 3
transform = NormalizePAD((input_channel, self.imgH, resized_max_w))
resized_images = []
for image in images:
w, h = image.size
ratio = w / float(h)
if math.ceil(self.imgH * ratio) > self.imgW:
resized_w = self.imgW
else:
resized_w = math.ceil(self.imgH * ratio)
resized_image = image.resize((resized_w, self.imgH), Image.BICUBIC)
resized_images.append(transform(resized_image))
image_tensors = np.concatenate([np.expand_dims(t, 0) for t in resized_images], 0)
return image_tensors
#文本标签到索引标签之间的转换
#将文本转换为数字编码
#CHARACTER: "0123456789abcdefghijklmnopqrstuvwxyz"
#则‘hate’----->[18,11,30,15]
class CTCLabelConverter():
""" Convert between text-label and text-index """
def __init__(self, character):
# character (str): set of the possible characters.
dict_character = list(character)
self.dict = {}
for i, char in enumerate(dict_character):
self.dict[char] = i
self.character = dict_character + ['[blank]'] # dummy '[blank]' token for CTCLoss (index 0)
self.dict['[blank]'] = len(dict_character)
def encode(self, text):
"""convert text-label into text-index.
input:
text: text labels of each image. [batch_size]
output:
text: concatenated text index for CTCLoss.
[sum(text_lengths)] = [text_index_0 + text_index_1 + ... + text_index_(n - 1)]
length: length of each text. [batch_size]
"""
length = [len(s) for s in text]
text = ''.join(text)
text = [self.dict[char] for char in text]
return np.array(text), np.array(length)
def decode(self, text_index, length):
""" convert text-index into text-label. """
texts = []
index = 0
for l in length:
t = text_index[index:index + l]
char_list = []
for i in range(l):
# if t[i] != self.dict['[blank]'] and (not (i > 0 and t[i - 1] == t[i])): # removing repeated characters and blank.
if t[i] != self.dict['[blank]'] and (
not (i > 0 and t[i - 1] == t[i])): # removing repeated characters and blank.
char_list.append(self.character[t[i]])
text = ''.join(char_list)
texts.append(text)
index += l
return texts
def reverse_encode(self, text_index, length):
""" convert text-index into text-label. """
texts = []
index = 0
for l in length:
t = text_index[index:index + l]
char_list = []
for i in range(l):
if t[i] != self.dict['[blank]']: # removing repeated characters and blank.
char_list.append(self.character[t[i]])
text = ''.join(char_list)
texts.append(text)
index += l
return texts
#训练数据集封装
class ST_MJ_Generator_batch_fixed_length:
def __init__(self):
self.align_collector = AlignCollate()
self.converter = CTCLabelConverter(config.CHARACTER)
self.env = lmdb.open(config.TRAIN_DATASET_PATH, max_readers=32, readonly=True, lock=False, readahead=False,
meminit=False)
if not self.env:
print('cannot create lmdb from %s' % (config.TRAIN_DATASET_PATH))
raise ValueError(config.TRAIN_DATASET_PATH)
with open(config.TRAIN_DATASET_INDEX_PATH, 'rb') as f:
self.st_mj_filtered_index_list = pickle.load(f)
print(f'num of samples in ST_MJ dataset: {len(self.st_mj_filtered_index_list)}')
if config.run_distribute:
self.rank_id = get_rank()
self.rank_size = get_group_size()
self.dataset_size = len(self.st_mj_filtered_index_list) // config.TRAIN_BATCH_SIZE // self.rank_size
else:
self.dataset_size = len(self.st_mj_filtered_index_list) // config.TRAIN_BATCH_SIZE
self.batch_size = config.TRAIN_BATCH_SIZE
def __len__(self):
return self.dataset_size
def __getitem__(self, item):
img_ret = []
text_ret = []
if config.run_distribute:
rank_item = (item * self.rank_size) + self.rank_id
else:
rank_item = item
for i in range(rank_item * self.batch_size, (rank_item + 1) * self.batch_size):
index = self.st_mj_filtered_index_list[i]
img, label = get_img_from_lmdb(self.env, index)
img_ret.append(img)
text_ret.append(label)
img_ret = self.align_collector(img_ret)
text_ret, length = self.converter.encode(text_ret)
label_indices = []
for i, _ in enumerate(length):
for j in range(length[i]):
label_indices.append((i, j))
label_indices = np.array(label_indices, np.int64)
sequence_length = np.array([config.FINAL_FEATURE_WIDTH] * config.TRAIN_BATCH_SIZE, dtype=np.int32)
text_ret = text_ret.astype(np.int32)
return img_ret, label_indices, text_ret, sequence_length
#对于验证数据集就无需使得提取的数据的标签具有相同的长度
#验证数据集封装
def IIIT_Generator_batch():
max_len = int((26 + 1) // 2)
align_collector = AlignCollate()
converter = CTCLabelConverter(config.CHARACTER)
env = lmdb.open(config.TEST_DATASET_PATH, max_readers=32, readonly=True, lock=False, readahead=False, meminit=False)
if not env:
print('cannot create lmdb from %s' % (config.TEST_DATASET_PATH))
sys.exit(0)
with env.begin(write=False) as txn:
nSamples = int(txn.get('num-samples'.encode()))
nSamples = nSamples
# Filtering
filtered_index_list = []
for index in range(nSamples):
index += 1 # lmdb starts with 1
label_key = 'label-%09d'.encode() % index
label = txn.get(label_key).decode('utf-8')
if len(label) > max_len:
continue
illegal_sample = False
for char_item in label.lower():
if char_item not in config.CHARACTER:
illegal_sample = True
break
if illegal_sample:
continue
filtered_index_list.append(index)
img_ret = []
text_ret = []
print(f'num of samples in IIIT dataset: {len(filtered_index_list)}')
for index in filtered_index_list:
img, label = get_img_from_lmdb(env, index)
img_ret.append(img)
text_ret.append(label)
if len(img_ret) == config.TEST_BATCH_SIZE:
img_ret = align_collector(img_ret)
text_ret, length = converter.encode(text_ret)
label_indices = []
for i, _ in enumerate(length):
for j in range(length[i]):
label_indices.append((i, j))
label_indices = np.array(label_indices, np.int64)
sequence_length = np.array([26] * config.TEST_BATCH_SIZE, dtype=np.int32)
text_ret = text_ret.astype(np.int32)
yield img_ret, label_indices, text_ret, sequence_length, length
img_ret = []
text_ret = []
class CNNCTC_Model(nn.Cell):
def __init__(self, num_class, hidden_size, final_feature_width):
super(CNNCTC_Model, self).__init__()
self.num_class = num_class
self.hidden_size = hidden_size
self.final_feature_width = final_feature_width
self.FeatureExtraction = ResNet(3, 512, BasicBlock, [1, 2, 5, 3])
self.Prediction = nn.Dense(self.hidden_size, self.num_class)
self.transpose = P.Transpose()
self.reshape = P.Reshape()
def construct(self, x):
x = self.FeatureExtraction(x)
#进行维度转换,方便划分图片时按列划分
x = self.transpose(x, (0, 3, 1, 2)) # [b, c, h, w] -> [b, w, c, h]
x = self.reshape(x, (-1, self.hidden_size))
x = self.Prediction(x)
x = self.reshape(x, (-1, self.final_feature_width, self.num_class))
return x
class ResNet(nn.Cell):
def __init__(self, input_channel, output_channel, block, layers):
super(ResNet, self).__init__()
self.output_channel_block = [int(output_channel / 4), int(output_channel / 2), output_channel, output_channel]
self.inplanes = int(output_channel / 8)
self.conv0_1 = nn.Conv2d(input_channel, int(output_channel / 16), kernel_size=3, stride=1, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn0_1 = nn.BatchNorm2d(int(output_channel / 16), momentum=0.1)
self.conv0_2 = nn.Conv2d(int(output_channel / 16), self.inplanes, kernel_size=3, stride=1, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn0_2 = nn.BatchNorm2d(self.inplanes, momentum=0.1)
self.relu = P.ReLU()
self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2, pad_mode='valid')
self.layer1 = self._make_layer(block, self.output_channel_block[0], layers[0])
self.conv1 = nn.Conv2d(self.output_channel_block[0], self.output_channel_block[0], kernel_size=3, stride=1, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn1 = nn.BatchNorm2d(self.output_channel_block[0], momentum=0.1)
self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2, pad_mode='valid')
self.layer2 = self._make_layer(block, self.output_channel_block[1], layers[1])
self.conv2 = nn.Conv2d(self.output_channel_block[1], self.output_channel_block[1], kernel_size=3, stride=1, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn2 = nn.BatchNorm2d(self.output_channel_block[1], momentum=0.1)
self.pad = P.Pad(((0, 0), (0, 0), (0, 0), (2, 2)))
self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=(2, 1), pad_mode='valid')
self.layer3 = self._make_layer(block, self.output_channel_block[2], layers[2])
self.conv3 = nn.Conv2d(self.output_channel_block[2], self.output_channel_block[2], kernel_size=3, stride=1, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn3 = nn.BatchNorm2d(self.output_channel_block[2], momentum=0.1)
self.layer4 = self._make_layer(block, self.output_channel_block[3], layers[3])
self.conv4_1 = nn.Conv2d(self.output_channel_block[3], self.output_channel_block[3], kernel_size=2, stride=(2, 1), padding=0, pad_mode='valid', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn4_1 = nn.BatchNorm2d(self.output_channel_block[3], momentum=0.1)
self.conv4_2 = nn.Conv2d(self.output_channel_block[3], self.output_channel_block[3], kernel_size=2, stride=1, padding=0, pad_mode='valid', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn4_2 = nn.BatchNorm2d(self.output_channel_block[3], momentum=0.1)
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.SequentialCell(
[nn.Conv2d(self.inplanes, planes * block.expansion, kernel_size=1, stride=stride, padding=0, pad_mode='same', weight_init=TruncatedNormal(0.02), has_bias=False),
nn.BatchNorm2d(planes * block.expansion, momentum=0.1)]
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for _ in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.SequentialCell(layers)
def construct(self, x):
x = self.conv0_1(x)
x = self.bn0_1(x)
x = self.relu(x)
x = self.conv0_2(x)
x = self.bn0_2(x)
x = self.relu(x)
x = self.maxpool1(x)
x = self.layer1(x)
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.maxpool2(x)
x = self.layer2(x)
x = self.conv2(x)
x = self.bn2(x)
x = self.relu(x)
x = self.maxpool3(x)
x = self.layer3(x)
x = self.conv3(x)
x = self.bn3(x)
x = self.relu(x)
x = self.layer4(x)
x = self.pad(x)
x = self.conv4_1(x)
x = self.bn4_1(x)
x = self.relu(x)
x = self.conv4_2(x)
x = self.bn4_2(x)
x = self.relu(x)
return x
class BasicBlock(nn.Cell):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=3, stride=stride, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn1 = nn.BatchNorm2d(planes, momentum=0.1)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, pad_mode='pad', weight_init=TruncatedNormal(0.02), has_bias=False)
self.bn2 = nn.BatchNorm2d(planes, momentum=0.1)
self.relu = P.ReLU()
self.downsample = downsample
self.add = P.Add()
def construct(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
residual = self.downsample(x)
out = self.add(out, residual)
out = self.relu(out)
return out
这里采用mindspore已经实现的CTCloss函数接口
from mindspore.ops import operations as P
import mindspore.nn as nn
#loss设计
class ctc_loss(nn.Cell):
def __init__(self):
super(ctc_loss, self).__init__()
self.loss = P.CTCLoss(preprocess_collapse_repeated=False,
ctc_merge_repeated=True,
ignore_longer_outputs_than_inputs=False)
self.mean = P.ReduceMean()
self.transpose = P.Transpose()
self.reshape = P.Reshape()
def construct(self, inputs, labels_indices, labels_values, sequence_length):
inputs = self.transpose(inputs, (1, 0, 2))
loss, _ = self.loss(inputs, labels_indices, labels_values, sequence_length)
loss = self.mean(loss)
return loss
根据训练的总的step数,设置学习率 在warmup_steps以内的step,学习率线性改变 在warmup_steps之后的step,学习率改变符合余弦分布
import math
def dynamic_lr(config, steps_per_epoch):
"""dynamic learning rate generator"""
base_lr = config.base_lr
total_steps = steps_per_epoch * config.TRAIN_EPOCHS
warmup_steps = int(config.warmup_step)
decay_steps = total_steps - warmup_steps
lr = []
for i in range(total_steps):
if i < warmup_steps:
lr_inc = (float(base_lr) - float(base_lr * config.warmup_ratio)) / float(warmup_steps)
learning_rate = float(base_lr * config.warmup_ratio) + lr_inc * i
lr.append(learning_rate)
else:
base = float(i - warmup_steps) / float(decay_steps)
learning_rate = (1 + math.cos(base * math.pi)) / 2 * base_lr
lr.append(learning_rate )
return lr
在GPU训练中,为处理梯度溢出的现象,将梯度裁剪和模型训练进行组合封装
from mindspore.ops import operations as P
from mindspore.ops import composite as C
#封装model与loss函数
class WithLossCell(nn.Cell):
def __init__(self, backbone, loss_fn):
super(WithLossCell, self).__init__(auto_prefix=False)
self._backbone = backbone
self._loss_fn = loss_fn
def construct(self, img, label_indices, text, sequence_length):
model_predict = self._backbone(img)
return self._loss_fn(model_predict, label_indices, text, sequence_length)
@property
def backbone_network(self):
return self._backbone
#梯度裁剪,使用L2范数控制梯度
class ClipGradients(nn.Cell):
"""
Clip large gradients, typically generated from overflow.
"""
def __init__(self):
super(ClipGradients, self).__init__()
self.clip_by_norm = nn.ClipByNorm()
self.cast = P.Cast()
self.dtype = P.DType()
def construct(self, grads, clip_min, clip_max):
new_grads = ()
for grad in grads:
dt = self.dtype(grad)
t = C.clip_by_value(grad, self.cast(F.tuple_to_array((clip_min,)), dt),
self.cast(F.tuple_to_array((clip_max,)), dt))
t = self.cast(t, dt)
new_grads = new_grads + (t,)
return new_grads
grad_scale = C.MultitypeFuncGraph("grad_scale")
reciprocal = P.Reciprocal()
GRADIENT_CLIP_MIN = -64000
GRADIENT_CLIP_MAX = 64000
@grad_scale.register("Tensor", "Tensor")
def tensor_grad_scale(scale, grad):
return grad * F.cast(reciprocal(scale), F.dtype(grad))
class CNNCTCTrainOneStepWithLossScaleCell(nn.Cell):
"""
Encapsulation class of CNNCTC network training.
Used for GPU training in order to manage overflowing gradients.
Args:
network (Cell): The training network. Note that loss function should have been added.
optimizer (Optimizer): Optimizer for updating the weights.
scale_sense (Cell): Loss scaling value.
"""
def __init__(self, network, optimizer, scale_sense):
super(CNNCTCTrainOneStepWithLossScaleCell, self).__init__(auto_prefix=False)
self.network = network
self.optimizer = optimizer
if isinstance(scale_sense, nn.Cell):
self.loss_scaling_manager = scale_sense
self.scale_sense = Parameter(Tensor(scale_sense.get_loss_scale(),
dtype=mstype.float32), name="scale_sense")
elif isinstance(scale_sense, Tensor):
if scale_sense.shape == (1,) or scale_sense.shape == ():
self.scale_sense = Parameter(scale_sense, name='scale_sense')
else:
raise ValueError("The shape of scale_sense must be (1,) or (), but got {}".format(
scale_sense.shape))
else:
raise TypeError("The scale_sense must be Cell or Tensor, but got {}".format(
type(scale_sense)))
self.network.set_grad()
self.weights = ParameterTuple(network.trainable_params())
self.grad = C.GradOperation(get_by_list=True,
sens_param=True)
self.reducer_flag = False
self.parallel_mode = context.get_auto_parallel_context("parallel_mode")
if self.parallel_mode not in ParallelMode.MODE_LIST:
raise ValueError("Parallel mode does not support: ", self.parallel_mode)
if self.parallel_mode in [ParallelMode.DATA_PARALLEL, ParallelMode.HYBRID_PARALLEL]:
self.reducer_flag = True
self.grad_reducer = None
if self.reducer_flag:
mean = context.get_auto_parallel_context("gradients_mean")
degree = get_group_size()
self.grad_reducer = DistributedGradReducer(optimizer.parameters, mean, degree)
self.is_distributed = (self.parallel_mode != ParallelMode.STAND_ALONE)
self.clip_gradients = ClipGradients()
self.cast = P.Cast()
self.addn = P.AddN()
self.reshape = P.Reshape()
self.hyper_map = C.HyperMap()
self.less_equal = P.LessEqual()
self.allreduce = P.AllReduce()
#grad_scale = C.MultitypeFuncGraph("grad_scale")
#reciprocal = P.Reciprocal()
def construct(self, img, label_indices, text, sequence_length):
weights = self.weights
loss = self.network(img, label_indices, text, sequence_length)
scaling_sens = self.scale_sense
grads = self.grad(self.network, weights)(img, label_indices, text, sequence_length,
self.cast(scaling_sens, mstype.float32))
grads = self.hyper_map(F.partial(grad_scale, scaling_sens), grads)
grads = self.clip_gradients(grads, -64000, 64000)
if self.reducer_flag:
#apply grad reducer on grads
grads = self.grad_reducer(grads)
self.optimizer(grads)
return (loss, scaling_sens)
利用MindSpore框架,基于Callback自定义了回调类LossCallBack,为了满足在模型训练时,能够在每一次epoch结束后计算训练一个epoch的时间,以及每个step的平均耗时
from mindspore.train.callback import Callback
import time
class AverageMeter():
"""Computes and stores the average and current value"""
def __init__(self):
self.reset()
def reset(self):
self.val = 0
self.avg = 0
self.sum = 0
self.count = 0
def update(self, val, n=1):
self.val = val
self.sum += val * n
self.count += n
self.avg = self.sum / self.count
#自定义loss回调函数
class LossCallBack(Callback):
"""
Monitor the loss in training.
If the loss is NAN or INF terminating training.
Note:
If per_print_times is 0 do not print loss.
Args:
per_print_times (int): Print loss every times. Default: 1.
"""
def __init__(self, per_print_times=1):
super(LossCallBack, self).__init__()
if not isinstance(per_print_times, int) or per_print_times < 0:
raise ValueError("print_step must be int and >= 0.")
self._per_print_times = per_print_times
self.loss_avg = AverageMeter()
self.timer = AverageMeter()
self.start_time = time.time()
def step_end(self, run_context):
cb_params = run_context.original_args()
loss = np.array(cb_params.net_outputs)
cur_step_in_epoch = (cb_params.cur_step_num - 1) % cb_params.batch_num + 1
cur_num = cb_params.cur_step_num
if cur_step_in_epoch % 2000 == 1:
self.loss_avg = AverageMeter()
self.timer = AverageMeter()
self.start_time = time.time()
else:
self.timer.update(time.time() - self.start_time)
self.start_time = time.time()
self.loss_avg.update(loss)
if self._per_print_times != 0 and cur_num % self._per_print_times == 0:
loss_file = open("./loss.log", "a+")
loss_file.write("epoch: %d step: %d , loss is %s" % (
cb_params.cur_epoch_num, cur_step_in_epoch,
self.loss_avg.avg))
loss_file.write("\n")
loss_file.close()
定义训练函数train()和测试函数test(),通过config传递参数;使用前面模块定义的ST_MJ_Generator_batch_fixed_length函数提取训练数据集,并使用mindspore中的GeneratorDataset函数进行封装,使用前面模块定义的CNNCTC_Model类作为基本网络,根据需求可选择运行环境和是否分布式运行。 在进行测试时,用前面模块定义的IIIT_Generator_batch函数提取数据。
#train过程
import mindspore
from mindspore.dataset import GeneratorDataset
from mindspore.train.serialization import load_checkpoint, load_param_into_net
from mindspore import Tensor
from mindspore.train.model import Model
from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, TimeMonitor
import numpy as np
import time
import os
def train(config):
target = config.device_target
mindspore.set_context(device_target=target)
if target == "Ascend":
device_id1 = int(os.getenv('DEVICE_ID', '0'))
#mindspore.set_context(device_target=target,device_id=device_id1)
if config.run_distribute:
init()
context.set_auto_parallel_context(parallel_mode="data_parallel")
ckpt_save_dir = config.SAVE_PATH
else:
device_id1 = int(os.getenv('DEVICE_ID', '0'))
mindspore.set_context(device_id=device_id1)
if config.run_distribute:
init()
context.set_auto_parallel_context(device_num=get_group_size(), parallel_mode="data_parallel", gradients_mean=False, gradient_fp32_sync=False)
ckpt_save_dir = config.SAVE_PATH + "ckpt_" + str(get_rank()) + "/"
print(ckpt_save_dir)
else:
ckpt_save_dir = config.SAVE_PATH + "ckpt_gpu_1p/"
st_dataset = ST_MJ_Generator_batch_fixed_length()
ds = GeneratorDataset(st_dataset, ['img', 'label_indices', 'text', 'sequence_length'], num_parallel_workers=4)
net = CNNCTC_Model(config.NUM_CLASS, config.HIDDEN_SIZE, config.FINAL_FEATURE_WIDTH)
net.set_train(True)
if config.PRED_TRAINED:
param_dict = load_checkpoint(config.PRED_TRAINED)
load_param_into_net(net, param_dict)
print('parameters loaded!')
else:
print('train from scratch...')
criterion = ctc_loss()
print("-----------------ctc_loss over-----------------")
dataset_size = ds.get_dataset_size()
lr = Tensor(dynamic_lr(config, dataset_size), mstype.float32)
print("-----------------lr over-----------------")
opt = mindspore.nn.RMSProp(params=net.trainable_params(),
centered=True,
learning_rate=lr,
momentum=config.MOMENTUM,
loss_scale=config.LOSS_SCALE)
net = WithLossCell(net, criterion)
print("-----------------with losscell over-----------------")
if target == "Ascend":
loss_scale_manager = mindspore.train.loss_scale_manager.FixedLossScaleManager(
config.LOSS_SCALE, False)
net.set_train(True)
model = Model(net, optimizer=opt, loss_scale_manager=loss_scale_manager, amp_level="O2")
else:
scaling_sens = Tensor(np.full((1), config.LOSS_SCALE), dtype=mstype.float32)
net = CNNCTCTrainOneStepWithLossScaleCell(net, opt, scaling_sens)
print("-----------------onestepwithlossscale over-----------------")
net.set_train(True)
model = Model(net)
loss_cb = LossCallBack()
time_cb = TimeMonitor(data_size=dataset_size)
config_ck = CheckpointConfig(save_checkpoint_steps=config.SAVE_CKPT_PER_N_STEP,
keep_checkpoint_max=config.KEEP_CKPT_MAX_NUM)
ckpoint_cb = ModelCheckpoint(prefix="CNNCTC", config=config_ck, directory=ckpt_save_dir)
callbacks = [loss_cb, time_cb, ckpoint_cb]
if config.run_distribute:
if device_id == 0:
model.train(config.TRAIN_EPOCHS,
ds,
callbacks=callbacks,
dataset_sink_mode=False)
else:
callbacks.remove(ckpoint_cb)
model.train(config.TRAIN_EPOCHS, ds, callbacks=callbacks, dataset_sink_mode=False)
else:
model.train(config.TRAIN_EPOCHS,
ds,
callbacks=callbacks,
dataset_sink_mode=False)
#验证过程
def test(config):
target = config.device_target
mindspore.set_context(device_target=target)
ds = GeneratorDataset(IIIT_Generator_batch, ['img', 'label_indices', 'text',
'sequence_length', 'label_str'])
net = CNNCTC_Model(config.NUM_CLASS, config.HIDDEN_SIZE, config.FINAL_FEATURE_WIDTH)
ckpt_path = config.CHECKPOINT_PATH
param_dict = load_checkpoint(ckpt_path)
load_param_into_net(net, param_dict)
print('parameters loaded! from: ', ckpt_path)
converter = CTCLabelConverter(config.CHARACTER)
model_run_time = AverageMeter()
npu_to_cpu_time = AverageMeter()
postprocess_time = AverageMeter()
count = 0
correct_count = 0
for data in ds.create_tuple_iterator():
img, _, text, _, length = data
img_tensor = Tensor(img, mstype.float32)
model_run_begin = time.time()
model_predict = net(img_tensor)
model_run_end = time.time()
model_run_time.update(model_run_end - model_run_begin)
npu_to_cpu_begin = time.time()
model_predict = np.squeeze(model_predict.asnumpy())
npu_to_cpu_end = time.time()
npu_to_cpu_time.update(npu_to_cpu_end - npu_to_cpu_begin)
postprocess_begin = time.time()
preds_size = np.array([model_predict.shape[1]] * config.TEST_BATCH_SIZE)
preds_index = np.argmax(model_predict, 2)
preds_index = np.reshape(preds_index, [-1])
preds_str = converter.decode(preds_index, preds_size)
postprocess_end = time.time()
postprocess_time.update(postprocess_end - postprocess_begin)
label_str = converter.reverse_encode(text.asnumpy(), length.asnumpy())
if count == 0:
model_run_time.reset()
npu_to_cpu_time.reset()
postprocess_time.reset()
else:
print('---------model run time--------', model_run_time.avg)
print('---------npu_to_cpu run time--------', npu_to_cpu_time.avg)
print('---------postprocess run time--------', postprocess_time.avg)
print("Prediction samples: \n", preds_str[:5])
print("Ground truth: \n", label_str[:5])
for pred, label in zip(preds_str, label_str):
if pred == label:
correct_count += 1
count += 1
print('accuracy: ', correct_count / count)
模型训练时设置的batch_size大小最好与数据集处理时设置的batch_size大小一致,这样才能保证,提取的每一个batch的数据的所有标签总长度基本一致,这里将batch_size设置为64.
使用Model()将优化器与网络结合,对于Ascend环境,mindspore定义好了梯度裁剪功能,但对于GPU环境,需要自己定义梯度裁剪,CNNCTCTrainOneStepWithLossScaleCell()
import argparse
parser = argparse.ArgumentParser(description='default name', add_help=False)
parser.add_argument('--device_target', type=str, default='Ascend', help='Ascend\GPU\CPU')
parser.add_argument('--CHARACTER', type=str, default="0123456789abcdefghijklmnopqrstuvwxyz", help=' ')
parser.add_argument('--NUM_CLASS', type=int, default=37, help=' ')
parser.add_argument('--HIDDEN_SIZE', type=int, default=512, help=' ')
parser.add_argument('--FINAL_FEATURE_WIDTH', type=int, default=26, help=' ')
parser.add_argument('--IMG_H', type=int, default=32, help=' ')
parser.add_argument('--IMG_W', type=int, default=100, help=' ')
parser.add_argument('--TRAIN_BATCH_SIZE', type=int, default=64, help=' ')
parser.add_argument('--TEST_BATCH_SIZE', type=int, default=64, help=' ')
parser.add_argument('--TRAIN_EPOCHS', type=int, default=3, help=' ')
parser.add_argument('--TRAIN_DATASET_PATH', type=str, default="cnnctc_data/ST_MJ/", help=' ')
parser.add_argument('--TRAIN_DATASET_INDEX_PATH', type=str, default="cnnctc_data/st_mj_fixed_length_index_list.pkl", help=' ')
parser.add_argument('--TEST_DATASET_PATH', type=str, default="cnnctc_data/IIIT/", help=' ')
parser.add_argument('--CHECKPOINT_PATH', type=str, default="/home/wch/cnnctc_wch/ckpt_gpu_1p/CNNCTC-3_650.ckpt", help=' ')
parser.add_argument('--PRED_TRAINED', type=str, default='', help='pred training')
parser.add_argument('--SAVE_PATH', type=str, default='./ckpt_ascend', help='saving the ckpl')
parser.add_argument('--run_distribute', type=bool, default=False, help=' ')
parser.add_argument('--base_lr', type=float, default=0.0005, help=' ')
parser.add_argument('--warmup_ratio', type=float, default=0.0625, help=' ')
parser.add_argument('--MOMENTUM', type=float, default=0.8, help=' ')
parser.add_argument('--warmup_step', type=int, default=2000, help=' ')
parser.add_argument('--LOSS_SCALE', type=int, default=8096, help=' ')
parser.add_argument('--SAVE_CKPT_PER_N_STEP', type=int, default=1000, help=' ')
parser.add_argument('--KEEP_CKPT_MAX_NUM', type=int, default=5, help=' ')
config = parser.parse_args(args=[])
if __name__ == "__main__":
train(config)
进行验证时,注意修改模型训练时保存的参数文件,验证数据集采用IIIT数据集
import argparse
parser = argparse.ArgumentParser(description='default name', add_help=False)
parser.add_argument('--device_target', type=str, default='Ascend', help='Ascend\GPU\CPU')
parser.add_argument('--CHARACTER', type=str, default="0123456789abcdefghijklmnopqrstuvwxyz", help=' ')
parser.add_argument('--NUM_CLASS', type=int, default=37, help=' ')
parser.add_argument('--HIDDEN_SIZE', type=int, default=512, help=' ')
parser.add_argument('--FINAL_FEATURE_WIDTH', type=int, default=26, help=' ')
parser.add_argument('--IMG_H', type=int, default=32, help=' ')
parser.add_argument('--IMG_W', type=int, default=100, help=' ')
parser.add_argument('--TRAIN_BATCH_SIZE', type=int, default=64, help=' ')
parser.add_argument('--TEST_BATCH_SIZE', type=int, default=64, help=' ')
parser.add_argument('--TRAIN_EPOCHS', type=int, default=3, help=' ')
parser.add_argument('--TRAIN_DATASET_PATH', type=str, default="cnnctc_data/ST_MJ/", help=' ')
parser.add_argument('--TRAIN_DATASET_INDEX_PATH', type=str, default="cnnctc_data/st_mj_fixed_length_index_list.pkl", help=' ')
parser.add_argument('--TEST_DATASET_PATH', type=str, default="cnnctc_data/IIIT/", help=' ')
parser.add_argument('--CHECKPOINT_PATH', type=str, default="ckpt_ascend/CNNCTC-3_650.ckpt", help=' ')
parser.add_argument('--PRED_TRAINED', type=str, default='', help='pred training')
parser.add_argument('--SAVE_PATH', type=str, default='./ckpt_ascend', help='saving the ckpl')
parser.add_argument('--run_distribute', type=bool, default=False, help=' ')
parser.add_argument('--base_lr', type=float, default=0.0005, help=' ')
parser.add_argument('--warmup_ratio', type=float, default=0.0625, help=' ')
parser.add_argument('--MOMENTUM', type=float, default=0.8, help=' ')
parser.add_argument('--warmup_step', type=int, default=2000, help=' ')
parser.add_argument('--LOSS_SCALE', type=int, default=8096, help=' ')
parser.add_argument('--SAVE_CKPT_PER_N_STEP', type=int, default=1000, help=' ')
parser.add_argument('--KEEP_CKPT_MAX_NUM', type=int, default=5, help=' ')
config = parser.parse_args(args=[])
if __name__ == "__main__":
test(config)
parameters loaded! from: ckpt_ascend/CNNCTC-3_650.ckpt
num of samples in IIIT dataset: 2976
[WARNING] MD(2328,fffe1d7fa1e0,python):2022-10-23-14:29:57.766.559 [mindspore/ccsrc/minddata/dataset/engine/datasetops/source/generator_op.cc:195] operator()] Bad performance attention, it takes more than 25 seconds to generator.__next__ new row, which might cause `GetNext` timeout problem when sink_mode=True. You can increase the parameter num_parallel_workers in GeneratorDataset / optimize the efficiency of obtaining samples in the user-defined generator function.
Prediction samples:
['private', 'parking', 'salutes', 'doice', 'cabbana']
Ground truth:
['private', 'parking', 'salutes', 'dolce', 'gabbana']
---------model run time-------- 0.04247260093688965
---------npu_to_cpu run time-------- 0.0005238056182861328
---------postprocess run time-------- 0.0022215843200683594
Prediction samples:
['fund', 'state', 'banik', 'of', 'india']
Ground truth:
['fund', 'state', 'bank', 'of', 'india']
---------model run time-------- 0.04137468338012695
---------npu_to_cpu run time-------- 0.00047707557678222656
---------postprocess run time-------- 0.0021326541900634766
Prediction samples:
['pamus', 'its', 'the', 'cola', 'youfd']
Ground truth:
['venus', 'its', 'the', 'cola', 'youd']
---------model run time-------- 0.04078904787699381
---------npu_to_cpu run time-------- 0.00044997533162434894
---------postprocess run time-------- 0.0021044413248697915
Prediction samples:
['india', 'kafe', 'aibeds', 'baby', 't']
Ground truth:
['india', 'kafe', 'carlsberg', 'baby', 't']
---------model run time-------- 0.04036366939544678
---------npu_to_cpu run time-------- 0.00043779611587524414
---------postprocess run time-------- 0.002089381217956543
Prediction samples:
['summer', 'joeus', 'think', 'big', 'lnupr']
Ground truth:
['summer', 'joes', 'think', 'big', 'imax']
---------model run time-------- 0.040511035919189455
---------npu_to_cpu run time-------- 0.0009461402893066406
---------postprocess run time-------- 0.0020813941955566406
Prediction samples:
['sei', 'customer', 'beees', 'sbi', 'lijfe']
Ground truth:
['sbi', 'customer', 'beers', 'sbi', 'life']
---------model run time-------- 0.04064408938090006
---------npu_to_cpu run time-------- 0.000881354014078776
---------postprocess run time-------- 0.002094308535257975
Prediction samples:
['pog', '21', 'gataldos', 'cua', 'dell']
Ground truth:
['pod', '21', 'cataldos', 'villa', 'del']
---------model run time-------- 0.04071501323154995
---------npu_to_cpu run time-------- 0.0008221353803362165
---------postprocess run time-------- 0.0021024772099086215
Prediction samples:
['suawderly', 'witr', 'puastie', 'cuency', 'ot']
Ground truth:
['strawberry', 'with', 'plastic', 'currency', 'of']
---------model run time-------- 0.04088762402534485
---------npu_to_cpu run time-------- 0.0007749497890472412
---------postprocess run time-------- 0.002101302146911621
Prediction samples:
['mamie', 'io', 'pergnea', '1a', 'resdenca']
Ground truth:
['maple', 'la', 'residencia', 'la', 'residencia']
---------model run time-------- 0.04097112019856771
---------npu_to_cpu run time-------- 0.0007485283745659722
---------postprocess run time-------- 0.002102110120985243
Prediction samples:
['6', 'tg', 'imaizumt', 'mead', 'vengoma']
Ground truth:
['6', '15', 'imaizumi', 'heart', 'welcome']
---------model run time-------- 0.04110002517700195
---------npu_to_cpu run time-------- 0.0007165670394897461
---------postprocess run time-------- 0.002100491523742676
Prediction samples:
['bitu', 'mnolesale', 'beb', 'refined', 'soyabean']
Ground truth:
['bitu', 'wholesale', '888', 'refined', 'soyabean']
---------model run time-------- 0.043298656290227715
---------npu_to_cpu run time-------- 0.0007126981561834162
---------postprocess run time-------- 0.0021564310247247868
Prediction samples:
['correslaw', 'youpe', 'golpe', 'gugy', 'gupe']
Ground truth:
['correslaw', 'south', 'holme', 'glen', 'side']
---------model run time-------- 0.043481131394704185
---------npu_to_cpu run time-------- 0.000705560048421224
---------postprocess run time-------- 0.0021639466285705566
Prediction samples:
['of', 'india', 'associate', 'banks', '8']
Ground truth:
['of', 'india', 'associate', 'banks', '8']
---------model run time-------- 0.04334875253530649
---------npu_to_cpu run time-------- 0.0006855634542611929
---------postprocess run time-------- 0.002158531775841346
Prediction samples:
['give', 'way', '24l9', 'bellway', 'styish']
Ground truth:
['give', 'way', '2419', 'bellway', 'stylish']
---------model run time-------- 0.04311626298086984
---------npu_to_cpu run time-------- 0.0006714378084455218
---------postprocess run time-------- 0.0021534647260393414
Prediction samples:
['hdardinge', 'keiowna', 'bmw', 'a', 'i']
Ground truth:
['hoardings', 'kelowna', 'bmw', '4', '1']
---------model run time-------- 0.04288930892944336
---------npu_to_cpu run time-------- 0.0006557305653889974
---------postprocess run time-------- 0.002152252197265625
Prediction samples:
['ree', 'tthe', 'dmo', 'hwn', 'aidoalba']
Ground truth:
['street', 'the', 'salmon', 'www', 'alibaba']
---------model run time-------- 0.04265661537647247
---------npu_to_cpu run time-------- 0.000639989972114563
---------postprocess run time-------- 0.0021451860666275024
Prediction samples:
['22', 'wwisteria', 'giouse', 'welve', 'oewe']
Ground truth:
['22', 'wisteria', 'house', 'twelve', 'dene']
---------model run time-------- 0.042521364548627066
---------npu_to_cpu run time-------- 0.0006262554841883042
---------postprocess run time-------- 0.00213475788340849
Prediction samples:
['touse', 'o', 'an', 'ani', 'qoper']
Ground truth:
['house', 'tigh', 'an', 'eilean', 'fuller']
---------model run time-------- 0.04235209359063043
---------npu_to_cpu run time-------- 0.0006169478098551432
---------postprocess run time-------- 0.002133713828192817
Prediction samples:
['23', 'g', 'printed', 'hoardings', 'fuil']
Ground truth:
['23', '8', 'printed', 'hoardings', 'full']
---------model run time-------- 0.04225596628691021
---------npu_to_cpu run time-------- 0.0006084191171746505
---------postprocess run time-------- 0.002130746841430664
Prediction samples:
['ldoking', 'the', '19tha', 'its', 'every']
Ground truth:
['looking', 'the', '19th', 'its', 'every']
---------model run time-------- 0.04210841655731201
---------npu_to_cpu run time-------- 0.0005985975265502929
---------postprocess run time-------- 0.0021300435066223143
Prediction samples:
['hantech', 'cautionary', 'sicns', 'en', 'alibaba']
Ground truth:
['hantech', 'cautionary', 'signs', 'en', 'alibaba']
---------model run time-------- 0.04201432636805943
---------npu_to_cpu run time-------- 0.0005902222224644252
---------postprocess run time-------- 0.0021302586510067896
Prediction samples:
['available', 'nowe', 'cail', 'ogoo', 'billboard']
Ground truth:
['available', 'now', 'call', '0800', 'billboard']
---------model run time-------- 0.04189561713825573
---------npu_to_cpu run time-------- 0.0005826624956997958
---------postprocess run time-------- 0.002128037539395419
Prediction samples:
['peatn', 'unitt', 'many', 'of', 'these']
Ground truth:
['peditri', 'unit', 'many', 'of', 'these']
---------model run time-------- 0.041792682979417885
---------npu_to_cpu run time-------- 0.0005749723185663638
---------postprocess run time-------- 0.0021287047344705334
Prediction samples:
['the', 'coke', 'side', 'of', 'ifer']
Ground truth:
['the', 'coke', 'side', 'of', 'life']
---------model run time-------- 0.041723410288492836
---------npu_to_cpu run time-------- 0.0005678335825602213
---------postprocess run time-------- 0.0021340250968933105
Prediction samples:
['bec', 'hotl', 'zoog', 'ft', 'myers']
Ground truth:
['kfc', 'hot', '2003', 'ft', 'myers']
---------model run time-------- 0.04162442207336426
---------npu_to_cpu run time-------- 0.0005614280700683594
---------postprocess run time-------- 0.0021345138549804686
Prediction samples:
['bacipack', 'biboards', 'text', '5eee', 'in']
Ground truth:
['backpack', 'billboards', 'text', '5683', 'in']
---------model run time-------- 0.0415348823253925
---------npu_to_cpu run time-------- 0.0005585138614361102
---------postprocess run time-------- 0.002133635374215933
Prediction samples:
['ltd', 'teli', 'a', 'checking', 'account']
Ground truth:
['ltd', 'tel', 'a', 'checking', 'account']
---------model run time-------- 0.04143884446885851
---------npu_to_cpu run time-------- 0.0005537139044867621
---------postprocess run time-------- 0.002134632181238245
Prediction samples:
['marcus', 'nations', 'destinee', '2', 'agtion']
Ground truth:
['marcus', 'nations', 'destinee', '3d', 'action']
---------model run time-------- 0.041369344506944926
---------npu_to_cpu run time-------- 0.0005625486373901367
---------postprocess run time-------- 0.0021323561668395996
Prediction samples:
['streep', 'pictures', 'of', 'the', 'poubt']
Ground truth:
['streep', 'pictures', 'of', 'the', 'doubt']
---------model run time-------- 0.041363362608284784
---------npu_to_cpu run time-------- 0.0005609742526350351
---------postprocess run time-------- 0.002131980040977741
Prediction samples:
['deey', 'the', 'future', 'prunlee', 'pergil']
Ground truth:
['defy', 'the', 'future', 'prince', 'persia']
---------model run time-------- 0.04128400484720866
---------npu_to_cpu run time-------- 0.0005584955215454102
---------postprocess run time-------- 0.002130866050720215
Prediction samples:
['direcgtor', 'to', 'take', 'them', 'jomes']
Ground truth:
['director', 'to', 'take', 'them', 'jones']
---------model run time-------- 0.041211443562661446
---------npu_to_cpu run time-------- 0.0005543308873330393
---------postprocess run time-------- 0.0021316697520594443
Prediction samples:
['uy', 'restauram', 'irish', 'ipipes', 'main']
Ground truth:
['iv', 'restaurant', 'irish', 'spider', 'man']
---------model run time-------- 0.04114946722984314
---------npu_to_cpu run time-------- 0.0005500465631484985
---------postprocess run time-------- 0.0021315068006515503
Prediction samples:
['on', 'earthim', 'spotting', 'isick', 'boy']
Ground truth:
['on', 'earth', 'spotting', 'sick', 'boy']
---------model run time-------- 0.04108232440370502
---------npu_to_cpu run time-------- 0.0005456317554820668
---------postprocess run time-------- 0.0021306890429872456
Prediction samples:
['needs', 'a', 'tie', 'soloist', 'harfy']
Ground truth:
['needs', 'a', 'the', 'soloist', 'harry']
---------model run time-------- 0.041022419929504395
---------npu_to_cpu run time-------- 0.0005839011248420266
---------postprocess run time-------- 0.0021289937636431527
Prediction samples:
['eterna', 'paiadi', 'prdposwpion', 'terry', 'gilliam']
Ground truth:
['eterna', 'paladin', 'proposition', 'terry', 'gilliam']
---------model run time-------- 0.041003179550170896
---------npu_to_cpu run time-------- 0.0005801405225481306
---------postprocess run time-------- 0.0021273885454450333
Prediction samples:
['vozila', 'prodajia', 'holtywood', 'alilstars', 'hdemle']
Ground truth:
['vozila', 'prodaja', 'hollywood', 'allstars', 'mobile']
---------model run time-------- 0.040953841474321157
---------npu_to_cpu run time-------- 0.000575807359483507
---------postprocess run time-------- 0.0021270977126227487
Prediction samples:
['86sgtbag', 'coupon', 'codes', 'onlypayicenty', 'website']
Ground truth:
['9365978448', 'coupon', 'code', 'onlypay1cent', 'website']
---------model run time-------- 0.040926295357781486
---------npu_to_cpu run time-------- 0.0005714184529072529
---------postprocess run time-------- 0.0021257916012325804
Prediction samples:
['service', 'indonesia', 'comap', 'for', 'mored']
Ground truth:
['service', 'indonesia', 'comap', 'for', 'more']
---------model run time-------- 0.04087934995952405
---------npu_to_cpu run time-------- 0.0005671977996826172
---------postprocess run time-------- 0.002124962053800884
Prediction samples:
['bank', 'of', 'the', 'way', 'original']
Ground truth:
['bank', 'of', 'the', 'way', 'original']
---------model run time-------- 0.040828741513765775
---------npu_to_cpu run time-------- 0.0005631691370254908
---------postprocess run time-------- 0.0021251837412516275
Prediction samples:
['t', 'state', 'chandicarh', 'bank', 'df']
Ground truth:
['virgin', 'state', 'chandigarh', 'bank', 'of']
---------model run time-------- 0.04080718159675598
---------npu_to_cpu run time-------- 0.0005594074726104736
---------postprocess run time-------- 0.0021219074726104735
Prediction samples:
['to', 'feel', 'the', 'of', 'the']
Ground truth:
['to', 'feel', 'the', 'of', 'the']
---------model run time-------- 0.04076695442199707
---------npu_to_cpu run time-------- 0.0005560851678615663
---------postprocess run time-------- 0.0021208321175924163
Prediction samples:
['beds', 'instored', 'per', 'cold', 'as']
Ground truth:
['beds', 'instore', 'pet', 'cold', 'as']
---------model run time-------- 0.04072521981738862
---------npu_to_cpu run time-------- 0.0005528359186081659
---------postprocess run time-------- 0.0021213406608218237
Prediction samples:
['in', 'password', 'manedement', 'login', 'to']
Ground truth:
['in', 'password', 'management', 'login', 'to']
---------model run time-------- 0.04071358747260515
---------npu_to_cpu run time-------- 0.000550613846889762
---------postprocess run time-------- 0.00212052256561989
Prediction samples:
['smiler', 'enetorn', 'saes', 'chehist', 'open']
Ground truth:
['smile', 'factory', 'sale', 'chemist', 'open']
---------model run time-------- 0.04071019454435869
---------npu_to_cpu run time-------- 0.0005479617552323775
---------postprocess run time-------- 0.0021184303543784404
Prediction samples:
['inda', 'rancht', 'watt', 'longerg', 'cenee']
Ground truth:
['india', 'ranchi', 'wait', 'longer', 'centre']
---------model run time-------- 0.04069504737854004
---------npu_to_cpu run time-------- 0.0005453003777398003
---------postprocess run time-------- 0.002118486828274197
Prediction samples:
['loan', 'schedule', 'of', 'about', 'services']
Ground truth:
['loan', 'schedule', 'of', 'about', 'services']
num of samples in IIIT dataset: 2976
accuracy: 0.5285326086956522