Comments (6)
from pp-yoloe.
这是在训练过程中出现的,已经训练了100轮
from pp-yoloe.
看报错是纬度问题,但是如果前面正常训练过的话,数据还有模型,其他的应该不会有问题。损失值有什么变化吗?
from pp-yoloe.
对,报错是维度问题,损失值也下降了;但是感觉能训练应该之前的数据 应该没有问题~,就不知道哪里出错了,每次到100轮的时候就终止了
from pp-yoloe.
再次启动时,还是出现以下问题:
declare_namespace(pkg)
----------- Configuration Arguments -----------
batch_size: 4
eval_anno_path: dataset/eval.json
image_dir: dataset/
learning_rate: 0.000125
log_interval: 500
model_type: M
num_classes: 4
num_epoch: 1000
num_workers: 8
pretrained_model: None
resume_model: None
save_model_path: models/
train_anno_path: dataset/train.json
use_gpu: True
use_random_crop: True
use_random_distort: True
use_random_expand: True
use_random_flip: True
True
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
W0609 16:01:00.625028 6134 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 11.1, Runtime API Version: 11.1
W0609 16:01:00.627727 6134 gpu_resources.cc:91] device: 0, cuDNN Version: 8.1.
W0609 16:01:00.627743 6134 gpu_resources.cc:117] WARNING: device: 0. The installed Paddle is compiled with CUDA 11.2, but CUDA runtime version in your machine is 11.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDA version.
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
[2023-06-09 16:01:02.739166 INFO ] trainer:train:285 - 训练数据:720
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.0.blocks.0.conv2.alpha. backbone.stages.0.blocks.0.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.0.blocks.1.conv2.alpha. backbone.stages.0.blocks.1.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.1.blocks.0.conv2.alpha. backbone.stages.1.blocks.0.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.1.blocks.1.conv2.alpha. backbone.stages.1.blocks.1.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.1.blocks.2.conv2.alpha. backbone.stages.1.blocks.2.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.1.blocks.3.conv2.alpha. backbone.stages.1.blocks.3.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.2.blocks.0.conv2.alpha. backbone.stages.2.blocks.0.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.2.blocks.1.conv2.alpha. backbone.stages.2.blocks.1.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.2.blocks.2.conv2.alpha. backbone.stages.2.blocks.2.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.2.blocks.3.conv2.alpha. backbone.stages.2.blocks.3.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.3.blocks.0.conv2.alpha. backbone.stages.3.blocks.0.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for backbone.stages.3.blocks.1.conv2.alpha. backbone.stages.3.blocks.1.conv2.alpha is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.0.weight. yolo_head.pred_cls.0.weight receives a shape [365, 576, 3, 3], but the expected shape is [4, 576, 3, 3].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.0.bias. yolo_head.pred_cls.0.bias receives a shape [365], but the expected shape is [4].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.1.weight. yolo_head.pred_cls.1.weight receives a shape [365, 288, 3, 3], but the expected shape is [4, 288, 3, 3].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.1.bias. yolo_head.pred_cls.1.bias receives a shape [365], but the expected shape is [4].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.2.weight. yolo_head.pred_cls.2.weight receives a shape [365, 144, 3, 3], but the expected shape is [4, 144, 3, 3].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for yolo_head.pred_cls.2.bias. yolo_head.pred_cls.2.bias receives a shape [365], but the expected shape is [4].
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
[2023-06-09 16:01:02.990542 INFO ] trainer:__load_pretrained:151 - 成功加载预训练模型:pretrained_models/ppyoloe_crn_m_obj365_pretrained.pdparams
[2023-06-09 16:01:03.230492 INFO ] trainer:__load_checkpoint:169 - 成功恢复模型参数和优化方法参数:models/PPYOLOE_M/last_model
W0609 16:01:06.713155 6134 gpu_resources.cc:217] WARNING: device: . The installed Paddle is compiled with CUDNN 8.2, but CUDNN version in your machine is 8.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/nn/layer/norm.py:712: UserWarning: When training, we now always track global mean and variance.
warnings.warn(
Traceback (most recent call last):
File "train.py", line 44, in
trainer.train(num_epoch=args.num_epoch,
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/trainer.py", line 300, in train
self.__train_epoch(max_epoch=num_epoch, epoch_id=epoch_id, log_interval=log_interval, local_rank=local_rank, writer=writer)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/trainer.py", line 204, in __train_epoch
output = self.model(data)
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 1012, in call
return self.forward(*inputs, **kwargs)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/meta_arch.py", line 53, in forward
out = self.get_loss()
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/yolo.py", line 46, in get_loss
return self._forward()
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/yolo.py", line 36, in _forward
yolo_losses = self.yolo_head(neck_feats, self.inputs)
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 1012, in call
return self.forward(*inputs, **kwargs)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/ppyoloe_head.py", line 202, in forward
return self.forward_train(feats, targets)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/ppyoloe_head.py", line 142, in forward_train
return self.get_loss([
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/ppyoloe_head.py", line 308, in get_loss
self.assigner(
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 1012, in call
return self.forward(*inputs, **kwargs)
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/decorator.py", line 232, in fun
return caller(func, *(extras + args), **kw)
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/fluid/dygraph/base.py", line 375, in _decorate_function
return func(*args, **kwargs)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/task_aligned_assigner.py", line 93, in forward
ious = iou_similarity(gt_bboxes, pred_bboxes)
File "/data/dongxz/competion/cv/PP-YOLOE/ppyoloe/model/bbox_utils.py", line 42, in iou_similarity
x2y2 = paddle.minimum(px2y2, gx2y2)
File "/data/anaconda3/envs/dongxz_paddlepaddle/lib/python3.8/site-packages/paddle/tensor/math.py", line 1008, in minimum
return _C_ops.minimum(x, y)
ValueError: (InvalidArgument) The 3-th dimension of input tensor is expected to be equal with the 3-th dimension of output tensor 2 or 1, but received 0. (at /paddle/paddle/phi/kernels/funcs/broadcast_function.h:77)
from pp-yoloe.
from pp-yoloe.
Related Issues (13)
- VOC训练问题 HOT 2
- 训练中断 HOT 2
- 2020-10-21 22:58:00,947-WARNING: Found an invalid bbox in annotations: xml_file: dataset/annotation/001113.xml, x1: 3765.0, y1: 1851.0, x2: 3765.0, y2: 1871.0. HOT 4
- How to do inference with batch_size > 1 ? HOT 2
- Error: Cannot load cudnn shared library. HOT 1
- 运行train.py报错 HOT 2
- 模型训练很慢是什么问题呀? HOT 6
- 多卡训练该如何配置 HOT 3
- dataset HOT 21
- CUDA_VISIBLE_DEVICES 无法指定 GPU HOT 3
- oriented object detection HOT 3
- Error Broadcast dimension mismatch HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pp-yoloe.