Cannot get the AP result in the paper

[megvii-model]

Cannot get the AP result in the paper

[chensnathan]

Could you provide more details about how you modified the settings? It will be great if you can provide your training log file.

[yqjiang6]

Here is the modified settings:

SOLVER:
IMS_PER_BATCH: 32
BASE_LR: 0.06
WARMUP_FACTOR: 0.00066667
WARMUP_ITERS: 1500
STEPS: (15000, 25000)
MAX_ITER: 45000
CHECKPOINT_PERIOD: 5000

Other parts are not changed.

[chensnathan]

Try:

WARMUP_FACTOR: 0.00033333
WARMUP_ITERS: 3000

[yqjiang6]

Thanks for your reply. I tried this warmup settings and only got an AP of 35.8365. Here is part of my training log ( I have to delete some lines since it exceeds the maximum character length of the comment)

[08/06 16:42:57] detectron2 INFO: Rank of current process: 0. World size: 1
[08/06 16:42:59] detectron2 INFO: Environment info:

---

sys.platform linux
Python 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
numpy 1.20.3
detectron2 0.4 @/home/yqjiang6/.conda/envs/torch/lib/python3.7/site-packages/detectron2
Compiler GCC 7.3
CUDA compiler CUDA 11.0
detectron2 arch flags /home/yqjiang6/.conda/envs/torch/lib/python3.7/site-packages/detectron2/_C.cpython-37m-x86_64-linux-gnu.so; cannot find cuobjdump
DETECTRON2_ENV_MODULE
PyTorch 1.7.1 @/home/yqjiang6/.conda/envs/torch/lib/python3.7/site-packages/torch
PyTorch debug build False
GPU available True
GPU 0 GeForce RTX 3090 (arch=8.6)
CUDA_HOME /usr/local/cuda-11.1:/usr/local/cuda:/usr/local/cuda - invalid!
Pillow 8.1.0
torchvision 0.8.2 @/home/yqjiang6/.conda/envs/torch/lib/python3.7/site-packages/torchvision
torchvision arch flags /home/yqjiang6/.conda/envs/torch/lib/python3.7/site-packages/torchvision/_C.so; cannot find cuobjdump
fvcore 0.1.3.post20210317
cv2 4.5.1

---

PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.0
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

[08/06 16:42:59] detectron2 INFO: Full config saved to output/yolof/R_50_C5_1x/config.yaml
[08/06 16:42:59] d2.utils.env INFO: Using a generated random seed 59790419

[08/06 16:43:39] d2.engine.train_loop INFO: Starting training from iteration 0
[08/06 16:59:29] d2.utils.events INFO: eta: 23:10:17 iter: 499 total_loss: 1.071 loss_cls: 0.5044 loss_box_reg: 0.5646 time: 1.8843 data_time: 0.0813 lr: 0.0099967 max_mem: 18157M
[08/06 17:15:19] d2.utils.events INFO: eta: 22:55:59 iter: 999 total_loss: 0.9464 loss_cls: 0.4346 loss_box_reg: 0.5072 time: 1.8908 data_time: 0.0790 lr: 0.019993 max_mem: 18165M
[08/06 17:31:11] d2.utils.events INFO: eta: 22:42:53 iter: 1499 total_loss: 0.9174 loss_cls: 0.4104 loss_box_reg: 0.5095 time: 1.8939 data_time: 0.0797 lr: 0.02999 max_mem: 18165M
[08/06 17:46:58] d2.utils.events INFO: eta: 22:29:24 iter: 1999 total_loss: 0.9137 loss_cls: 0.3883 loss_box_reg: 0.5181 time: 1.8931 data_time: 0.0763 lr: 0.039987 max_mem: 18165M
[08/06 18:02:41] d2.utils.events INFO: eta: 22:12:31 iter: 2499 total_loss: 0.8506 loss_cls: 0.3894 loss_box_reg: 0.4744 time: 1.8913 data_time: 0.0779 lr: 0.049983 max_mem: 18165M
[08/06 18:18:26] d2.utils.events INFO: eta: 21:54:17 iter: 2999 total_loss: 0.8421 loss_cls: 0.3799 loss_box_reg: 0.4543 time: 1.8906 data_time: 0.0764 lr: 0.05998 max_mem: 18165M
[08/06 18:34:19] d2.utils.events INFO: eta: 21:45:28 iter: 3499 total_loss: 0.7868 loss_cls: 0.3562 loss_box_reg: 0.4351 time: 1.8923 data_time: 0.0749 lr: 0.06 max_mem: 18165M
[08/06 18:50:11] d2.utils.events INFO: eta: 21:31:17 iter: 3999 total_loss: 0.7821 loss_cls: 0.3521 loss_box_reg: 0.4414 time: 1.8933 data_time: 0.0764 lr: 0.06 max_mem: 18165M
[08/06 19:05:55] d2.utils.events INFO: eta: 21:08:00 iter: 4499 total_loss: 0.7937 loss_cls: 0.3441 loss_box_reg: 0.4506 time: 1.8923 data_time: 0.0775 lr: 0.06 max_mem: 18165M
[08/06 19:21:42] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0004999.pth
[08/06 19:21:42] d2.utils.events INFO: eta: 20:51:58 iter: 4999 total_loss: 0.7583 loss_cls: 0.3357 loss_box_reg: 0.4187 time: 1.8923 data_time: 0.0807 lr: 0.06 max_mem: 18165M
[08/06 19:37:28] d2.utils.events INFO: eta: 20:37:31 iter: 5499 total_loss: 0.7648 loss_cls: 0.3342 loss_box_reg: 0.4288 time: 1.8917 data_time: 0.0784 lr: 0.06 max_mem: 18165M
[08/06 19:53:12] d2.utils.events INFO: eta: 20:17:26 iter: 5999 total_loss: 0.7361 loss_cls: 0.3296 loss_box_reg: 0.4145 time: 1.8912 data_time: 0.0805 lr: 0.06 max_mem: 18165M
[08/06 20:08:55] d2.utils.events INFO: eta: 20:00:42 iter: 6499 total_loss: 0.7534 loss_cls: 0.337 loss_box_reg: 0.4211 time: 1.8905 data_time: 0.0805 lr: 0.06 max_mem: 18165M
[08/06 20:24:41] d2.utils.events INFO: eta: 19:46:44 iter: 6999 total_loss: 0.7508 loss_cls: 0.3329 loss_box_reg: 0.4281 time: 1.8904 data_time: 0.0787 lr: 0.06 max_mem: 18165M
[08/06 20:40:29] d2.utils.events INFO: eta: 19:35:15 iter: 7499 total_loss: 0.7171 loss_cls: 0.3073 loss_box_reg: 0.4006 time: 1.8906 data_time: 0.0802 lr: 0.06 max_mem: 18165M
[08/06 20:56:10] d2.utils.events INFO: eta: 19:18:09 iter: 7999 total_loss: 0.7059 loss_cls: 0.3103 loss_box_reg: 0.3963 time: 1.8899 data_time: 0.0761 lr: 0.06 max_mem: 18165M
[08/06 21:11:54] d2.utils.events INFO: eta: 19:02:03 iter: 8499 total_loss: 0.7095 loss_cls: 0.3155 loss_box_reg: 0.396 time: 1.8896 data_time: 0.0724 lr: 0.06 max_mem: 18165M
[08/06 21:27:38] d2.utils.events INFO: eta: 18:48:24 iter: 8999 total_loss: 0.7053 loss_cls: 0.3148 loss_box_reg: 0.3939 time: 1.8895 data_time: 0.0770 lr: 0.06 max_mem: 18165M
[08/06 21:43:20] d2.utils.events INFO: eta: 18:30:46 iter: 9499 total_loss: 0.709 loss_cls: 0.3035 loss_box_reg: 0.4075 time: 1.8890 data_time: 0.0813 lr: 0.06 max_mem: 18165M
[08/06 21:59:06] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0009999.pth
[08/06 21:59:07] d2.utils.events INFO: eta: 18:12:32 iter: 9999 total_loss: 0.6842 loss_cls: 0.3053 loss_box_reg: 0.3857 time: 1.8890 data_time: 0.0817 lr: 0.06 max_mem: 18165M
[08/06 22:14:51] d2.utils.events INFO: eta: 17:58:30 iter: 10499 total_loss: 0.6992 loss_cls: 0.314 loss_box_reg: 0.3978 time: 1.8889 data_time: 0.0815 lr: 0.06 max_mem: 18165M
[08/06 22:30:38] d2.utils.events INFO: eta: 17:43:57 iter: 10999 total_loss: 0.6936 loss_cls: 0.3021 loss_box_reg: 0.3923 time: 1.8890 data_time: 0.0802 lr: 0.06 max_mem: 18165M
[08/06 22:46:21] d2.utils.events INFO: eta: 17:29:34 iter: 11499 total_loss: 0.6921 loss_cls: 0.3016 loss_box_reg: 0.3842 time: 1.8888 data_time: 0.0758 lr: 0.06 max_mem: 18165M
[08/06 23:02:01] d2.utils.events INFO: eta: 17:11:10 iter: 11999 total_loss: 0.6753 loss_cls: 0.2991 loss_box_reg: 0.3748 time: 1.8883 data_time: 0.0803 lr: 0.06 max_mem: 18165M
[08/06 23:17:50] d2.utils.events INFO: eta: 16:54:31 iter: 12499 total_loss: 0.6742 loss_cls: 0.2989 loss_box_reg: 0.3764 time: 1.8886 data_time: 0.0740 lr: 0.06 max_mem: 18165M
[08/06 23:33:32] d2.utils.events INFO: eta: 16:38:32 iter: 12999 total_loss: 0.6913 loss_cls: 0.2926 loss_box_reg: 0.3856 time: 1.8883 data_time: 0.0792 lr: 0.06 max_mem: 18165M
[08/06 23:49:21] d2.utils.events INFO: eta: 16:25:27 iter: 13499 total_loss: 0.6701 loss_cls: 0.2903 loss_box_reg: 0.3772 time: 1.8885 data_time: 0.0811 lr: 0.06 max_mem: 18165M
[08/07 00:05:12] d2.utils.events INFO: eta: 16:13:43 iter: 13999 total_loss: 0.6779 loss_cls: 0.2996 loss_box_reg: 0.3756 time: 1.8890 data_time: 0.0796 lr: 0.06 max_mem: 18165M
[08/07 00:20:53] d2.utils.events INFO: eta: 15:54:26 iter: 14499 total_loss: 0.6941 loss_cls: 0.3006 loss_box_reg: 0.3838 time: 1.8886 data_time: 0.0795 lr: 0.06 max_mem: 18165M
[08/07 00:36:32] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0014999.pth
[08/07 00:36:32] d2.utils.events INFO: eta: 15:32:55 iter: 14999 total_loss: 0.6601 loss_cls: 0.2847 loss_box_reg: 0.3831 time: 1.8881 data_time: 0.0777 lr: 0.06 max_mem: 18165M
[08/07 00:52:11] d2.utils.events INFO: eta: 15:19:12 iter: 15499 total_loss: 0.6018 loss_cls: 0.2568 loss_box_reg: 0.3427 time: 1.8877 data_time: 0.0782 lr: 0.006 max_mem: 18165M
[08/07 01:07:59] d2.utils.events INFO: eta: 15:06:58 iter: 15999 total_loss: 0.5774 loss_cls: 0.2447 loss_box_reg: 0.319 time: 1.8878 data_time: 0.0745 lr: 0.006 max_mem: 18165M
[08/07 01:23:47] d2.utils.events INFO: eta: 14:51:22 iter: 16499 total_loss: 0.587 loss_cls: 0.2493 loss_box_reg: 0.3367 time: 1.8880 data_time: 0.0776 lr: 0.006 max_mem: 18165M
[08/07 01:39:31] d2.utils.events INFO: eta: 14:35:16 iter: 16999 total_loss: 0.5484 loss_cls: 0.2523 loss_box_reg: 0.3108 time: 1.8879 data_time: 0.0743 lr: 0.006 max_mem: 18165M
[08/07 01:55:20] d2.utils.events INFO: eta: 14:23:00 iter: 17499 total_loss: 0.5807 loss_cls: 0.2527 loss_box_reg: 0.3251 time: 1.8881 data_time: 0.0802 lr: 0.006 max_mem: 18165M
[08/07 02:11:08] d2.utils.events INFO: eta: 14:09:33 iter: 17999 total_loss: 0.5903 loss_cls: 0.2401 loss_box_reg: 0.3454 time: 1.8883 data_time: 0.0807 lr: 0.006 max_mem: 18165M
[08/07 02:26:54] d2.utils.events INFO: eta: 13:53:13 iter: 18499 total_loss: 0.5712 loss_cls: 0.2353 loss_box_reg: 0.3314 time: 1.8883 data_time: 0.0776 lr: 0.006 max_mem: 18165M
[08/07 02:42:41] d2.utils.events INFO: eta: 13:36:52 iter: 18999 total_loss: 0.5739 loss_cls: 0.2359 loss_box_reg: 0.3344 time: 1.8883 data_time: 0.0792 lr: 0.006 max_mem: 18165M
[08/07 02:58:30] d2.utils.events INFO: eta: 13:21:17 iter: 19499 total_loss: 0.5657 loss_cls: 0.2435 loss_box_reg: 0.3233 time: 1.8885 data_time: 0.0788 lr: 0.006 max_mem: 18165M
[08/07 03:14:20] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0019999.pth
[08/07 03:14:20] d2.utils.events INFO: eta: 13:08:50 iter: 19999 total_loss: 0.5819 loss_cls: 0.2356 loss_box_reg: 0.3518 time: 1.8887 data_time: 0.0775 lr: 0.006 max_mem: 18165M
[08/07 03:30:05] d2.utils.events INFO: eta: 12:50:52 iter: 20499 total_loss: 0.5802 loss_cls: 0.2486 loss_box_reg: 0.3284 time: 1.8886 data_time: 0.0808 lr: 0.006 max_mem: 18165M
[08/07 03:45:56] d2.utils.events INFO: eta: 12:34:13 iter: 20999 total_loss: 0.5701 loss_cls: 0.2386 loss_box_reg: 0.3318 time: 1.8889 data_time: 0.0831 lr: 0.006 max_mem: 18165M
[08/07 04:01:45] d2.utils.events INFO: eta: 12:21:25 iter: 21499 total_loss: 0.5621 loss_cls: 0.2257 loss_box_reg: 0.3196 time: 1.8890 data_time: 0.0802 lr: 0.006 max_mem: 18165M
[08/07 04:17:36] d2.utils.events INFO: eta: 12:04:21 iter: 21999 total_loss: 0.5496 loss_cls: 0.2382 loss_box_reg: 0.3178 time: 1.8892 data_time: 0.0892 lr: 0.006 max_mem: 18165M
[08/07 04:33:24] d2.utils.events INFO: eta: 11:48:54 iter: 22499 total_loss: 0.5637 loss_cls: 0.2318 loss_box_reg: 0.3346 time: 1.8894 data_time: 0.0786 lr: 0.006 max_mem: 18165M
[08/07 04:49:16] d2.utils.events INFO: eta: 11:33:25 iter: 22999 total_loss: 0.5575 loss_cls: 0.2286 loss_box_reg: 0.3148 time: 1.8896 data_time: 0.0782 lr: 0.006 max_mem: 18165M
[08/07 05:05:09] d2.utils.events INFO: eta: 11:18:50 iter: 23499 total_loss: 0.5545 loss_cls: 0.2303 loss_box_reg: 0.3208 time: 1.8899 data_time: 0.0806 lr: 0.006 max_mem: 18165M
[08/07 05:20:59] d2.utils.events INFO: eta: 11:03:47 iter: 23999 total_loss: 0.5531 loss_cls: 0.2299 loss_box_reg: 0.314 time: 1.8901 data_time: 0.0769 lr: 0.006 max_mem: 18165M
[08/07 05:36:49] d2.utils.events INFO: eta: 10:47:20 iter: 24499 total_loss: 0.574 loss_cls: 0.2352 loss_box_reg: 0.3389 time: 1.8902 data_time: 0.0790 lr: 0.006 max_mem: 18165M
[08/07 05:52:37] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0024999.pth
[08/07 05:52:37] d2.utils.events INFO: eta: 10:30:28 iter: 24999 total_loss: 0.5408 loss_cls: 0.2383 loss_box_reg: 0.3075 time: 1.8902 data_time: 0.0802 lr: 0.006 max_mem: 18165M
[08/07 06:08:29] d2.utils.events INFO: eta: 10:16:16 iter: 25499 total_loss: 0.5538 loss_cls: 0.2382 loss_box_reg: 0.3237 time: 1.8904 data_time: 0.0812 lr: 0.0006 max_mem: 18165M
[08/07 06:24:19] d2.utils.events INFO: eta: 10:01:29 iter: 25999 total_loss: 0.5474 loss_cls: 0.2232 loss_box_reg: 0.3196 time: 1.8906 data_time: 0.0785 lr: 0.0006 max_mem: 18165M
[08/07 06:40:10] d2.utils.events INFO: eta: 9:44:11 iter: 26499 total_loss: 0.5393 loss_cls: 0.2243 loss_box_reg: 0.3222 time: 1.8907 data_time: 0.0772 lr: 0.0006 max_mem: 18165M
[08/07 06:56:00] d2.utils.events INFO: eta: 9:27:22 iter: 26999 total_loss: 0.5521 loss_cls: 0.2303 loss_box_reg: 0.3245 time: 1.8908 data_time: 0.0751 lr: 0.0006 max_mem: 18165M
[08/07 07:11:49] d2.utils.events INFO: eta: 9:11:39 iter: 27499 total_loss: 0.552 loss_cls: 0.2208 loss_box_reg: 0.3237 time: 1.8909 data_time: 0.0774 lr: 0.0006 max_mem: 18165M
[08/07 07:27:42] d2.utils.events INFO: eta: 8:57:25 iter: 27999 total_loss: 0.5313 loss_cls: 0.2184 loss_box_reg: 0.3171 time: 1.8911 data_time: 0.0782 lr: 0.0006 max_mem: 18165M
[08/07 07:43:32] d2.utils.events INFO: eta: 8:40:28 iter: 28499 total_loss: 0.5428 loss_cls: 0.2176 loss_box_reg: 0.3267 time: 1.8912 data_time: 0.0796 lr: 0.0006 max_mem: 18165M
[08/07 07:59:21] d2.utils.events INFO: eta: 8:23:31 iter: 28999 total_loss: 0.5286 loss_cls: 0.2211 loss_box_reg: 0.3084 time: 1.8913 data_time: 0.0749 lr: 0.0006 max_mem: 18165M
[08/07 08:15:12] d2.utils.events INFO: eta: 8:08:51 iter: 29499 total_loss: 0.5395 loss_cls: 0.2264 loss_box_reg: 0.3174 time: 1.8914 data_time: 0.0852 lr: 0.0006 max_mem: 18165M
[08/07 08:31:03] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0029999.pth
[08/07 08:31:03] d2.utils.events INFO: eta: 7:53:48 iter: 29999 total_loss: 0.542 loss_cls: 0.2208 loss_box_reg: 0.3125 time: 1.8916 data_time: 0.0795 lr: 0.0006 max_mem: 18165M
[08/07 08:46:53] d2.utils.events INFO: eta: 7:38:20 iter: 30499 total_loss: 0.5504 loss_cls: 0.2211 loss_box_reg: 0.3203 time: 1.8916 data_time: 0.0772 lr: 0.0006 max_mem: 18165M
[08/07 09:02:41] d2.utils.events INFO: eta: 7:22:46 iter: 30999 total_loss: 0.5328 loss_cls: 0.2192 loss_box_reg: 0.3059 time: 1.8917 data_time: 0.0742 lr: 0.0006 max_mem: 18165M
[08/07 09:18:27] d2.utils.events INFO: eta: 7:07:00 iter: 31499 total_loss: 0.5368 loss_cls: 0.2146 loss_box_reg: 0.3158 time: 1.8916 data_time: 0.0801 lr: 0.0006 max_mem: 18165M
[08/07 09:34:19] d2.utils.events INFO: eta: 6:52:10 iter: 31999 total_loss: 0.5441 loss_cls: 0.2251 loss_box_reg: 0.3275 time: 1.8918 data_time: 0.0834 lr: 0.0006 max_mem: 18165M
[08/07 09:50:10] d2.utils.events INFO: eta: 6:36:10 iter: 32499 total_loss: 0.5508 loss_cls: 0.2332 loss_box_reg: 0.324 time: 1.8919 data_time: 0.0777 lr: 0.0006 max_mem: 18165M
[08/07 10:05:58] d2.utils.events INFO: eta: 6:18:43 iter: 32999 total_loss: 0.5314 loss_cls: 0.2205 loss_box_reg: 0.3161 time: 1.8919 data_time: 0.0856 lr: 0.0006 max_mem: 18165M
[08/07 10:21:49] d2.utils.events INFO: eta: 6:02:35 iter: 33499 total_loss: 0.5418 loss_cls: 0.2272 loss_box_reg: 0.2985 time: 1.8920 data_time: 0.0801 lr: 0.0006 max_mem: 18165M
[08/07 10:37:36] d2.utils.events INFO: eta: 5:46:06 iter: 33999 total_loss: 0.5236 loss_cls: 0.2187 loss_box_reg: 0.3059 time: 1.8920 data_time: 0.0763 lr: 0.0006 max_mem: 18165M
[08/07 10:53:22] d2.utils.events INFO: eta: 5:29:51 iter: 34499 total_loss: 0.5452 loss_cls: 0.2207 loss_box_reg: 0.3242 time: 1.8919 data_time: 0.0720 lr: 0.0006 max_mem: 18165M
[08/07 11:09:10] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0034999.pth
[08/07 11:09:11] d2.utils.events INFO: eta: 5:14:31 iter: 34999 total_loss: 0.5357 loss_cls: 0.2241 loss_box_reg: 0.3073 time: 1.8920 data_time: 0.0796 lr: 0.0006 max_mem: 18165M
[08/07 11:25:01] d2.utils.events INFO: eta: 4:59:38 iter: 35499 total_loss: 0.5283 loss_cls: 0.2153 loss_box_reg: 0.3114 time: 1.8920 data_time: 0.0781 lr: 0.0006 max_mem: 18165M
[08/07 11:40:48] d2.utils.events INFO: eta: 4:44:23 iter: 35999 total_loss: 0.5485 loss_cls: 0.2313 loss_box_reg: 0.3175 time: 1.8920 data_time: 0.0728 lr: 0.0006 max_mem: 18165M
[08/07 11:56:40] d2.utils.events INFO: eta: 4:28:45 iter: 36499 total_loss: 0.5555 loss_cls: 0.2253 loss_box_reg: 0.3251 time: 1.8922 data_time: 0.0749 lr: 0.0006 max_mem: 18165M
[08/07 12:12:30] d2.utils.events INFO: eta: 4:13:13 iter: 36999 total_loss: 0.5496 loss_cls: 0.2161 loss_box_reg: 0.3274 time: 1.8922 data_time: 0.0804 lr: 0.0006 max_mem: 18165M
[08/07 12:28:19] d2.utils.events INFO: eta: 3:57:00 iter: 37499 total_loss: 0.5412 loss_cls: 0.225 loss_box_reg: 0.3231 time: 1.8923 data_time: 0.0775 lr: 0.0006 max_mem: 18165M
[08/07 12:44:06] d2.utils.events INFO: eta: 3:40:44 iter: 37999 total_loss: 0.5439 loss_cls: 0.2273 loss_box_reg: 0.3069 time: 1.8923 data_time: 0.0855 lr: 0.0006 max_mem: 18165M
[08/07 12:59:58] d2.utils.events INFO: eta: 3:25:12 iter: 38499 total_loss: 0.5331 loss_cls: 0.2265 loss_box_reg: 0.3283 time: 1.8924 data_time: 0.0823 lr: 0.0006 max_mem: 18165M
[08/07 13:15:50] d2.utils.events INFO: eta: 3:09:49 iter: 38999 total_loss: 0.5315 loss_cls: 0.219 loss_box_reg: 0.3211 time: 1.8925 data_time: 0.0798 lr: 0.0006 max_mem: 18165M
[08/07 13:31:41] d2.utils.events INFO: eta: 2:53:48 iter: 39499 total_loss: 0.5276 loss_cls: 0.2208 loss_box_reg: 0.3083 time: 1.8925 data_time: 0.0776 lr: 0.0006 max_mem: 18165M
[08/07 13:47:31] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0039999.pth
[08/07 13:47:31] d2.utils.events INFO: eta: 2:37:28 iter: 39999 total_loss: 0.5459 loss_cls: 0.2189 loss_box_reg: 0.3298 time: 1.8926 data_time: 0.0777 lr: 0.0006 max_mem: 18165M
[08/07 14:03:17] d2.utils.events INFO: eta: 2:21:45 iter: 40499 total_loss: 0.5279 loss_cls: 0.2103 loss_box_reg: 0.312 time: 1.8926 data_time: 0.0791 lr: 0.0006 max_mem: 18165M
[08/07 14:19:06] d2.utils.events INFO: eta: 2:06:11 iter: 40999 total_loss: 0.5267 loss_cls: 0.218 loss_box_reg: 0.3155 time: 1.8926 data_time: 0.0789 lr: 0.0006 max_mem: 18165M
[08/07 14:34:53] d2.utils.events INFO: eta: 1:50:22 iter: 41499 total_loss: 0.5176 loss_cls: 0.2172 loss_box_reg: 0.296 time: 1.8926 data_time: 0.0825 lr: 0.0006 max_mem: 18165M
[08/07 14:50:49] d2.utils.events INFO: eta: 1:34:48 iter: 41999 total_loss: 0.5441 loss_cls: 0.2135 loss_box_reg: 0.3219 time: 1.8928 data_time: 0.0784 lr: 0.0006 max_mem: 18165M
[08/07 15:06:37] d2.utils.events INFO: eta: 1:18:53 iter: 42499 total_loss: 0.534 loss_cls: 0.2167 loss_box_reg: 0.323 time: 1.8928 data_time: 0.0750 lr: 0.0006 max_mem: 18165M
[08/07 15:22:28] d2.utils.events INFO: eta: 1:03:09 iter: 42999 total_loss: 0.5132 loss_cls: 0.2168 loss_box_reg: 0.2942 time: 1.8928 data_time: 0.0788 lr: 0.0006 max_mem: 18165M
[08/07 15:38:16] d2.utils.events INFO: eta: 0:47:25 iter: 43499 total_loss: 0.5376 loss_cls: 0.2207 loss_box_reg: 0.3236 time: 1.8929 data_time: 0.0773 lr: 0.0006 max_mem: 18165M
[08/07 15:54:07] d2.utils.events INFO: eta: 0:31:35 iter: 43999 total_loss: 0.515 loss_cls: 0.2184 loss_box_reg: 0.3051 time: 1.8929 data_time: 0.0751 lr: 0.0006 max_mem: 18165M
[08/07 16:09:52] d2.utils.events INFO: eta: 0:15:46 iter: 44499 total_loss: 0.5291 loss_cls: 0.2269 loss_box_reg: 0.306 time: 1.8928 data_time: 0.0750 lr: 0.0006 max_mem: 18165M
[08/07 16:25:36] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_0044999.pth
[08/07 16:25:37] fvcore.common.checkpoint INFO: Saving checkpoint to output/yolof/R_50_C5_1x/model_final.pth
[08/07 16:25:37] d2.utils.events INFO: eta: 0:00:00 iter: 44999 total_loss: 0.5258 loss_cls: 0.2177 loss_box_reg: 0.3009 time: 1.8928 data_time: 0.0774 lr: 0.0006 max_mem: 18165M
[08/07 16:25:37] d2.engine.hooks INFO: Overall training speed: 44998 iterations in 23:39:30 (1.8928 s / it)
[08/07 16:25:37] d2.engine.hooks INFO: Total training time: 23:41:47 (0:02:17 on hooks)
[08/07 16:25:38] d2.data.datasets.coco INFO: Loaded 5000 images in COCO format from ./datasets/coco/annotations/instances_val2017.json

[08/07 16:27:45] d2.engine.defaults INFO: Evaluation results for coco_val_rgb in csv format:
[08/07 16:27:45] d2.evaluation.testing INFO: copypaste: Task: bbox
[08/07 16:27:45] d2.evaluation.testing INFO: copypaste: AP,AP50,AP75,APs,APm,APl
[08/07 16:27:45] d2.evaluation.testing INFO: copypaste: 35.8365,55.3913,38.2575,17.0571,40.6716,50.1802

The settings different from the orginal settings are below:
SOLVER:
IMS_PER_BATCH: 32
BASE_LR: 0.06
WARMUP_FACTOR: 0.00033333
WARMUP_ITERS: 3000
STEPS: (15000, 25000)
MAX_ITER: 45000
CHECKPOINT_PERIOD: 5000

[chensnathan]

I checked the setting once again. The steps should be modified. The initial steps STEPS: (15000, 20000) are set for a batch size of 64. You should double them as you use a batch size of 32.

Try:

STEPS: (30000, 40000)

You could get reasonable results this time.

[yqjiang6]

It works! Thank you for your help!