multi-gpu is unstable?

[blandocs]

multi-gpu is unstable?

[defqoon]

@zhoudongliang did you check that the number of classes in your config file is correct? I had the same bug and I fixed it by setting cfg.MODEL.ROI_HEADS.NUM_CLASSES for my Faster Rcnn model to the correct value

[ppwwyyxx]

I suggest you first try using the existing COCO configs on COCO dataset, to see whether there is anything special in your dataset that can cause the issue. This would make it much easier to isolate the potential causes.

The error messages you saw does indicate that the anchors may have a non-positive size. However, the way the anchors are generated (in anchor_generator.py and then rpn.py, and then rpn_outputs.py) should guarantee a positive size. You may need to track its generation to see how it can become non-positive.

[blandocs]

Hi there. Thank you for reply! I looked dataset carefully, but I can't find any differences between original coco dataset and my dataset. I described both below. Additionally, the bugs came out randomly. When I tried training, three bugs came out randomly as I mentioned in the article and Training suddenly succeeded during several attempts. It's so weird.

coco dataset as follows:

	"images": [{
		"license": 4,
		"file_name": "000000397133.jpg",
		"coco_url": "http://images.cocodataset.org/val2017/000000397133.jpg",
		"height": 427,
		"width": 640,
		"date_captured": "2013-11-14 17:02:52",
		"flickr_url": "http://farm7.staticflickr.com/6116/6255196340_da26cf2c9e_z.jpg",
		"id": 397133
	},

	"annotations": [{
		"segmentation": [
			[510.66, 423.01, 511.72, 420.03, . . . , 510.03, 423.01, 510.45, 423.01]
		],
		"area": 702.1057499999998,
		"iscrowd": 0,
		"image_id": 289343,
		"bbox": [473.07, 395.93, 38.65, 28.67],
		"category_id": 18,
		"id": 1768
	}, {
		"segmentation": [

my dataset as follows:

  "images": [
    {
      "file_name": "0763229.jpg",
      "width": 400,
      "id": 763229,
      "license": 3,
      "height": 600
    },

"annotations": [
    {
      "segmentation": [
        [
          146,
          363,
          154,
          373,
           . . . 
          147,
          349,
          144,
          363
        ]
      ],
      "area": 28951,
      "iscrowd": 0,
      "image_id": 763229,
      "bbox": [
        122,
        168,
        131,
        221
      ],
      "category_id": 9,
      "id": 10
    },    {
      "segmentation": [

[blandocs]

After I add "bbox_mode": 1 in my json file, above errors are resolved(My dataset has XYWH_ABS type of BBox). But I think there are still race conditions. Because cuda error sometimes occurs even if the code is same.

"Input boxes to Box2BoxTransform are not valid!"
AssertionError: Input boxes to Box2BoxTransform are not valid!

hello, I have also met the problem.
"Input boxes to Box2BoxTransform are not valid!"
AssertionError: Input boxes to Box2BoxTransform are not valid!
I checked my bbox, they are valid, could you provide me how to solve the first bug ?

[ChauncyFr]

After I add "bbox_mode": 1 in my json file, above errors are resolved(My dataset has XYWH_ABS type of BBox). But I think there are still race conditions. Because cuda error sometimes occurs even if the code is same.

I set this item in the json file, it is not possible, no matter the number of GPUs is 1 or 2. Who can answer me, when I train, this error always occurs and the training is interrupted, which is uncomfortable!