altimis / confusion-matrix-for-mask-r-cnn Goto Github PK
View Code? Open in Web Editor NEWConfusion matrix for Mask R-CNN (Matterport implementation)
Confusion matrix for Mask R-CNN (Matterport implementation)
Hello Altimis,
You helped me a lot for calculating confusion matrix for Mask R-CNN. right now I trained YOLACT algorithm that is similar to Mask RCNN but I cannot create confusion matrix for that. Do you know how can I calculate it in the eval.py
the code link is here
https://github.com/dbolya/yolact/blob/master/eval.py
Thank you
Hey Altimis,
I used this repo to calculate the confusion matrix for classes (background and trees). I am really not sure about the totals in sum_lin and sum_col.
Here 56.79% is precision and 88.33% is recall for class trees right?
As the tp,fp, fn as per code are as follows:
As precision is calculated TP/(TP+FP) and Recall TP/(TP+FN).
Then I calculated mAP, mAR and F1 and got this
Shouldnt the mAP, mAR calculated and in confusion matrix be same or I am completely misunderstanding the orientation of false and trues in the matrix? Could you please help me understand it.
Many Thanks.
How can this code be used to compute the matrix and values for the entire dataset. For example, I used this code:
def confusion_matrix(dataset, model, cfg):
gt_tot = np.array([])
pred_tot = np.array([])
# mAP list
mAP_ = []
for image_id in dataset.image_ids:
# load image, bounding boxes and masks for the image id
image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(dataset, cfg, image_id)
# convert pixel values (e.g. center)
scaled_image = modellib.mold_image(image, cfg)
# convert image into one sample
sample = np.expand_dims(scaled_image, 0)
# make prediction
yhat = model.detect([image], verbose=0, mask_threshold=0.5)
# extract results for first sample
r = yhat[0]
# compute gt_tot and pred_tot
gt, pred = utils_confusion.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'])
gt_tot = np.append(gt_tot, gt)
pred_tot = np.append(pred_tot, pred)
AP_, precision_, recall_, overlap_ = mrcnn.utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"],
r["scores"], r['masks'])
mAP_.append(AP_)
tp, fp, fn = utils_confusion.plot_confusion_matrix_from_data(gt_tot, pred_tot, fz=18, figsize=(20, 20), lw=0.5)
del tp[0]
del fp[0]
del fn[0]
print("tp for each class :", tp)
print("fp for each class :", fp)
print("fn for each class :", fn)
confusion_matrix(val_set,model_inference,Icfg)
This gives the individual matrix+tp/fp/fn values for each image. How can I modify it so that I get the values/matrix of the combined dataset i.e. total number of fp,fn,tp.
Also is there a way to compute precision/recall from the entire dataset from this?
Confusion-matrix-for-Mask-R-CNN/utils.py
Line 346 in ea4134e
Hello
I copied your code to the end of my utils.py file.
I tried to calculate the mAP. I get this error:
AttributeError: module 'mrcnn.utils' has no attribute 'gt_pred_lists'
Your help will be appreciated.
Hi Altimis,
is there a way to implement True Negatives into the confusion matrix?
In my dataset, there are several images, that do not contain any object, that should be detected, and if the model correctly does not detect anything on that image, than I think that would count as a TN, right?
I would assume, this is only viable, if the TP,FP,TN,FN are considered imagewise and not detection wise, what do you think?
Thanks and best regards!
Hi, I don't understand really well where you setup the squares and nb of classes.
Is there some needs to change that into the utils.py function?
Hello @Altimis , I tried to use your code to generate confusion matrix , but I got the following error.
I have NUM_CLASSES = 1 + 4 (4 class and 1 background). please help me in resoving this issue .
Hello Altimis,
I am using Mask-RCNN with 11 classes. I have been able to generate a confusion matrix using your repo.
The issue is that I'm only getting TPs. The FPs and FNs are always 0 for both validation and test datasets. However, the mAP for both is 0.85 and 0.918 respectively.
I've plotted the predictions as well for verification. The model mispredicts some of the classes and sometimes, it does not predict all of the classes in an image.
I fail to understand the issue here. Could you tell me what could be the problem?
Hello @Altimis , I am getting a strange error when trying to print confusion matrix.Please help me in resolving this,
Could you please explain the illustration in the README a bit more? What do the green, what do the red values mean?
Hi there, I was able to plot the confusion matrix when I only have 1 class. However, when I have 2 different classes, I couldnt plot the confusion matrix using the code you given, could you tell me what was the problem? I have no idea what i should do to fix this issue....
the actual len of the pred vect is : 102
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 103
the actual len of the pred vect is : 103
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 104
the actual len of the pred vect is : 104
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 105
the actual len of the pred vect is : 105
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 106
the actual len of the pred vect is : 106
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 107
the actual len of the pred vect is : 107
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 108
the actual len of the pred vect is : 108
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 109
the actual len of the pred vect is : 109
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 110
the actual len of the pred vect is : 110
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 111
the actual len of the pred vect is : 111
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 112
the actual len of the pred vect is : 112
The actual mean average precision for the whole images (matterport methode): 1.0
the actual len of the gt vect is : 113
the actual len of the pred vect is : 113
The actual mean average precision for the whole images (matterport methode): 1.0
ValueError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in create_block_manager_from_blocks(blocks, axes)
1670 blocks = [
-> 1671 make_block(values=blocks[0], placement=slice(0, len(axes[0])))
1672 ]
6 frames
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/blocks.py in make_block(values, placement, klass, ndim, dtype)
2743
-> 2744 return klass(values, ndim=ndim, placement=placement)
2745
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/blocks.py in init(self, values, placement, ndim)
130 raise ValueError(
--> 131 f"Wrong number of items passed {len(self.values)}, "
132 f"placement implies {len(self.mgr_locs)}"
ValueError: Wrong number of items passed 2, placement implies 3
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
in ()
74 #print the confusion matrix and compute true postives, false positives and false negative for each class:
75 #ps : you can controle the figure size and text format by choosing the right values
---> 76 tp, fp, fn = utils.plot_confusion_matrix_from_data(gt_tot, pred_tot, dataset_test.class_names, fz=18, figsize=(20,20), lw=0.5)
77
78 print("mAP: ", np.mean(APs))
/usr/local/lib/python3.7/dist-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py in plot_confusion_matrix_from_data(y_test, predictions, columns, annot, cmap, fmt, fz, lw, cbar, figsize, show_null_values, pred_val_axis)
354
355 #plot
--> 356 df_cm = DataFrame(confm, index=columns, columns=columns)
357
358 pretty_plot_confusion_matrix(df_cm, fz=fz, cmap=cmap, figsize=figsize, show_null_values=show_null_values,
/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py in init(self, data, index, columns, dtype, copy)
495 mgr = init_dict({data.name: data}, index, columns, dtype=dtype)
496 else:
--> 497 mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
498
499 # For data is list-like, or Iterable (will consume into list)
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/construction.py in init_ndarray(values, index, columns, dtype, copy)
232 block_values = [values]
233
--> 234 return create_block_manager_from_blocks(block_values, [columns, index])
235
236
/usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in create_block_manager_from_blocks(blocks, axes)
1679 blocks = [getattr(b, "values", b) for b in blocks]
1680 tot_items = sum(b.shape[0] for b in blocks)
-> 1681 raise construction_error(tot_items, blocks[0].shape[1:], axes, e)
1682
1683
ValueError: Shape of passed values is (2, 2), indices imply (3, 3)
Hello Altimis,
My train data=476, Val data=95
Received : mAP =1, Precision=75% recall =63%
NUM_CLASSES = 1 + 2 (2 class and 1 background)
I referred your code for confusion matrix, I am confuse what is this confusion matrix is illustrating in percentage, can you please tell inferences of below confusion matix?
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.