afergadis/irrcac

Chance-corrected Agreement Coefficients

irrcac's Issues

Metrics not working as it requires positional argument: 'confidence'

I am trying to construct a simple example for annotator agreement using e.g. Fleiss Kappa. I am doing the following:

from irrCAC.raw import CAC

rater_df = pd.DataFrame()
rater_df['annotator1'] = [0,0,0,1,0]
rater_df['annotator2'] = [0,1,0,1,0]

cac2raters = CAC(rater_df)
print(cac2raters.krippendorff())
print(cac2raters.fleiss())

However I keep getting the error rv_generic.interval() missing 1 required positional argument: 'confidence' no matter what statistic I make use off. What could be going on here?

Custom weights are not supported currently and documentation does not reflect it

Custom weights are not supported currently and the documentation does not reflect it. I keep getting:
ValueError: weights values can be any of ('identity', 'quadratic', 'ordinal', 'linear', 'radical', 'ratio', 'circular', 'bipolar')
when I supply an ndarray as a weight.

Interpretation of Results

How should the results be interpreted?

For example, in the following outputs of .gwet function:
What is the meaning of confidence value, interval, p_value, se, pa, z, etc?
Which of these variables tell me about gwet value?

{'coefficient_value': 0.118, 'coefficient_name': 'AC1', 'confidence_interval': (0.1021, 0.1339), 'p_value': 0.0, 'z': 14.91258, 'se': 0.00791, 'pa': 0.2766, 'pe': 0.17981}, 'weights': array([[1., 0., 0., 0., 0.],

Thank you.

Percent Agreement in docs is different when calculated with Kripendorff

First of all, thank you for porting the package to Python. Much appreciated.

That is not a bug that I have faced, but I noticed that the pa value for the Kruppendorff's alpha in the Raw Ratings Documentation is different from every other kappa. I also noticed that the formula in the code differs from the others.

My apologies for not having time to assert which formula is right or wrong. Thank you for checking that out.

Best,
Regards