ENH: inter_rater.fleiss_kappa p-values and confidence interval

[josef-pkt]

ENH: inter_rater.fleiss_kappa p-values and confidence interval

[jseabold]

I needed this today as well coincidentally, so coded something up based on Fleiss, Nee, and Landis (1979) "Large sample variance of kappa in the case of different set of raters." Equation 3 in this paper (which says don't do it). This is what stata uses. If the number of raters is not the same for each subject, they don't produce anything for inference.

def fleiss_standard_error(table):
    n, k = table.shape  # n_subjects, n_choices
    m = table.sum(axis=1)[0]  # assume they all have the same ratings count
    p_bar = table.sum(axis=0) / (n * m)
    q_bar = 1 - p_bar

    return (
        (2 ** .5 / (p_bar.dot(q_bar) * np.sqrt(n * m * (m - 1))))
        * (
            (p_bar.dot(q_bar) ** 2) - np.sum(p_bar * q_bar * (q_bar - p_bar))
        ) ** .5
    )