Rho for Inter Rater Reliability
Convert codeset to contingency table
Convert a codeset to a contingency table
Calculate Baserate
Calculate Baserate (CT)
Calculate Baserate (Set)
Helper function to return special values on a rating set
contingency_table
Create Simulated codeSet
generate_kp_list
getBootPvalue_c
getHand_kappa
Get Handset
Get Handset
Generate a Handset
Get Test Set
kappa_ct
Calculate kappa
Calculate kappa (contingency Table)
Calculate kappa (Set)
random_contingency_table
recall
Rho using a file
Rho
Rho (contingency Table)
Rho (kappa)
Rho Min
rhoR: A package for computing rho
Rho (set)
sample_contingency_table
Rho is used to test the generalization of inter rater reliability (IRR) statistics. Calculating rho starts by generating a large number of simulated, fully-coded data sets: a sizable collection of hypothetical populations, all of which have a kappa value below a given threshold -- which indicates unacceptable agreement. Then kappa is calculated on a sample from each of those sets in the collection to see if it is equal to or higher than the kappa in then real sample. If less than five percent of the distribution of samples from the simulated data sets is greater than actual observed kappa, the null hypothesis is rejected and one can conclude that if the two raters had coded the rest of the data, we would have acceptable agreement (kappa above the threshold).
Useful links