Top Menu

Measuring agreement between different raters

Monday, October 28, 2013

Rating averages (mean values) are definitely useful and they are, above all, very easy to interpret and communicate. They provide a very simple measure of the overall ratings provided by different respondents. Their main drawback is that they do not convey the level of agreement (or disagreement) between the different raters.

Primalogik 360 uses graphs and statistics to provide better insight into the level of agreement/disagreement behind an average.

Statistics and Frequency Distribution graphs

Frequency distribution graphs

They are easy to read and interpret. A quick glance at a graph allows you see if there's a disagreement or not (responses are scattered through the rating scale). They also allow you to identify if there are responses on the extremes. We've added frequency distribution graphs to Primalogik 360 back in june. They are easily accessible through the 360-degree survey report and they are included in the PDF report as well.

Standard deviation and coefficient of variation

We've just added these two statistical measures as an additional tool for response analysis. Both the standard deviation and the coefficient of variation provide an indication of how far the different responses are from the average.

The standard deviation shows how much variation from the average exists. A low standard deviation indicates that the data points tend to be very close to the average (mean); a high standard deviation indicates that the data points are spread out over a large range of values. 

The coefficient of variation is a normalized measure of dispersion of the frequency distribution. The coefficient of variation is useful because the standard deviation of data must always be understood in the context of the mean of the data. In contrast, the actual value of the CV is independent of the unit in which the measurement has been taken, so it is a dimensionless number. [wikipedia]