Intra-Operator Reliability Study Part 1: Devon vs Dulwich Tuskers (2011)

• Facebook

• Instagram

• Twitter

• Mail

• YouTube

• WordPress

Introduction

Every measurement tool should be valid and reliable, i.e. it should measure what it says and it should measure accurately in a consistent manner, respectively.  If either condition is not fulfilled then subsequent analyses will be faulty. 

This study investigated the degree of intra-operator reliability (the operator being the author) of one volleyball match. 

Method

Point ending actions data of one match (Devon versus Dulwich Tuskers, 3 October 2010) was collected.  I used NCAA’s (2010) operational definitions of volleyball point ending actions.  V₁ values were manually collected.  V₂ values were collected using software (LongoMatch).  After a familiarisation trial of using LongoMatch, the second trial’s data was used to compare with the original manual collected data.  As recommended by Hughes (2008), the following formula is used to calculate error percentage:

(∑ (mod (V₁-V₂)) / V_mean) × 100%

Where V₁ and V₂ are variables, V_mean their mean, mod = modulus and ∑ means ‘sum of’. 

Results

Dulwich Tuskers	V1	V2	Match Totals	V1	V2
Service Ace	2	2		Service Ace	14	13
Kill	21	21		Kill	63	62
Block Solo	4	5		Block Solo	5	6
Block Assist	1	0		Block Assist	2	2
Service Error	1	1		Service Error	5	5
Reception Error	12	11		Reception Error	14	13
Attack Error	17	15		Attack Error	39	35
Blocking Error	0	1		Blocking Error	1	1
Assist Error	2	0		Assist Error	2	0
Ball Handling Error	0	0		Ball Handling Error	0	0
∑ (mod (V1-V2)		8		∑ (mod (V1-V2)		10
∑ (V1-V2)/2		58		∑ (V1-V2)/2		141
% Error Overall		13.79		% Error Overall		7.09

Discussion

A satisfactory degree of agreement is set below 5%.  It can be seen that the % Error Overall values are above this agreement value at 13.79% and 7.09% for Dulwich Tuskers and Match (both teams) Totals respectively.  Thus, intra-operator reliability is unsatisfactory. 

However, further analysis showed that 3 manual collected data (V₁) values were technical errors due to operator inexperience and not fully understanding operational definitions.  When these 3 values were accounted for, % Error Overall was recalculated to show 5.31% and 4.29% for Dulwich Tuskers and Match Totals, improving intra-operator reliability.

Conclusion

The mini study showed that intra-operator reliability was unsatisfactory.  When technical errors were accounted for, intra-operator reliability improved to the point where Match Totals became reliable data (>0.5) and Dulwich Tuskers data was not far behind at 0.53.

This study shows that the volleyball statistics collected for Dulwich Tuskers during the 2010/11 season may be wholly unreliable.  However, the data used in this study was the first match of the season and operator inexperience in manual data collection may have played a decisive factor.  If the intra-operator reliability study is extended to include the whole season, the author believes the data is sufficiently reliable (>0.5) to perform subsequent analyses with confidence.  

References

Hughes, M. (2008) “Analysis of Notational Data: Reliability” in Hughes, M. and Franks, I.M. (eds) “The Essentials of Performance Analysis: An Introduction”, Routledge, Oxon

NCAA (2010) “Volleyball Statisticians’ Manual” [online]. Available from: http://fs.ncaa.org/Docs/stats/Stats_Manuals/VB/2010reg.pdf [Accessed 19 August 2011]

*** Update 1 February 2021 | I recently did an intra-operator reliability study which you can read on this link.  This is where I statted the same match twice, months apart, and compared the two timeline of events.  This is a better way to test reliability compared to a simple error percentage because two errors can cancel each other out ***