.png)
Intra-Operator Reliability Study Part 1: Devon vs Dulwich Tuskers (2011)
Introduction
Every measurement tool should be valid and reliable, i.e. it should measure what it says and it should measure accurately in a consistent manner, respectively. If either condition is not fulfilled then subsequent analyses will be faulty.
This study investigated the degree of intra-operator reliability (the operator being the author) of one volleyball match.
Method
Point ending actions data of one match (Devon versus Dulwich Tuskers, 3 October 2010) was collected. I used NCAA’s (2010) operational definitions of volleyball point ending actions. V1 values were manually collected. V2 values were collected using software (LongoMatch). After a familiarisation trial of using LongoMatch, the second trial’s data was used to compare with the original manual collected data. As recommended by Hughes (2008), the following formula is used to calculate error percentage:
(∑ (mod (V1-V2)) / Vmean) × 100%
Where V1 and V2 are variables, Vmean their mean, mod = modulus and ∑ means ‘sum of’.
Results
Dulwich Tuskers | V1 | V2 | Match Totals | V1 | V2 | |
Service Ace | 2 | 2 | Service Ace | 14 | 13 | |
Kill | 21 | 21 | Kill | 63 | 62 | |
Block Solo | 4 | 5 | Block Solo | 5 | 6 | |
Block Assist | 1 | 0 | Block Assist | 2 | 2 | |
Service Error | 1 | 1 | Service Error | 5 | 5 | |
Reception Error | 12 | 11 | Reception Error | 14 | 13 | |
Attack Error | 17 | 15 | Attack Error | 39 | 35 | |
Blocking Error | 0 | 1 | Blocking Error | 1 | 1 | |
Assist Error | 2 | 0 | Assist Error | 2 | 0 | |
Ball Handling Error | 0 | 0 | Ball Handling Error | 0 | 0 | |
∑ (mod (V1-V2) | 8 | ∑ (mod (V1-V2) | 10 | |||
∑ (V1-V2)/2 | 58 | ∑ (V1-V2)/2 | 141 | |||
% Error Overall | 13.79 | % Error Overall | 7.09 |
Discussion
A satisfactory degree of agreement is set below 5%. It can be seen that the % Error Overall values are above this agreement value at 13.79% and 7.09% for Dulwich Tuskers and Match (both teams) Totals respectively. Thus, intra-operator reliability is unsatisfactory.
However, further analysis showed that 3 manual collected data (V1) values were technical errors due to operator inexperience and not fully understanding operational definitions. When these 3 values were accounted for, % Error Overall was recalculated to show 5.31% and 4.29% for Dulwich Tuskers and Match Totals, improving intra-operator reliability.
Conclusion
The mini study showed that intra-operator reliability was unsatisfactory. When technical errors were accounted for, intra-operator reliability improved to the point where Match Totals became reliable data (>0.5) and Dulwich Tuskers data was not far behind at 0.53.
This study shows that the volleyball statistics collected for Dulwich Tuskers during the 2010/11 season may be wholly unreliable. However, the data used in this study was the first match of the season and operator inexperience in manual data collection may have played a decisive factor. If the intra-operator reliability study is extended to include the whole season, the author believes the data is sufficiently reliable (>0.5) to perform subsequent analyses with confidence.
References
Hughes, M. (2008) “Analysis of Notational Data: Reliability” in Hughes, M. and Franks, I.M. (eds) “The Essentials of Performance Analysis: An Introduction”, Routledge, Oxon
NCAA (2010) “Volleyball Statisticians’ Manual” [online]. Available from: http://fs.ncaa.org/Docs/stats/Stats_Manuals/VB/2010reg.pdf [Accessed 19 August 2011]
*** Update 1 February 2021 | I recently did an intra-operator reliability study which you can read on this link. This is where I statted the same match twice, months apart, and compared the two timeline of events. This is a better way to test reliability compared to a simple error percentage because two errors can cancel each other out ***
Comments