Intra- and interobserver agreement study of a grading system for behavioural observation in ridden sports horses

Assessment of unrideability ("Unrittigkeit") is mostly subjective, so that a grading scheme was developed in the present study, which records and evaluates well defined signs individually, to allow a more objective graduation of rideability of horses. The purpose of the present study was to apply this grading by different clinicians. For this prospective study, 30 Warmblood horses of which 15 were presented due to rideability issues were repeatedly presented under the rider and assessed by different clinicians. To ensure that the clinical examination was as objective as possible, it was performed blinded on three consecutive days by three experienced clinicians at two different times under controlled conditions. The clinical examination included a complete orthopaedic examination and the evaluation of the horse on the hand and under the rider. The presence of lameness led to exclusion from the study. All horses were ridden by their standard rider according to a standardized protocol and evaluated blinded by the three clinicians. The degree of severity of the signs was allocated to scores. For example, "tail swishing" and "restless ears" were rated with 1 point, "head position high" or "twisting the head and neck" with 8 points and "rearing" was rated with 10 points. Their score resulted in the degree of "Unrittigkeit". The sum of the individual points appeared in 5 degrees of "Unrittigkeit" for the horses. Grade 1 represented a very low and grade 5 a very high level of "Unrittigkeit". The number of these behavioural responses shown by a horse was also relevant and the sum of all signs was considered. A cut-off value was defined as 2 points, so a horse with at least 2 points was considered as "unrittig". In the following days, the horses were presented under the saddle by a reference rider according to a standardized protocol and evaluated again blinded by the three clinicians. If the horses showed signs of "Unrittigkeit" under the familiar rider and under the reference rider, another performance under the reference rider was evaluated blinded by the three clinicians. In summary at least 2 examinations by the 3 clinicians under two different riders were available for the evaluation. In none of the horses did the clinicians diverge in their individual assessment. There was excellent agreement (kappa = 1.0) for allocation of the horses to the two groups. For the majority of signs at least a moderate agreement was shown. Some signs ("teeth grinding", "reduced flexion and bending" and "nose-line in front of the vertical not possible") showed an excellent interobserver agreement (kappa = 0.82-1.0). The current study shows that with the presented grading scheme a straightforward differentiation between a "rideable" and an "unrideable" horse is feasible in practice. In particular, the degree of impaired performance under the saddle can be reproducibly determined during the veterinary examination of a ridden horse.