Introduction Otoscopy is a key clinical skill which following the introduction of the Medical Licensing Assessment all newly qualified doctors in the United Kingdom will be required to be able to perform independently. At present there is no consensus on a standardised method for objectively assessing otoscopy skills. Methods Prospective mixed methods study comparing face, content and construct validity of three different platforms for otoscopy skills assessment, using a traditional otoscope with manikin, digital otoscope (Tympahealth) with manikin, and traditional otoscope with a low-cost model ear (SimEar). Skills were assessed using a standardised mark scheme, and five expert assessors and twelve participants who rotated through three Objective Structured Clinical Examination (OSCE) stations representing each model. Assessors numerically ranked validity of each model, and participated in a semi-structured interview of opinion. Results Each platform differed in face, construct and content validity scores, with no one platform consistently outperforming others. Three main themes were identified during thematic analysis of expert assessor interviews: ability to assess what is seen, anatomical reality, and ease of use. The low-cost model showed greatest potential, where modification to include a silicone ear could lead to high validity with marginal increase in cost. Conclusion Several modalities for assessing otoscopy skills exist, each with advantages and disadvantages. Modifications to a low-cost model, for use with either a traditional or digital otoscope, could prove to be the best model.

Key Points: 1. Efforts to combat the large global burden of ear and hearing disorders are hampered by poor availability of expert diagnosis 2. We report the first study to directly compare prediction model, non-expert and tele-diagnosis of middle and external ear disorders. 3. A prediction model based upon a novel automated otological symptom questionnaire performed poorly, but absence of otorrhoea was found to reliably exclude a diagnosis of chronic suppurative otitis media. 4. Both on-site non-expert and expert tele-diagnosis had high diagnostic specificity, but low sensitivity. 5. Future work could explore how the validity of these diagnostic methods may be improved.