Introduction: Orthotic and prosthetic clinicians need to self-assess their performance and make decisions about practice limitations and continuing education throughout their careers. The objectives of this study were to evaluate the internal consistency of a self-assessment survey, examine the latent common factors and responsiveness of the survey, and use the results of the analysis to develop a shortened self-assessment survey.
Methods: Residents completed a self-assessment 29-item survey based on the six domains of the American Board for Certification in Orthotics, Prosthetics, and Pedorthics (ABC) Practice Analysis of Certified Practitioners in the Disciplines of Orthotics and Prosthetics multiple times during the clinical residency. Internal consistency was examined through Cronbachα. Confirmatory and exploratory factor analyses were used to examine latent common factors and inform item reduction. Responsive- ness was examined through repeated-measures analysis of variance (ANOVA).
Results: Analysis revealed a Cronbachαof of 0.927. Confirmatory factor analysis indicated a poor fit of the six-factor model. The final and best-fitting model suggested four latent common factors: patient-centeredness, regulatory awareness, device evaluation, and professional responsibility. Review and revision of the items resulted in a 14-item instrument. The instrument was responsive to changes over time.
Conclusions: The results indicate a need to reexamine the current clinical practice framework in orthotics and prosthetics, particularly as it relates to clinical competence. Future research should evaluate the performance of the shortened self-assessment survey and consider implications for educational standards.
Clinical Relevance: The use of validated educational tools can strengthen the efficacy of assessments in clinical education and residency. Improved clinical education assessments may result in standardization of competency and improved graduate preparedness. (J Prosthet Orthot.2022;00:00–00)
The ability to self-assess competency is a crucial component of a health care provider, both during training and throughout a clinical career.1The National Commission on Orthotic and Prosthetic Education (NCOPE) defines competency as“a specific range of skill, knowledge, and ability to do something, especially measured against a standard.”2Implementing competency-based education requires attention to the process of competency identification, competency components, competency evaluation, and assessment of the competency process.3 This process is fundamentally tied to the use of objective and validated measures through which competency can be assessed. Ultimately, curricula centered around such measures can lead to directed learning and growth, not just at the graduate level but throughout a professional career.4–6
Research in self-assessment has demonstrated limitations in the accuracy of self-assessment.7,8 Although some studies have demonstrated a lack of correlation between trainee self-assessment and supervisor ratings,9,10 some education researchers have found that trainee self-assessment can align with supervisor ratings.11,12 Perhaps more important than the accuracy of a self-assessment is a resident's ability to reflect upon his or her own performance. Self-assessment has been shown to increase a learner's ability to identify strengths and weaknesses when determining how to improve13 and can be a valuable part of a robust self-regulatory process.14
Currently, no validated tools exist for assessing clinical competency and informing experiential learning in orthotics and prosthetics clinical education. One method to analyze the validity of an instrument is to perform confirmatory factor analysis and evaluate the goodness of fit of the model.15 Factor analysis involves statistically evaluating the relationships between items on a survey and using interpretation to determine the overarching concept relating to those items. Kane's validity framework,16which has been applied to the process of evaluating clinical competency and medical skills,17 offers a structure for examining an assessment tool within the context of statistical evaluation. This framework addresses the importance of the statistical analysis and the contextual or practical meaning of the survey or assessment.
The purposes of the study were to 1) evaluate the internal consistency of a competence self-assessment tool and the items contained within it, 2) examine the latent common factors being assessed by the tool, and 3) develop a refined survey with reduced items. Through the lens of Kane's validity framework,16 the researchers explored the quality of a competency self-assessment tool used in orthotics and prosthetics clinical education, specifically its internal consistency, factor structure, and responsiveness. The aim of these analyses was to generate a novel self-assessment survey, which may be useful for clinical education in orthotics and prosthetics.