Development and Psychometric Validation of the Non-Academic Qualities (NAQ) Scale and Companion Vignette Battery for MBBS Graduates: A Multi-Phase Instrument Validation Study

Abstract

Background: Non-academic qualities such as professionalism, communication, ethical reasoning, resilience, and self-management are increasingly recognised as essential determinants of physician competence. However, no existing instrument combines a structured, multi-domain importance scale with vignette-based behavioural validation.

Objectives: This study aimed to develop and psychometrically validate the 24-item Non-Academic Qualities (NAQ) scale and its companion five-vignette battery for assessing non-academic qualities of MBBS graduates.

Methods: A six-phase design was employed: expert content validity assessment (n = 10), pilot testing of clarity and relevance (n = 50), internal consistency analysis, exploratory factor analysis (EFA), test-retest reliability, and expert inter-rater agreement on the vignette scoring key.

Key Results: All 24 items achieved an Item-Content Validity Index (I-CVI) ≥ 0.80 (Scale-Content Validity Index, S-CVI = 0.82). Pilot testing confirmed excellent face validity (mean Clarity Index = 0.958; mean Relevance Index = 0.936). Overall Cronbach's alpha was 0.939, indicating strong internal consistency. EFA yielded a five-factor solution explaining 66.9% of total variance, closely mirroring the a priori domain structure. Test-retest reliability was good (overall ICC = 0.834), and expert agreement on the vignette scoring key was almost perfect (Fleiss' kappa = 0.823).

Conclusion: The NAQ scale and its vignette battery demonstrate strong, convergent evidence of validity and reliability, supporting their use as a methodologically sound instrument for assessing non-academic qualities of MBBS graduates.