Construct validity and cognitive diagnostic assessment

INTRODUCTION Cognitive diagnostic assessment (CDA) is increasingly a major focus in psychological and educational measurement. Instead of inferring a general response tendency or behavior consistency of an examinee over a target domain of measurement, diagnostic assessment results provide a detailed account of the underlying cognitive basis of the examinee's performance by mining the richer information that is afforded by specific response patterns. Sophisticated measurement procedures, such as the rule-space methodology (Tatsuoka, 1995), the attribute hierarchy method (Leighton, Gierl, & Hunka, 2004), the tree-based regression approach (Sheehan, 1997a, 1997b), and the knowledge space theory (Doignon & Falmagne, 1999), as well as specially parameterized psychometric models (De La Torre & Douglas, 2004; DiBello, Stout, & Roussos, 1995; Draney, Pirolli, & Wilson, 1995; Hartz, 2002; Junker & Sijtsma, 2001; Maris, 1999), have been developed for inferring diagnostic information. Although measurement models for diagnostic testing have become increasingly available, cognitive diagnosis must be evaluated by the same measurement criteria (e.g., construct validity) as traditional trait measures. With the goal of inferring more detailed information about an individual's skill profile, we are not just concerned about how many items have been correctly solved by an examinee. We are also concerned about the pattern of responses to items that differ in the knowledge, skills, or cognitive processes required for solution. Similar to traditional tests, empirical evidence and theoretical rationales that elaborate the underlying basis of item responses are required to support the inferences and interpretations made from diagnostic assessments.