TY - JOUR
T1 - Rater effects in creativity assessment
T2 - A mixed methods investigation
AU - Long, Haiying
AU - Pang, Weiguo
N1 - Publisher Copyright:
© 2014 Elsevier Ltd.
PY - 2015/3/1
Y1 - 2015/3/1
N2 - Rater effects in assessment are defined as the idiosyncrasies that exist in rater behaviors and cognitive process. They are composed of two aspects: the analysis of raw rating and rater cognition. This study employed mixed methods research to examine the two aspects of rater effects in creativity assessment that relies on raters' personal judgment. Quantitative data were collected from 2160 raw ratings made by 45 raters in three group and were analyzed by generalizability theory. Qualitative data were collected from raters' explanation of rationales for rating and their answers for questions about rating process as well as from 12 in-depth interviews and both were analyzed by framing analysis. The results indicated that the dependability coefficients were low for all the three rater groups, which were further explained by the variations and inconsistencies in raters' rating procedure, use of rating scales, and their beliefs about creativity.
AB - Rater effects in assessment are defined as the idiosyncrasies that exist in rater behaviors and cognitive process. They are composed of two aspects: the analysis of raw rating and rater cognition. This study employed mixed methods research to examine the two aspects of rater effects in creativity assessment that relies on raters' personal judgment. Quantitative data were collected from 2160 raw ratings made by 45 raters in three group and were analyzed by generalizability theory. Qualitative data were collected from raters' explanation of rationales for rating and their answers for questions about rating process as well as from 12 in-depth interviews and both were analyzed by framing analysis. The results indicated that the dependability coefficients were low for all the three rater groups, which were further explained by the variations and inconsistencies in raters' rating procedure, use of rating scales, and their beliefs about creativity.
KW - Creativity assessment
KW - Generalizability theory
KW - Mixed methods research
KW - Rater cognition
KW - Rater effects
UR - https://www.scopus.com/pages/publications/84912017601
U2 - 10.1016/j.tsc.2014.10.004
DO - 10.1016/j.tsc.2014.10.004
M3 - 文章
AN - SCOPUS:84912017601
SN - 1871-1871
VL - 15
SP - 13
EP - 25
JO - Thinking Skills and Creativity
JF - Thinking Skills and Creativity
ER -