A comparative evaluation of score results from computerized and paper & pencil mathematics testing in a large scale state assessment program

The present study reports results from a quasi-controlled empirical investigation addressing the impact on student test scores when using fixed form computer based testing (CBT) versus paper and pencil (P&P) testing as the delivery mode to assess student mathematics achievement in a state's large scale assessment program. Grade 7 students served as the target population. On a voluntary basis, participation resulted in 644 students being "double" tested: once with a randomly assigned CBT test form, and once with another randomly assigned and equated P&P test form. Both the equivalency of total test scores across different student groupings and the differential impact on individual items were examined. Descriptively there was very little difference in performance between the CBT and P&P scores obtained (less than 1 percentage point). Results make very clear that there existed no meaningful statistical differences in the composite test scores attained by the same students on a computerized fixed form assessment and an equated form of that assessment when taken in a traditional paper and pencil format. While a few items (9 of 204) were found to behave differently based on mode, close review and inspection of these items were not able to identified factors accounting for the differences.