Design, Analysis and Reporting in Large-scale Assessment

This monograph is a technical explication of the relationships between data-gathering methods and reporting methods in large-scale educational assessments, with the focus on 'marketbasket' reporting. Complex survey-based assessments such as the National Assessment for Educational Progress (NAEP) use advanced data-gathering designs such as student- and item-sampling. Analyzing and reporting the resulting data falls outside the methods of familiar every-pupil testing, and sophisticated models and analyses need to be carried out to bring the data to a common reporting scale. Marketbasket reporting, or the reporting of results in a form that can be understood in terms of scores on a tangible collection of actual test items, has been proposed as a way to make the results of complex surveys more accessible to consumers of the data. For the first time anywhere, this book fully explains the definitions, concepts, methods, and issues that underlie marketbasket reporting in a clear and rigorous presentation, and illustrates the ideas with actual data from NAEP. This knowledge is essential for sound policy decisions, assessment design, data analysis, and reporting.