Calibrating Standards-based Assessment Tasks for English as a First Foreign Language. Standard-setting Procedures in Germany
Description
This report is the second in a multi-part technical report series describing the development, calibration and validation of standards-based tests for English as a first foreign language at the Institute for Educational Progress (Institut zur Qualitätsentwicklung im Bildungswesen, IQB) in Berlin, Germany. It details the empirical basis of the calibration of the test item pool and criterion-referenced standard-setting procedures. The aim is to make transparent the decisions, methods and procedures which led to the setting of cut-scores in alignment with the National Educational Standards (NES) (i.e., the Länderübergreifende Bildungsstandards) and the Common European Framework of Reference for Languages (CEF) for English as a first foreign language. Standards were set for the lower secondary school level of Hauptschulabschluss and Mittlerer Bildungsabschluss.
This second report describes the process of relating the standards-based proficiency tests to the CEF levels, delineating the purpose of the tests, the aims of the standard-setting procedures, the rationale of the chosen procedures, and the use of the Manual for Linking Language Examinations to the CEF. This is followed by a detailed description of the two standard-setting methods employed, the Bookmark method and the computer-assisted Criterion Mapping method—developed and implemented by the Berkeley Evaluation and Assessment Research (BEAR) Center, University of California. This volume provides information on how the standard-setting sessions were conducted in collaboration with BEAR Center, and describes procedures, data gathering, and issues and problems which arose during the course of the study. The report concludes with a synthesis of the standard-setting study’s results, and discusses the implications of how these results are reported and presented to stakeholders and policy makers.
More details
Other editions
Additional editions
Content
2 - Preamble [Seite 8]
3 - Chapter 1: Setting Standards in Line with the Common European Framework of Reference [Seite 10]
3.1 - 1.1 Educational Standards and Standards-based Assessment [Seite 12]
3.2 - 1.2 Relating IQB Tests to CEF-levels [Seite 19]
3.3 - 1.3 Rationale for Selecting the Item Pool [Seite 27]
3.4 - 1.4 Description of Panelists [Seite 29]
3.5 - 1.5 Standard-setting Procedure at IQB [Seite 31]
3.6 - 1.6 Research into Validity of the Standard-setting Procedure ( DFG Research Grant) [Seite 35]
4 - Chapter 2: Standard- setting Item Pool [Seite 36]
4.1 - 2.1 Overview of Pilot Study Design [Seite 38]
4.2 - 2.2 Methodology [Seite 41]
4.3 - 2.3 Analyses for Reading and Listening Comprehension Scales [Seite 42]
4.4 - 2.4 Analyses for Written Expression Scale [Seite 46]
4.5 - 2.5 Descriptive Summary Statistics [Seite 48]
4.6 - 2.6 Results from Multi-faceted Rasch Analyses [Seite 51]
5 - Chapter 3: The Bookmark Standard- setting Method [Seite 56]
5.1 - 3.1 Background [Seite 58]
5.2 - 3.2 Bookmark Standard-setting Method [Seite 59]
5.3 - 3.3 Standard-setting Workshop Results [Seite 65]
6 - Chapter 4: The Standard- setting Criterion Mapping Method [Seite 68]
6.1 - 4.1 Background [Seite 70]
6.2 - 4.2 Preparation [Seite 73]
7 - Chapter 5: Procedural Standard- setting Issues [Seite 80]
7.1 - 5.1 Criterion Mapping [Seite 82]
7.2 - 5.2 Bookmark [Seite 92]
7.3 - 5.3 Summary and Conclusions [Seite 105]
8 - Chapter 6: Comparison and Synthesis of Multiple Standard- setting Methods and Panels [Seite 106]
8.1 - 6.1 Setting Educational Standards [Seite 108]
8.2 - 6.2 The Standard-setting Study [Seite 108]
8.3 - 6.3 General Conclusions [Seite 111]
8.4 - 6.4 Synthesizing and Reporting Results [Seite 111]
9 - Appendices [Seite 120]
9.1 - Appendix A [Seite 122]
9.2 - Appendix B [Seite 125]
9.3 - Appendix C [Seite 130]
9.4 - Appendix D [Seite 147]
9.5 - Appendix E [Seite 149]
9.6 - Appendix F [Seite 152]
9.7 - Appendix G [Seite 155]
9.8 - Appendix H [Seite 157]
9.9 - Appendix I [Seite 159]
