CHAPTER 1
How to Conceptualize and Implement a Language Assessment

Lyle F. Bachman and Barbara Damböck

Introduction

Language assessments have been pervasive in human societies for centuries. They have played a vital role in shaping the nature of societies, while at the same time reflecting the values of those societies. Although language assessments have taken many different forms, have been implemented with many different kinds of test takers, and have been used for many different purposes, they can all be conceptualized in essentially the same way. In this chapter, we first provide a brief historical overview of language assessments over the centuries and present what we believe to be the central conceptualization that informs all language assessments, both in their design and development and in their use. We then describe an approach to language assessment that is based on the work of Bachman and Palmer (2010) and Bachman and Damböck (2018) and that we believe instantiates the central conceptualization of language assessment. We describe the process of using language assessments to help us make decisions that will have beneficial consequences. Next, we discuss fairness and accountability in assessment and the process of assessment justification. We then discuss the process of designing and developing language assessments, using an example of a language assessment to illustrate this.

Brief Historical Overview of Language Assessments

The oldest and longest continuously implemented language assessment on record is the Chinese Imperial Civil Service Examination,1 which was initiated during the Sui dynasty (581-618) and lasted over a thousand years, until it was abolished in 1905, toward the end of the Qing dynasty (1644-1911) (Weerdt, 2022).2 The initial purpose of the examination was to select exceptionally talented young men to serve as senior bureaucrats by merit, rather than by birth. As the examination evolved over the years, it became a tool for selecting government officials who were loyal to the emperor, thus countering the influence of the military and the aristocracy on the imperial government.

The content of the examinations was based on the classic texts of Confucianism, i.e., The Analects and Mencius. However, they assessed not only a young man's knowledge of Confucian philosophy but also his writing and calligraphy, formal essay writing techniques, mathematics, law, government, poetry, and clear and coherent speaking (Cartwright, 2019). In the later years of the examinations, beginning during the Ming dynasty (1368-1644) and continuing into the Ching dynasty, the focus became more and more on language, especially the candidates' mastery of the "eight-legged" essay, an elaborate structure of formal writing which candidates were expected to follow strictly (Elman, 2009). Candidates' written answers were double marked, once by an associate examiner and once by a senior examiner, using a system of symbols indicating the quality of the answer. Only when both examiners agreed on the mark was the pass/fail decision made (O'Sullivan & Cheng, 2022).

In the earliest European universities, which were established during the 11th through 15th centuries, there were no formal "examinations." Students were assessed through the lectures they gave and through a series of "disputations," in which a student engaged in a formal debate with another student or a master. In these disputations, the student was assessed according to how well he demonstrated his ability to plausibly and convincingly argue a particular position according to Aristotle's syllogistic rules of argumentation. These disputations were conducted in Latin and covered the content of the university's curriculum, which included the writings of Aristotle and Euclid, as well as the Trivium (grammar, rhetoric, and logic), the Quadrivium (arithmetic, geometry, music, and astronomy), and the three philosophies (natural, moral, and metaphysical). Once students passed all of their disputations, they were considered to be "masters" and could take on students of their own (Rashdall, 1975).

It was not until the 18th century that formal written examinations were introduced in Europe. The first of these, the German Abitur, was introduced in Prussia in 1788 and later established in the other German states. Prior to this, each university in Germany had its own entrance examination, and the government wanted to standardize this to assure that all students accepted to German universities had the same qualifications. Although the Abitur served initially as a university entrance qualification, over time it also came to be used as a school-leaving examination. The written examination assessed both old languages, e.g., Latin and Greek; new languages, e.g., French; and the mother tongue, German. It also assessed knowledge of the sciences, especially history (Bölling, 2010). In 1808, the French Baccalauréat was established by Napoleon Bonaparte as a way to provide the opportunity for anyone with talent or intelligence to pass its rigorous exams. The original Baccalauréat assessed test takers' knowledge in the areas of science, literature, law, medicine, and theology (Villinger, 2018). Both the Abitur and the Baccalauréat continue to be administered up to the present day.

The late 18th century saw the rapid emergence of secondary school-leaving examinations in the United Kingdom. These were administrated largely by agencies within universities, e.g., the Royal Society of Arts Examinations Board (est. 1856), the University of Oxford Delegacy of Local Examinations (est. 1857), and the University of Cambridge Local Examinations Syndicate (est. 1858). All of these examinations aimed at the general goal of improving society. The purpose of the Cambridge examinations, for example, was to "raise standards in education by administering exams for people who were not members of the University and inspecting schools" (Cambridge Assessment International Education, 2023b). Similarly, the Oxford examinations were intended to "confer a great benefit on that large class of persons who cannot afford, or do not require, a University education for their children, by undertaking to examine boys, about the time of their leaving school" (Cambridge University Libraries, 2023). These examinations covered a wide range of academic subjects, including, e.g., drawing, English language, English literature, French, geography, German, Greek, history, Latin, law, mathematics, religious knowledge, and zoology (Cambridge Assessment International Education, 2023a).

Standardized tests began to appear in the United States in the mid-18th century. It is important to note that "standardized" at that time meant simply that "the tests were published, that directions were given for administration, that the exam could be answered in consistent and easily graded ways, and that there would be instructions on the interpretation of results" (U.S. Congress Office of Technology Assessment, 1992, p. 108). The first such test was a written examination initiated in the Boston Public School system in 1845 to replace oral examinations, which were seen as subjective and were expensive to administer. This written examination was intended to be used for two purposes: (i) classify schoolchildren according to how much they had learned and (ii) monitor the effectiveness of the school system (U.S. Congress Office of Technology Assessment, 1992). In 1865, the first preliminary Regents Examinations were administered to eighth-grade students in the state of New York; the High School Regents Examinations were administered for the first time in 1878. These examinations were intended to inform the distribution of state funding to encourage academic education. Although only 5 subjects, algebra, American history, elementary Latin, natural philosophy, and physical geography, were assessed in the first, by 1879 the list was expanded to 41 subjects, including rhetoric and English composition, Greek, French, and German (New York State Education Department, n.d.).

Testing for university and college admissions began in the United States in 1901 with the administration of the College Entrance Examination Board's first examinations. These "College Boards" consisted of essay examinations in English, French, German, Latin, Greek, history, mathematics, chemistry, and physics (Lawrence et al., 2003). Over the years, these subjects were greatly expanded to include literature, reading, and writing and became a part of the family of Scholastic Aptitude Tests (SATs), first as SAT Achievement Tests, and eventually as the SAT II (Atkinson & Geiser, 2009; Lawrence et al., 2003).

Probably the most influential assessment of speaking ability in the United States, in terms of its prestige and range of use, is the Oral Proficiency Interview (OPI), which is used both within numerous U.S. government agencies and by colleges, universities, and high schools as a measure of speaking ability. This assessment had its origins in the Foreign Service Oral Proficiency Interview that was developed in the 1950s.3 By 1956, this had been officially adopted by the U.S. Department of State for new employees and became mandatory for all Foreign Service Officers in 1957. Subsequently, it was adopted by a number of other U.S. government agencies and was renamed as the...