Investigating the Construct Validity of a Concordance-based Cloze Test: A Mixed-methods Study

This study has two primary goals. The first is to develop a concordance-based cloze test (henceforth ConCloze), the process of which can be divided into seven stages, ConCloze 1–7. The second goal is to investigate the validity of score interpretations, which can be categorized into five aspects according to Messick’s (1995) model of construct validity: content, substantive, structural, generalizability, and external. By validity, Messick (1989: 13) referred to an integrated judgment of adequacy and appropriateness of construct-related inferences based on, e.g., item responses, observations. The sampling methods are convenience and snowball samplings, seeking non-native English speakers of mixed backgrounds in first language, who are studying in or have graduated from higher education. The analytical measures include reliability analysis, verbalization analysis, usability testing, correlation analysis, content analysis, regression analysis, Rasch modeling, sample analysis, and thematic analysis. The construct domain is found to be very likely composite: at least lexical-semantic knowledge, knowledge of synonymy, knowledge of collocation, knowledge of grammatical structure, world knowledge, knowledge of word association, knowledge of semantic prosody, and knowledge of individual and compositional lexical-semantic content are tested. The test purpose is initially set to be a proficiency test on professional and academic English grammatical and vocabulary use, which is eventually refined in light of empirical findings in the investigation into a proficiency test on academic English vocabulary use, with the primary domains of knowledge of lexical-semantics and knowledge of word association. The subdomains involved become knowledge of core components in word meaning, knowledge of individual and compositional lexical-semantic content, knowledge of collocation, and knowledge of semantic prosody, with world knowledge, synonymy knowledge, and knowledge of grammatical structure functioning as construct-peripheral. Judged by the adequacy and appropriateness of response and score interpretations, an integrated evaluation is that the construct inferences for the ConCloze item type have validity.