This study built and supported a validity argument for a diagnostic reading test, the UDig reading test. The test aims to diagnose the reading ability of English language learners at the tertiary level in China. Drawing on the interpretive/use argument approach (IUA; Kane, 1992, 2001, 2013), this study specified three validity inferences which were of primary concern in the IUA for the UDig: the explanation inference, the decision inference, and the consequence inference. Corresponding to these inferences, three research questions were proposed:
1. To what extent are UDig reading scores accurate indicators of test-takers’ strengths and weaknesses of reading subskills?
2. How do students and teachers perceive the usefulness of UDig diagnostic feedback for making decisions on learning and teaching?
3. How is UDig diagnostic feedback used by students and teachers and what consequences does it have on learning and teaching?
        Employing a mixed-methods research design, the study triangulated multiple sources of evidence to support the validity inferences over two research phases. Phase 1 of the study explored the extent to which UDig scores could be interpreted as indicators of students’ mastery of reading subskills. Students’ think-aloud protocols while taking the UDig test and experts’ judgments were elicited to shed light on the associations between the test items and the reading subskills being measured. Based on the results, a Q-matrix denoting the item-by-subskill relationships was constructed and compared with the initial Q-matrix based on the test developer’s intention. The Q-matrix with a better model fit was validated and modified. Students’ mastery profiles were then generated using the G-DINA framework (de la Torre, 2011). Phase 2 investigated the ways in which the diagnostic feedback could promote remedial learning and teaching. At the beginning of an instructional semester, student questionnaire was distributed and the first round of student and teacher interviews was conducted, which provided insights into students’ and teachers’ perceptions of UDig feedback and their decisions on future use of the feedback. Situated within an Academic English Reading and Writing course, a longitudinal study comprising data from students’ and teachers’ reflective tasks and the second round of interviews was then adopted to document how students and teachers utilized UDig feedback throughout the semester, as well as the consequences of the feedback. Overall, the results provided strong backing for the validity inferences, whereas rebuttal evidence which posed threats to the validity of the UDig was also identified: while the majority of the intended reading subskills were supported by empirical data, a few modifications needed to be made regarding the item-by-subskill relationships; in general, students and teachers expressed positive views of the diagnostic feedback and showed willingness to utilize the information; most of the students and teachers integrated the diagnostic feedback into their learning and teaching process, which in return, exerted positive influences over students’ English study; a few deficiencies of the diagnostic feedback were pointed out, and detailed recommendations on how to refine the diagnostic feedback as well as the diagnostic procedure as a whole were given by students and teachers.