The modern approach to statistical analysis of MCQ items: Quality testing

The traditional model of scoring MCQ tests assumes that all items have similar weight and that the sum of items correct is a good indicator of student achievement. However, this approach has been abandoned in standardised selection and diagnostic tests in K-12 schooling and higher education in favour of the item response theory model. The IRT model weights items according to their relative difficulty and probabilistically estimates a score that reflects the difficulty point at which the candidate will get 50% of items correct. This score (usually ranging -3.00 to +3.00) has to be evaluated as to its merit by the test instructor in light of the kind of items associated with each grade range.

This seminar will overview the statistical properties of the classical and modern test theory approaches and describe how item information can be used to evaluate the quality of instructor-made tests, as well as student achievement. A Learning Enhancement Grant funded software system (in alpha version) will be demonstrated to illustrate the strengths and challenges of the modern analytic approach.