Data of a science knowledge test: Simulation of item preknowledge

Context

If test items are known in advance (item preknowledge) the validity of a test is in jeopardy. How can we know if this has happened?

 

Methods

In order to simulate how data would look like if some test takers possessed item preknowledge we compromised two original datasets: Responses to a predetermined set of items were set to correct for 10% of test takers, randomly selected. In reruns, the proportion was varied to be 5%, 2.5%, and 1%. Item difficulty, item discrimination, item fit, and local dependence were computed using an IRT 2PL model. Then changes in these item properties from the non-compromised to the compromised dataset were scrutinized for their sensitivity to item preknowledge.

 

Data

In each file the responses to a test with 80 items are given (0 = not correct / 1= correct). The compromised datasets (that contain simulated test takers with item preknowledge) are flagged with the suffix _pX, where X desribes the prevalence of item preknowledge.