A comparison of DIF detection and effect size measures among Mantel-Haenzel, SIBTEST, and Logistic regression using a science test data

Abstract

The Differential item functioning (DIF) occurs when individuals of the same a bility level from separate groups have different probability of answering an item correctly. This study was conducted in two parts: in the first part a real science test data was analyzed and the consistencies among the three DIF detection procedures were examined and in the second part, simulated data for a short test was used in comparing the consistencies among the procedures. In the first part, students’ response data from a large-scale science achievement test were analyzed for gender DIF. A sample of 2600 (1500 males and 1100 females) was randomly selected for this study. The Science test consisted of 49 items. DIF statistical analyses were conducted on the data using MH, SIBTEST, and LR. All test statistics were interpreted at an alpha-level of 0.05. For SIBTEST, the guideline developed by Roussos and Stout (1996) was employed. The study suggests that the different procedures provided consistent estimates on the magnitude and direction of DIF and thus supports the recommendation that multiple DIF detec tion procedures should be used in real testing situation to reduce the uncertainty associated with the analysis interpretation of empirical testing data.