Designing p-optimal item pools in computerized adaptive tests with polytomous items

Current CAT applications consist of predominantly dichotomous items, and CATs with polytomously scored items are limited. To ascertain the best approach to polytomous CAT, a significant amount of research has been conducted on item selection, ability estimation, and impact of termination rules based on polytomous IRT models. Few studies investigated the optimal pool characteristics for polytomous CAT implementation. Using the generalized partial credit model (GPCM) (Muraki, 1992), this study aims to identify an optimal item pool design for tests with polytomous items by extending the p-optimality method (Reckase, 2007). The extension includes the definition of a&thetas;-bin to describe polytomous items succinctly for pool design, and item generation strategy using constrained nonlinear optimization method. Optimal item pools are generated using CAT simulations with and without practical constraints. The item pool characteristics under each condition are summarized and their performance is evaluated against an extended operational item pool. The results indicated that the practical constraints of the a-stratified exposure control and content balancing do not affect pool size to a large extent. However, the a-stratified control affects the pool characteristics greatly: the items included in the simulated pools with the control have larger a-parameter and provide higher maximum information on average. On the other hand, the content balancing applied in this study has little impact on pool design. The evaluation results of the pool performance are closely related to the pool characteristics. When the a-stratified exposure control applied, the consistent results include: 1) the average test information is lower than that without the constraint; 2) RMSE is higher and the correlation between the true and estimated abilities is lower; 3) the percentage of the correct classification for the highest achievement level is lower. However, for all the simulated optimal pools, while the test information is consistently above the target level 10.0, it is concluded that the a-stratified method resulted in an efficient use of less discriminative items with small decreases in measurement precision. With regard to the item pool usage, the percentage of items that are fully used, well used, rarely used, and never used are quite comparable for the pools designed with constraints. However, compared with the extended operational pool, when the a-stratified method applied, the conditional test overlap rate of the simulated pools is consistently lower. For the pool blueprint, because the normal ability distribution is assumed, more items are included in the middle of the ability scale for all simulated optimal pools. The distributions of the a- and b- parameters are similar under all conditions. When the a-stratified method applied, there are fewer items in the first stratum and the number of items in the second and third stratum is nearly identical. In addition, with the content balancing control, the number of items in the first content area is slightly less than the second one.