Selection Errors in Scale Development: A Monte Study of Factor Analysis

Abstract

Previous research had devoted a lot to the practice of exploratory and confirmatory factor analysis. Despite the large volume of reviews and simulations, little had been devoted to the decision of selecting and discarding items in scale development. The present research examined the impact of sample size, number of items, number of factors, ratio of strong indicators to weak indicators (i.e. items having strong / weak associations with the latent construct), and magnitude of weak loadings on the rates of occurrence of selection errors. A Monte Carlo simulation with a 6 × 3 × 2 × 3 × 3 design was conducted, and the real life practice of factor analysis in scale was examined. Results showed that when sample size was not large, the selection errors were not negligible, and selection errors also lead to reductions in population reliability. The utility of fit-indexes in identifying selection errors was also examined, with the model χ2 being relatively more informative. It was also found that a large sample confirmatory factor analysis did not compensate for the instability of a small sample exploratory factor analysis. Some suggestions regarding sample size and procedures in scale development were discussed, with a general urge for more conservative procedure and careful sample size planning.