More or less principal components often give an over-fit or under-fit quantitative calibration model. In order to avoid over-fit or under-fit in spectra calibration, a principal components selection method based on a modified randomization test is proposed. Three near infrared spectra experiments (the complexity of the sample components in each experiment is increasing by degrees) are introduced in this paper for evaluating the proposed method. The method is compared with the cross-validation method. And the spectra model complexity of how to affect the prediction performance of calibration is discussed. Then the adaptability of this modified randomization test to the uncertainty complex spectra model is also discussed. The results indicate that the proposed method has no process of leaving some samples out like cross-validation does, and all the training samples are considered when selecting principal components, so the problem of over-fit or under-fit can be avoided, which is benefit to improve prediction performance of calibration in spectral analysis. And the modified randomization test method is different with the commonly used randomization test that a simplified criterion is introduced here and it is easy to implement. With the proposed method, the authors can have a visualized and interactive process when selecting principal components. For these three experiments, 4, 5 and 8 selected principal components are employed in calibration respectively and the prediction result is the best for the independent external prediction sets. It is also implied that the proposed method is adaptable to the complex samples with more variables and little samples.