The prediction accuracy of a Conventional Recursive Least Square (CRLS) predictor is strongly correlated with the inter-spectral correlation and is sensitive to the sequence in which the pixels are predicted. In view thereof, a lossless compression method for hyperspectral images was proposed. The method, which was based on the CRLS predictor, was modified to enable the selection of adaptive bands and to optimize the prediction sequence mode. First, to improve the correlation between the reference bands and the band to be predicted, the bands of the hyperspectral image were reordered according to the criterion of the maximum inter-spectral correlation coefficient in the preprocessing stage. Subsequently, the adaptive band selection strategy was used to select multiple bands with the highest correlation with the band to be predicted for use as prediction reference bands. Afterwards, the CRLS predictor with the best prediction sequence mode, selected by the minimum prediction residual entropy, was used for inter-spectral prediction. Finally, the arithmetic encoder was used to encode the prediction residual. Experiments on the AVIRIS 2006 dataset show that this method achieves bit rates of 3.314, 5.594, and 2.395 bpp on a 16-bit calibrated image, 16-bit uncalibrated image, and 12-bit uncalibrated image, respectively. These results indicate that this method can effectively improve the prediction accuracy of the CRLS predictor without significantly increasing the computational complexity. The best result of the proposed method closely approximates or is superior to that of other similar methods.