Accurately controlling the end point temperature of converter steelmaking can considerably enhance the quality of final tapping. Modified colorimetric thermometry was used to determine the temperature of the furnace mouth flame to enhance the molten steel temperature prediction accuracy at the end point; furthermore, the improved competitive adaptive reweighted algorithm was used to extract the characteristic wavelength of the flame spectrum. Finally, the image and spectral features were fused and analyzed. Subsequently, a steelmaking end point temperature prediction model was established. The root mean square error of the proposed model' prediction is 15.8556 K, the accuracy within the prediction error of ±20 K is 87.50%, and the accuracy within the prediction error of ±30 K is 95.00%. Compared with the model established solely using the image feature or spectral feature, the prediction error of the proposed model is the lowest, and the accuracy is the highest. This confirms that the model established in this experiment has a good end point temperature prediction and can successfully meet the field requirements of steelmaking production.