Timely and accurate crop yield estimation is crucial for making informed decisions regarding crop management and assessing food security. This study aims to develop a method that combines continuous wavelet transform (CWT) with machine learning to predict wheat yield accurately. This research is based on the spectral data of canopy height and yield data obtained from two-year field trials conducted during wheat growth’s flowering and filling stages in 2020—2021. Initially, CWT is employed to extract three wavelet features (WFs), namely Bortua-WFs based on the Bortua method, 1% R²-WFs representing WFs along with the top 1% determination coefficient for wheat yield, and SS-WFs encompassing all WFs under a single decomposition scale. Subsequently, three machine learning algorithms Random Forest (RF), K-nearest neighbor (KNN), and extreme gradient Lift (XGBoost) are utilized to construct the yield prediction model. Finally, optimal spectral features are selected using the same methodology for modeling and comparison purposes. The results demonstrate that: (1) all three WFs models combined with machine learning methods perform well, with higher accuracy and stability observed in the model built based on Boruta-WFs. (2) Compared to the spectral characteristic model, improved accuracy was achieved by utilizing Bortua-WFs at each growth stage; specifically, an increase in R² accuracy by 17.5%, 4%, and 39.6% during flowering stage, as well as an increase by 8.4%, 5.6%, and 16.9% during filling stage respectively were observed across different models.(3) The estimation model at the grouting stage outperformed that at the flowering stage; particularly noteworthy was the performance of XGBoost when combined with Bortua-WFs, which yielded an R² value of 0.83 accompanied by an RMSE value of 0.78 t·ha^-1. This study compared the performance of different characteristics and methods. It determined the best model accuracy under different schemes, which can provide technical references for the accurate wheat yield prediction by spectral technology.