Plant nitrogen content (PNC) is an essential indicator of crop growth and nitrogen nutrition status. Therefore, accurate and efficient access to PNC information is vital for dynamically monitoring potato growth and proper N fertilizer application. In this study, the UAV hyperspectral images were obtained at the budding stage, tuber formation stage, tuber growth stage, starch accumulation stage, and maturity stage of the potato. After preprocessing, the original canopy spectrum and first-order differential spectrum of five growth stages were extracted; Secondly, the correlation analysis was carried out between the extracted canopy spectrum and potato PNC, and the sensitive wavelength of PNC was screened out; Then, the texture and color of two image features of the hyperspectral image at the wavelength of the original spectral features of the canopy were extracted using the gray co-generation matrix and the 1st to 3rd-order color moments, respectively, and the extracted features were correlated with the potato PNC to filter out the top five image features with higher correlation; Finally, based on spectral features, image features, and map fusion features, potato PNC estimation models were established by using elastic network regression (ENR), Bayesian linear regression (BLR), and limit learning machine (ELM). The results showed that: (1) there are differences in the characteristic wavelengths of canopy spectra in the five growth stages of potatoes. Still, most of them were located in the visible region. (2) The correlation between the texture and color characteristics of the original spectral characteristic wavelength image of the canopy and PNC was high. The correlation from the budding stage to the starch store stage was significantly higher than that in the mature stage. (3) The estimation models of potato PNC based on a single spectral feature and a single image feature have a good effect from the budding stage to the starch accumulation stage but a poor effect at the maturity stage. (4) From the budding stage to the starch accumulation stage, the estimation effect of potato PNC based on the map fusion feature was significantly better than the single spectral feature and the single image feature. (5) In each growth period of potato, the PNC estimation models constructed by ENR based on the same variable were better, BLR was the second, and ELM was poor. Among them, the accuracy and stability of the PNC estimation models constructed by ENR with fusion characteristics as model variables were the best. The modeling R2 of five growth periods were 0.91, 0.75, 0.82, 0.77 and 0.69 respectively; RMSE were 0.24%, 0.31%, 0.26%, 0.22% and 0.29% respectively, and NRMSE were 6.59%, 9.79%, 9.58%, 7.87% and 11.03% respectively. This study can provide a fast and efficient technical tool for monitoring the nitrogen nutrition of potatoes.