The in-situ aberration detection of optical systems is of great significance for the processing and alignment of optical systems, the development of lithography machines, and the on-orbit adjustment of space cameras. Traditional in-situ detection methods for optical systems, such as Phase Retrieval (PR) and Phase Diversity (PD), perform excellently under specific conditions. But they have limitations when facing complex conditions such as large numerical apertures or when the lower bound of the Nyquist frequency of the optical system is not satisfied. Therefore, a method is proposed to combine the extended Nijboer-Zernike diffraction physical model with a deep neural network. Firstly, a deep residual network with the squeeze-and-excitation (SE) attention mechanism is constructed. Secondly, a mapping relationship from the intensity image point spread function (PSF) to the phase distribution is established to achieve feature extraction of the diffracted light intensity and prediction of the coefficients for phase description. Finally, the predicted coefficients are combined with the ENZ diffraction model to obtain the predicted PSF image, so as to realize the wavefront detection of the optical system. Experimental results show that when the numerical aperture (NA) of the optical system is large and the Nyquist sampling is not satisfied, the residual wavefront RMS between the real wavefront image and the reconstructed wavefront image is about 0.02λ, which is better than other methods. In comparison with other deep learning methods, this method is an unsupervised method, which not only reduces the dependence on a large amount of training data but also improves the accuracy of wavefront detection.