Quantitative phase detection is an important indicator for detecting the wavefront of incident light field. Shack-Hartmann wavefront sensors (SHWFSs) can simultaneously measure amplitude and phase distribution of wavefront with low resolution. The conventional incoherent phase detection methods require two CMOSs placed in different positions, estimating the phase information of wavefront by the exact difference between the diffraction images. This detection system has disadvantages of complexity and uncertainty. In this article, we propose to use a broadband light as the illumination light source, and the single-shot color image is needed as the input of the neural network. Gradient descent algorithm functions is used as the optimizer to complete the phase reconstruction. Both simulation and experimental results show that this technology combines physical models with neural networks to achieve phase reconstruction with high resolution, high speed, low cost. As the mean square error function (MSE) to evaluate the quality of reconstructed images, the minimum value of this method can reach 0.089 rad, and the phase reconstruction quality of transparent cells is 0.133 rad, which is better than the GS method (0.320rad) and the TIE method (0.378 rad). In consequence, the phase reconstruction method of broadband light source based on neural network can be used in adaptive optics, live cell biological real-time imaging field, etc.