Edge detection for low-contrast phase objects cannot be performed directly by spatial difference of intensity distribution. In this work, an all-optical diffractive neural network (DPENet) based on the differential interference contrast principle to detect the edges of phase objects in an all-optical manner has been proposed. Edge information is encoded into interference light field by dual Wollaston prisms without lenses and light-speed processed by the diffractive neural network to obtain the scale-adjustable edges. Results show that DPENet achieves F-scores of 0.9293 (MNIST) and 0.9356 (NIST) and enables real-time edge detection of biological cells, achieving an F-score of 0.7547.