Digital micromirror devices (DMDs) have emerged as essential spatial light modulators for holographic 3D near-eye displays due to their rapid refresh rates and precise wavefront modulation characteristics. However, since the modulation depth of DMDs is limited to binary levels, the quality of reproduced image from binary computer-generated hologram (CGH) is often deficient. In this paper, we propose a stochastic gradient descent (SGD) based binary CGH optimization framework where a convolutional neural network (CNN) is employed to perform the differentiable hologram binarization operation. The CNN based binary SGD optimization can significantly minimize the binary quantization noise in the generation of binary CGH, providing superior and high-fidelity holographic display. Our proposed method is experimentally verified by displaying both of high-quality 2D and true 3D images from optimized binary CGHs.