Optical neural networks, as the next generation of artificial neural networks, have the advantages of high speed and low power consumption. Optical fully connected neural networks and optical convolutional neural networks consisting of interconnected Mach–Zehnder interferometer were investigated with respect to optical neural networks currently under development. The characteristics and implementations of two optical neural networks in terms of algorithms and structures were presented. Using pytorch and INTERCONNECT software, two kinds of optical neural networks were simulated and analyzed, and the performance of the two kinds of optical neural networks were compared in terms of image recognition accuracy, network complexity, and energy consumption, so as to elucidate the advantages and disadvantages of each of the optical fully-connected neural networks and the optical convolutional neural networks, and to provide reference for the development of optical neural networks in the future.