Neural networks, as one of the most representative techniques in artificial intelligence, have been in rapid development towards high computational speed and low power cost. Due to intrinsic limitations brought by electronic devices, it can be hard for electronic implemented neural networks to further improve these two performances. Optical neural networks can combine both optoelectronic technique and neural network model to provide ways to break the bottleneck. In order to have a brighter view on the history, frontiers and future of optical neural networks, optical neural networks of feed-forward, recurrent and spiking models are illustrated in this paper. Challenges and future trends of optical neural networks on in situ training, nonlinear computing, expanding scale and applications will thus be revealed.