In view of the nonlinear effects of high-speed fiber communication transmission, a nonlinear distortion compensation method for fiber communication based on convolutional neural networks is proposed. The proposed method takes advantage of classical convolutional neural networks to capture the nonlinear features of signals transmission through fiber, and utilizes a regression layer as the final layer to realize nonlinear fitting for optical signals. Besides, quantum particle swarm optimization algorithm is adopted to search the super parameters of deep convolutional neural networks, in order to reduce the difficulty of convolutional neural networks training. Numerical simulation experimental results show that the quantum particle swarm optimization can optimize the super parameters of convolutional neural networks, at the same time, the trained convolutional neural networks can improve the communication quality of fiber transmission.