A method for displacement field measurement of digital speckle images using a convolutional neural network (CNN) is proposed. A series of digital speckle images with their exact displacement fields in multiple deformation modes are used to construct a dataset and CNN model for distinguish displacement field of digital speckle images are proposed. Verification experiments of simulated speckle images show that the proposed method is computationally efficient and achieves high test accuracy for random deformations, axial uniform deformations, shear deformations, and other deformation modes. Moreover, the uniaxial tensile test of silica gel shows that the proposed method accurately measures the displacement field of real speckle images and confirms its high computational efficiency. The proposed deep CNN can be used to efficiently and accurately test the displacement field of digital speckle images, thereby indicating good application prospects for material deformation testing.