Two visual methods are mainly used for measuring surface roughness based on laser speckle images. One method involves establishing the relationship between artificially designed speckle image feature parameters and surface roughness, and the other requires building a deep learning network prediction model. Both methods have limitations. The former involves a complex process in the feature parameter design, whereas the latter requires many sample images. This study proposes a method for predicting surface roughness based on laser speckle images and convolutional neural network-support vector regression (CNN-SVR). The proposed method incorporates transfer learning into a pretrained CNN, in which the deep features from the pooling layer of the network are input into an SVR network for surface roughness prediction. This approach automates the extraction of laser speckle image features and achieves high-precision predictions of surface roughness values with a few samples. Experimental results have demonstrated that the established model exhibits high accuracy in predicting the average absolute percentage errors of the surface roughness for plane grinding, horizontal milling, and vertical milling specimens, which are 3.46%, 3.20%, and 3.53%, respectively.