Aiming at the problem that the extraction of sample morphological information in quantitative phase imaging technology is cumbersome and not conducive to automatic detection and analysis, the feasibility and training strategy of an accurate recognition of phase objects with similar contour based on small-scale datasets are explored. The phase distribution and interference fringe datasets of four types of samples, including polystyrene microspheres and red blood cells are established accordingly. A convolution neural network (CNN) model is constructed to recognize the phase diagram successfully, and then the phase values of different samples are transformed to increase recognition difficulty. All sample types are successfully recognized on the verification set by improving the network model. To simplify the detection, the interference fringes corresponding to four types of samples are identified. The residual module is used to improve the network degradation of CNN model and realize an accurate classification. According to the actual situation of complex and changeable fringe visibility and carrier frequency, the impact on the recognition accuracy is investigated, respectively. The recognition efficiency of the model is improved via optimizing the training set, which shows the potential of machine learning technology in phase information recognition.