To solve the problems of complex operations, poor repeatability, and contact damage in traditional micro-nano fiber diameter measurement methods, a new measurement system based on machine vision was developed. Firstly, the image was preprocessed and binary segmented in the system, and then the edge of micro-nano fiber was initially located by the Canny edge operator. After that, a sub-pixel detection method based on improved Zernike moment was used to accurately locate the sub-pixel edge. Besides, our system also proposed an algorithm combining Hough transform and the least square method to fit sub-pixel edge points, which improved the measurement accuracy of micro-nano fiber diameters to the nanometer level. The measurement results of micro-nano fiber diameter show that the system can achieve high automation with a measurement error of less than 3.51%, and the running time is 2.671 s. It can be applied to the high-precision and real-time measurement of micro-nano fiber morphology.