Object-based image analysis (OBIA) method has become the mainstream method of high-resolution remote sensing image classification due to its advantages of considering object spectrum, shape and texture feature information. Image segmentation is an important step of OBIA method, which determines the accuracy of image classification. The same image has a variety of ground features, and a single segmentation scale will cause the phenomenon of image undersegmentation or wrong segmentation, therefore, different image segmentation scales are often required in practical applications. Taking the fractal network evolution algorithm as an example, this paper proposes a remote sensing image classification algorithm combining quadtree structure and scale estimation. In this algorithm, the image region is firstly divided by using the characteristics of uniform detection and segmentation of quadtree structure, the quantitative estimation of segmentation scale of each region is realized by using the spatial segmentation scale estimation method of statistical average local variance and the attribute segmentation scale estimation method of statistical local variance histogram, and then the regional images are segmented at multiple scales using the fractal network evolution algorithm. Finally, the training samples are determined through visual interpretation, and the nearest neighbor supervised classification method is used to realize image classification. The experimental results show that compared with the classical object-based classification algorithm, the classification accuracy of the proposed algorithm is improved by 1.2% and the kappa coefficient is improved by 0.036. Moreover, the problems of linear feature extraction fracture and classification error caused by wrong segmentation and under segmentation in classical algorithm can be effectively avoided in the proposed algorithm.