This paper proposes a method for extracting building LiDAR point cloud and orthophoto fusion based on improved genetic algorithm (IGA) for fine-grained 3D building modeling: The features based on point cloud and image are calculated and extracted to expand the feature space of point cloud; then, by using the improved genetic algorithm, the point cloud features are selected, construct and optimize feature space; finally, SVM classifier is used to achieve accurate extraction of building point cloud. The experimental results on ISPRS open data set Vaihingen test data show that the method proposed in this paper has high accuracy in building extraction. The experimental results on actual production data show that the building extraction accuracy is high and stable, which proves the advancement and universality of this method.