Airborne LiDAR systems are widely used in urban objects extraction and recognition because of the advantages in obtaining 3D information conveniently and rapidly. However, it considers geometrical features regardless of buildings and vegetation spectral features and error rate is high in the dense canopy. Aiming at this problem, an algorithm of building extraction by fusing spectral features in aerial images and geometrical features in LiDAR data is proposed. Firstly, the spectrum information can be obtained by registering with LiDAR data. Then, the new feature which fuses spectral and geometrical information is formed by improved tensor voting. Finally, building extraction is achieved by random forests algorithm. Simulation test datasets are provided by ISPRS. Through the comparison of results before and after fusing spectral features, the accuracy of the proposed algorithm is obviously high and the extraction quality of proposed algorithm reaches to 94.26%. The simulation results prove the importance of fusing spectral features in building extraction.