AlGaN is a key material for deep ultraviolet optoelectronic and electronic devices. With the increase of the Al composition ratio, the phase separation on the surface, caused by small-scale compositional fluctuations, is prone to affecting the performance of the device. In order to explore the mechanism of the phase separation on a nanoscale, the AlGaN wafers with different quantities of Al compositions are investigated by the confocal photoluminescence spectroscopy and the single-pass Kelvin force probe microscopy. The composition ratios of Al for the three samples are about 0.3, 0.5, and 0.7, respectively. The single-pass Kelvin force probe microscopy based on dual-frequency phase-locking is used to obtain high spatially resolved (about 10 nm) surface potential images. In the area where the phase separation phenomenon is obvious in the photoluminescence spectrum, the sharp change of the surface potential can be observed at the irregular steps and the edges of the surface pits. The potential changes can be ascribed to the inhomogeneous composition distribution. In the area where the topography turns into step flow, the surface pits shrink and merge. No obvious surface potential domain boundaries appear at the steps nor on the edges of the surface pits. Meanwhile, the phase separation phenomenon in the photoluminescence spectrum almost disappears. Our experiments show that the steps and the edges of the surface pits on AlGaN surfaces are main reasons for small-scale compositional fluctuations and the phase separation in the spectrum. Combining with in-situ confocal photoluminescence spectra, high spatially resolved surface potential image by single-pass Kelvin force probe microscopy is an effective method to characterize the phase separation on AlGaN surface on a nanoscale.