Accurate spectral measurement and wavelength determination are fundamental and vital for many fields. A compact spectrum analyzer with high performance is expected to meet the growing requirements, and speckle-based spectrum analyzer is a potential solution. The basic principle is based on using the random medium to establish a speckle-to-wavelength mapping relationship for spectrum reconstruction. This article introduces current speckle-based spectrum analyzers with different schemes and reviews recent advances in this field. Besides, some applications by using speckle-based spectrum analyzers are also introduced. Finally, the existing challenges and the future prospects of using speckle for spectrum recovery are discussed.