The interaction between light and nanostructures has always been one of the important topic in nanophotonics. The nanostructure of the core components play an important role in the function and performance of photonic devices. There are two approaches in the design of nanophotonic devices. One is based on physical principles and intuitive, while the other employs the idea of inverse design to obtain the optimal structure according to the required optical response. In recent years, inverse design has made great progress in nanophotonic devices. In particular, the technology of deep learning was recently introduced, promising for the design of high-performance nanophotonic devices. This article focuses on inverse design method of the nanophotonic devices. The background, key progress and typical applications of this emerging research direction are analyzed and summarized, and the challenges and prospect of inverse design are also presented.