Image restoration of sparse aperture imaging system is an important step of sparse aperture imaging technology. Exploratory research on restoration of blurred image caused by misaligned sparse aperture imaging system is carried out by using phase diversity based on defocus, and adaptive genetic algorithm is adopted as numerical method. At the same time, the restoration of image added Gaussian noise is also researched. The result of simulation experiment shows that phase diversity is a very effective method for restoration of sparse aperture system image and retrieval of sparse aperture system aberration.