To reach the optimal design of optical system, the binary coding of standard genetic algorithm was changed to real coding to improve its robustness and calculation efficiency, and an escape function was incorporated into the evaluation function of system to avoid falling into local minima. Then the improved genetic algorithm was applied to optimize a fisheye lens and a catadioptric panoramic image system. The imaging performances were simulated by raytracing calculation with Zemax and optimized with CODE V for comparison. The calculation results show that the optical system optimized with the improved algorithm has better imaging performance than which optimized with standard genetic algorithm mixed with escape function or optimized by CODE V. The study confirms that the proposed algorithm has satisfactory robustness and calculation efficiency in optimization of optical systems.