Mo/Si multilayer coating process is one of the key technologies for extreme ultraviolet lithography. In order to optimize the coating process of Mo/Si multilayers, we study the influences of environment pressure and target-substrate distance on the surface roughness of Mo/Si multilayer coating. A model of atomic deposition is established based on the physical process of magnetron sputtering. The variations of incident angle and the energy distribution of depositing atoms with environment pressure and target-substrate distance are investigated. Moreover, the experiments are performed to fabricate Mo/Si multilayer coating samples using a direct current magnetron sputtering coating machine, and the evolutions of the surface roughness and power spectral density with environment pressure and target-substrate distance are studied. The conclusion obtained from the model agrees well with that from the experiments, and the proposed model can provide a theoretical explanation for the results measured by the experiments.