The effect of atmospheric refraction on the background light bit error rate of the satellite-to-ground quantum key distribution (QKD) system was studied. Three different background light conditions of sunny day, full moon night, and moonless night were considered. First, a stratified satellite-to-ground QKD channel transmission model was established to simulate the influence of atmospheric refraction effect on the transmission distance. Then based on the model, the bit error rate of the background light varied with the zenith angle was simulated and analyzed for the cases in the presence and absence of the atmospheric refraction effects. The results show that, the atmospheric refraction has a certain impact which should not be neglected on the background light bit error rate. In addition, the relatively lowest bit error rate can be obtained under the moonless night background light condition.