Based on the physical model of atmospheric scattering and an optical reflectance imaging model, three major factors influencing the fog removal for a single image were discussed in detail. The dark channel phenomenon was explained by the optical model, and the method to solve the parameters of atmospheric scattering model was rigorously derived from a new view. The gray-scale opening operation was used to eliminate the interference from a while object to obtain the global atmospheric light and the fast joint bilateral filtering technique was proposed to greatly improve the speed and accuracy of atmospheric scattering function solving. Finally, the scene albedo was recovered by inverting this model. Experiments show that the method can remove effectively the effect of lights from sky and environments and can recover the color and definition of original scenes. The simulation results indicate that the processing time for an image of 576×768 spends only by 1.7 s. As compared with the existing algorithm, obtained results on a variety of outdoor foggy images demonstrate that the proposed method achieves good restoration for contrast and color deity, and improves image visibility greatly.