A fourth-order pulse amplitude modulation faster-than-Nyquist (4PAM-FTN) rate atmospheric optical transmission system was proposed to improve the transmission rate. The theoretical bit error rate (BER) equation in a log-normal fading channel was derived. Then, the influence of transmission distance and laser wavelength on the system BER performance was discussed. Furthermore, the relationship between the acceleration factor and the transmission rate was analyzed. Monte Carlo simulation results reveal that the BER is proportional to the transmission distance, but inversely proportional to the acceleration factor and the laser wavelength. When the acceleration factor is 0.9, the system shows a 6.1% transmission rate gain with an almost invariable BER performance, and when the acceleration factor is 0.83, it achieves a 16.7% transmission rate gain.