In the Visible Light Communication (VLC) system based on circle Color Shift Keying (CSK) modulation constellation, the bit error rate performance is poorer under certain number of modulation order. In this paper, we optimize the constellation point design, in keeping the constellation points under the condition of invariable pointing triangle center redesigned constellation structure. We change the constellation point location and increase the minimum code distance between constellation points to improve the bit error rate performance. In order to further improve the transmission quality of VLC signal, the principle of Finite State Machine (FSM) is introduced to carry out joint coding design to further improve bit error rate. Finally, simulation verification and performance analysis are carried out. The simulation results show that, compared with the traditional CSK modulation technology, the optimized extended circular constellation CSK and FSM combined coding proposed in this paper have better bit error rate performance. The gain of 4 Combination Coding-Finite State Machine(CC-FSM)code is 2.1 dB compared with that of 4-CSK, and the gain of 8CC-FSM code is 1.7 dB compared with that of 8-CSK.