Carrier reuse technology is an important method to reduce the cost of Passive Optical Networks (PON) and multiplexing technology is the main method to improve the transmission capacity of PON. A PON system is designed in this paper. By combing the Subcarrier Multiplexing (SCM) and Wavelength Division Multiplexing (WDM) technology, the transmission capacity of PON is increased significantly. The passive reflection filter is used in the system to realize carrier reuse, which greatly reduces the system cost. In the SCM/WDM-PON system, the carrier reuse is achieved by the reflection filter, which is built in the Optisystem software. Using 3 and 6 GHz radio frequency signals as subcarriers in the SCM system, 8 wavelengths signal with a wavelength interval of 200 GHz ranging from 193.1~194.5 THz are used as optical carriers in WDM, and the data rate of users for uplink and downlink transmission is 1 Gbit/s. The effects of the reflection coefficient, optical carrier power and transmission distance on uplink and downlink data transmission performance are studied. When the reflection coefficient of the reflection filter is 97%, the overall system performance is optimal with the optical carrier power of 5 dBm and the transmission distance less than 27.2 km.