Initial defects are identified as one of the root causes affecting the mechanical properties of concrete. In this paper, expanded polystyrene (EPS) particles were utilized to prefabricate initial defects of rock-filled concrete (RFC), and the defect content was quantified by porosity. Uniaxial compression tests of four porosities of RFC at four strain rates were systematically conducted, and the effects of strain rate and porosity on RFC failure modes, compressive strength, elastic modulus, and critical strain were studied. The results show that the RFC failure modes under different operating conditions are similar to the “hourglass” type. The compressive strength, elastic modulus and critical strain of RFC all have strain rate strengthening effect. As the porosity increases, the sensitivity of RFC compressive strength, elastic modulus, and critical strain to strain rate decreases. Compared with strain rate, the effect of porosity on compressive strength, elastic modulus, and critical strain of RFC is more significant. Additionally, based on the Sargin model, a dynamic constitutive model of RFC with initial pores is constructed. It is used to illustrate the mechanical behavior of RFC with initial pores at low strain rates.