In scalable video coding, the original bit-stream can be cropped and generate multiple sub-bit-streams. In order to optimize coding efficiency of both the original bit-stream and the sub-bit-streams, partial decoding based on a Rate Distortion Optimization (RDO) algorithm is proposed. In the enhancement layer’s RDO process, the influence of sub-bit-streams on coding efficiency is predicted by selectively truncating Discrete Cosine Transform(DCT) coefficients.The propagation distortion map is then used to estimate the drift error. In the base layer’s RDO process, the multi-layer coding method is employed to cope with the constraint between the layers’ mode choice. In order to reduce the computational complexity of the algorithm, both the partially decoded reconstruction process and the base layer’s mode decision are optimized. Finally, the configuration of parameters is discussed and chosen adaptively based on the error propagation sensitivity on a macroblock basis. From experimental results, the proposed algorithm improves the sub-bit-streams’ Peak Signal to Noise Ratio(PSNR) by 0.03-0.53 dB. For the scalable video system, the proposal can meet the needs of high coding efficiency and flexible sub-bit-rate extraction.