Cross-polarized image can provide both polarization-sensitive information and a more clear sub-surface structure of the sample. As a result, getting the cross-polarized image by using optical coherence tomography (OCT) becomes more and more popular. A novel fiber-based cross-polarized optical coherence tomography (CP-OCT) system is implemented, which is capable of acquiring both cross-polarized signal and co-polarized signal of a sample simultaneously. Forming principles of both cross-polarized and co-polarized signals are described, and the expressions of them are derived using Jones matrix. The images of a glass slide, a λ/4 wave plate at a wavelength of 1310 nm, and vivo skin obtained with the system confirm that CP-OCT can provide both polarization-sensitive and polarization-insensitive informations of the sample.