Polarimetry is a highly sensitive method to quantify changes of the polarization state of light when passing through matter and is therefore widely applied in material science. The progress of synchrotron and X-Ray Free Electron Laser (XFEL) sources has led to significant developments of X-ray polarizers, opening perspectives for new applications of polarimetry to study source and beamline parameters as well as sample characteristics. X-ray polarimetry has shown to date a polarization purity of <1.4e-11, enabling detection of very small signals from ultrafast phenomena. A prominent application is the detection of vacuum birefringence. Vacuum birefringence is predicted in Quantum Electrodynamics (QED) and expected to be probed by combining an XFEL with a petawatt-class optical laser. We review how source and optical elements affect X-ray polarimeters in general and what qualities are required for detection of vacuum birefringence.