With the classical ensemble model, carrier envelope phase (CEP), which is dependent strong-field double ionization (DI) of xenon atoms by elliptically polarized laser pulses, is investigated. The results show that the momentum distributions of Xe2+ ion strongly depend on CEP. Sequential double ionization (SDI) and nonsequential double ionization (NSDI) of final state exist simultaneously in DI events with this laser field. The yield of SDI decreases firstly, then increases as CEP increases. But the yield of NSDI increases firstly, then decreases as CEP increases. The yields of SDI and NSDI both show a periodic change, and the period is π. Analysis of trajectories shows that the events of NSDI occur still via recollision, and this process can be interpreted well through the three-step model. In addition, the ionization time of two electrons in SDI and the recollision time in NSDI strongly depend on CEP which leads the final-state momentum distribution of Xe2+ ion changes with CEP.