The random phase errors of the optical carriers are discussed and controlled for passive millimeter-wave sparse aperture (PMMW SA) upconversion imaging. A two-channel model is set up for analyzing characteristics of the phase errors, and an active optical control technique based on stochastic parallel gradient decent algorithm (SPGD controller) is proposed to calibrate the phase errors. To demonstrate the feasibility of the SPGD controller, simulations are performed and an experimental system with a two-channel fiber array is set up. Simulation and experiment results show that the SPGD controller can effectively and rapidly compensate the phase errors of the optical carrier, and the accuracy of the phase control is sufficient for imaging systems.