The red (R), green (G), and blue (B) filtering that has high transmittance in the near-infrared (NIR) band (i.e., R+NIR, G+NIR, and B+NIR, respectively) is a common way for an electron-multiplying charge-coupled device (EMCCD) to achieve true-color imaging and maintain high imaging sensitivity under low illumination. However, the introduction of NIR components can cause color distortion and color distribution compression. An orthogonal color transfer model was built under the constraint that a pair of pixel-registered source and reference images shared the same coordinate representation in the standard orthogonal color space. A feature dimension was introduced into the model through the convolution neural network to alleviate the one-to-many mapping problem caused by color deviation and color distribution compression. An end-to-end color transfer network was created. It consisted of two parts: a pre-trained front-end network that clustered pixels into different feature channels according to the texture and semantic meaning of an EMCCD image and a trainable back-end network that performed the color transfer of each cluster based on the coding statistics characteristics of pixels of each feature image. The proposed model, tested by real-world images, proved to have wide applicability and be able to achieve natural color in different scenes under different illuminances. Experiments show that the peak signal-to-noise ratio of a true-color image transferred by the proposed method is increased by 75.78% on average compared with that of a color-distorted image. The structural similarity index measurement is increased by 103.74%, and the chromatic aberration is decreased by 67.48%.