Fringe projection technology has attracted widespread attention in academic research and engineering applications because of its advantages of non-contact, high precision, and high efficiency. On this basis, to enhance the measurement efficiency of the system, multi-channel fringe projection technology has emerged. It encodes sinusoidal fringe patterns in the red, green, and blue channels, significantly reducing the number of images captured by a camera. However, the characteristic of multi-channel usage introduces both crosstalk and chromatic aberration into the system, becoming critical factors affecting measurement accuracy. Therefore, it is crucial to compensate for both crosstalk and chromatic aberration. The existing methods mostly separately correct crosstalk and chromatic aberration, which has the disadvantages of capturing multiple images, multiple procedures, and complex operations. In order to solve this problem, this paper proposes a pixel-by-pixel correction method based on orthogonal color fringes for crosstalk and chromatic aberration. Firstly, the fringe intensity information from each channel is extracted by projecting orthogonal color fringe patterns. A mathematical model is then established for crosstalk coefficients in relation to average intensity and background intensity. Then fringe intensity is corrected, thereby achieving the elimination of crosstalk. Secondly, the unwrapped phase is computed for each channel in both horizontal and vertical directions. Based on this, pixel matching relationships between different color channels are constructed. Chromatic aberration correction is then accomplished through interpolation. Using a color camera and projector, a fringe projection system has been constructed to conduct experiments on the proposed method. Measurements were performed on two objects of a plane and a standard step, indicating that the proposed method significantly enhances the measurement accuracy and efficiency. Moreover, it outperforms traditional methods in terms of measurement precision and efficiency. When measuring the plane, the proposed method achieves a measurement precision of 0.040 mm, which is an improvement of 0.029 mm compared to 0.069 mm of the traditional method. For the standard step, the measurement error is reduced from 0.647 mm to 0.031 mm. Compared to the measurement precision of 0.045 mm achieved by the traditional method, the proposed method demonstrates an improvement of 0.014 mm in measurement precision. And the number of fringe patterns required to be captured has been reduced by half compared to the traditional methods. Additionally, the measurement error distribution range is smaller, indicating higher stability. Therefore, the proposed method can improve the measurement accuracy and efficiency of multi-channel fringe projection technology effectively and stably.