The existing algorithms for detecting straight line segments are all designed for grayscale images. When implemented on color images, they discard the input image's color information; this is undesirable for accurately detecting straight line segments. To solve this problem, this study proposes an algorithm that directly detects straight line segments in color images. First, DiZenzo operator is utilized to compute gradient vectors in three color channels, and image edges are extracted based on the average gradient magnitudes and orientations. Then, the pixels on each image edge are tracked and they are linked or split to generate candidate straight line segments. Finally, based on the gradients of the points on the candidate line segments, Helmholtz criterion is used to eliminate the false line segments caused by noise, yielding the final set of line segments. Experimental results show that the new algorithm can fully exploit the color information of the input image, leading to significantly improved detection efficiency when compared with the existing algorithms. On the YorkUrbanDB image database, the highest accuracy of the proposed algorithm increases from 0.2207 to 0.2687 and a high F-score is achieved.