For detecting the shadow in outdoor illumination conditions rapidly and efficiently, a shadow detection approach based on pixel-wise orthogonal decomposition is proposed. Based on linear model in and out of shadows in an outdoor scene image, a linear equation set is built for each pixel value vector and orthogonally decomposed. By the decomposition of the linear equation solution space, a color illumination invariant image and an illumination variation image are obtained. The color illumination invariant image is classified into some regions using K-means algorithm, each region has the same spectral albedo. According to the classification results, a Gaussian mixture model with expectation maximization algorithm is proposed for modeling the illumination variation image, and then the shadow areas are extracted. The extracted shadow areas are optimized with morphological operator. The proposed method does not need complex learning process of feature operators and greatly reduces the time complexity of computation. It also does not require any prior knowledge and can be directly applied to the real-time scene processing.