A novel anomaly detection algorithm based on the theory of extended mathematical morphology and spectral similarity measurement is proposed for hyperspectral imagery. The spatial and spectral information has been used to locate and detect targets under the condition of none prior knowledge of targets and background. The extended mathematical morphological erosion and dilation operations are performed respectively to extract the targets features. The orthogonal projection divergence is used to calculate the cumulative distance in the erosion and dilation operations to determine the ordering relation. And the orthogonal projection divergence is also performed to measure the spectral similarity to fuse the results of feature extraction. The synthesized hyperspectral images collected by object modularization imaging spectrometer (OMIS) is applied to evaluate the proposed algorithm, the proposed algorithm is compared with RX algorithm by a specifically designed experiment, andit is applied to distinguish the targets with similar spectral characteristics. From the results of experiments, it is illuminated that the proposed algorithm can detect anomalous targets with low false alarm rate and its performance is better than that of RX algorithm under the same condition. It is also illuminated that the proposed algorithm can differentiate targets with similar spectral characteristics well with low false alarm rate.