Aiming at the problem of real time and reliability of visual object tracking in a clutter background,an adaptive tracking algorithm by fusion of the color rectangle feature with the Edge Orientation Histogram is presented based on the Rao-Blackwellized Particle Filter(RBPF). To improve the tracking speed and precision, the RBPF is adopted to improve the filter performance,and an integral image is used to quickly compute the color rectangle feature and Edge Orientation Histogram.Finally,a fuzzy logic is applied to adaptively adjust each feature weight and to improve tracking accuracy in real time. The simulation result from a video tracking illustrates that the algorithm is robust and effective for visible and infrared object tracking under complex backgrounds.