The performance of the target tracking in video sequences was unstable only using the color information.In order to track the interested targets robustly in the conditions of cluttered backgrounds and illumination changes,an efficient Particle Filter(PF) tracking algorithm combining the color and structural information was proposed.A structural model was developed by using target gray level image,and structural similarity index was used to measure target similarity between two frames in correlation tracking and applied in color histogram-based particle filter framework.The experiments show that our algorithm can track the targets more robustly than traditional color histogram-based PF tracking algorithm when using the same particles,and can be applied in suspicious target tracking in specific situation.