The images captured by the Division-of-Focal-Plane (DoFP) infrared polarimeters have checkboard effect. Thus, a polarization demosaicking processing of DoFP images is demanded to recover the full resolution polarization images, based on which, the subsequent tasks are then performed. However, the demosaicking processing is usually time-consuming and may introduce demosaicking errors. To achieve object detection by directly using infrared polarization DoFP image, a polarization-weighted local contrast object detection method is proposed. The difference of polarization characteristics between the object and background is first analyzed. Then, a convolution kernel is designed to calculate the Stokes vector directly from original infrared polarization DoFP images. A polarization-weighted saliency map of the degree of polarization image is also proposed, which is used for object detection with the adaptive thresholding. In addition, an edge detection method is used to refine the target detection results and obtain more complete detection results. The experiment results on the infrared polarization DoFP images dataset demonstrate that the proposed object detection algorithm is robust to the conditions of complex background and bad weather.