In interferometry, environmental, imaging system thermal noise and ununiform reflectance of the test surface all lead to inconsistent phase distribution, which in the form of residues occurred in the wrapped phase maps. Under this condition, how to completely recover the phase unwrapping result with high-precision becomes to be an unavoidable issue. Different from conventional unwrapping techniques, which construct branch cuts between the residues to separate the noise-affected areas; or different from the approaches implementing minimize norm methods, at the cost of losing local information of the details, to the best of my knowledge, harmonic mean, is for the first time applied in phase unwrapping. Based on its specific property of handling large outliers, the four neighborhood pixels around the residue are analogous to the harmonized mean data, giving rise to a straightforward way to identify the noise-related pixels and to obtain a global unwrapping result. Finally, a computer simulated complex surface with high level of noise is used to verify the accuracy, efficiency and simplicity of the proposed algorithm in implementation, through both quantitative and qualitative analysis.