The digital image correlation (DIC) method is an important non-contact displacement and strain measurement method. To accurately measure inhomogeneous strain, least absolute deviation fitting of displacements is introduced into the DIC method. As least absolute deviation fitting cannot be solved by analytical methods, a hybrid particle swarm optimization algorithm based on simulated annealing is used. By conducting numerical experiments on the formation of fictitious shear band and the similar material fault slip experiment, the strain measurement results of least square fitting and least absolute deviation fitting for inhomogeneous strain measurements are compared. Results show that the inhomogeneous strain measurement accuracy for least absolute deviation fitting is better than that for least square fitting, while these two methods exhibit the same accuracy for the homogeneous strain measurement.