In massive Multiple-Input-Multiple-Output (MIMO) system, traditional detection algorithms suffer from more performance loss and higher complexity with increase in the number of antennas. In order to solve the complexity problem, a likelihood criterion based on neighborhood search as a branch strategy is proposed. Firstly, the quadratic programing model is constructed ,and the model is used to search tree as the root. Then, the node with the smallest value of the objective function is found as the branching node. Finally, the likelihood criterion is applied to the branching nodes for reducing the complexity of the branch. Specially, when the bit error rate achieves 10-4, the performance gain of the proposed algorithm is increased by 1.5 dB and the complexity of the proposed algorithm is reduced by about 69.84 percentage points compared with the traditional search tree algorithm in 16 quadrature amplitude modulation. The simulation results show that the proposed algorithm has better error performance and lower complexity.