In order to realize the fast calculation of computer-generated holograms, a ray-tracing algorithm is proposed, which is based on the OptiX ray-tracing engine and the NVIDIA graphic processing unit (GPU). This algorithm makes full use of the ray-tracing hardware core in GPU and thus effectively increase the calculation speed of holograms. When the three-dimension model is consisted of 1.6×10 ⁴ polygons and 4×10 ⁴ points, the calculation speed of the proposed algorithm is 11.5 times that of the traditional point source hologram generation algorithm based on GPU.