In this paper, calcium carbide slag was used as raw material to prepare calcium sulfate whiskers in dilute sulfuric acid solution. On the basis of the optimal preparation conditions of the conventional hydrothermal method, the crystallization process of calcium sulfate whiskers was strengthened by using a microwave reactor without adding a surfactant. The effects of microwave reaction time and microwave radiation energy on the crystalline product were analyzed. The prepared product was characterized by X-ray diffractometry and scanning electron microscopy. The results show that the calcium carbide slag and dilute sulfuric acid solution react under microwave enhanced conditions for 10 min, and the microwave radiant energy was 400 W. The prepared calcium sulfate whiskers had uniform morphology, the average diameter of whiskers reaches 2.2 μm, and the average aspect ratio was 70. On this basis, the microwave effect and the growth mechanism of calcium sulfate whiskers were further analyzed.