The construction of filling retaining walls is one of the necessary process links in phosphate mining using the filling method. Using phosphogypsum-based blocks to build filling retaining walls is an important way to support green mining technology and the utilization of phosphogypsum resources. However, the compressive strength, permeability, and water resistance of phosphogypsum-based blocks prepared on-site near common mining sites are difficult to ensure. This paper proposed a type of special-shaped brick structure formed by extrusion process using high-temperature calcined phosphogypsum, calcium powder, cement, etc. as basic raw materials. Through indoor test block production and testing, the compressive strength of phosphogypsum-based extruded special-shaped brick (PG-ESB) meets the brick strength requirements. To clarify the internal structural characteristics of PG-ESB, the internal pore distribution characteristics of PG-ESB were studied by low-temperature liquid nitrogen adsorption test, high-pressure mercury intrusion test, scanning electron microscope (SEM), X-ray computed tomography and other methods. PG-ESB has a lower cost and its strength meets the standard for building retaining walls. The results of low-temperature liquid nitrogen adsorption test, high-pressure mercury intrusion test, and SEM show that PG-ESB has a large number of pores with a size of 0.5~2.0 μm. X-ray computed tomography results further confirm that there are a large number of irregular micropores inside PG-ESB, 11.02% of the micropores are distributed in a circular shape, 85.23% of the micropores are distributed in a nearly circular or elliptical shape, and 3.75% of the micropores are in an irregular shape. To correct the shortcomings of test results, this paper proposes to use a three-parameter distribution method to completely describe the internal structural micropore distribution characteristics of PG-ESB. The research results provide references for further enriching and perfecting the damage theory of phosphogypsum-based cementitious materials.