There are a variety of elementary and collective terahertz-frequency excitations in condensed matter whose magnetic field dependence contains significant insight into the states and dynamics of the electrons involved. Often, determining the frequency, temperature, and magnetic field dependence of the optical conductivity tensor, especially in high magnetic fields, can clarify the microscopic physics behind complex many-body behaviors of solids. While there are advanced terahertz spectroscopy techniques as well as high magnetic field generation techniques available, a combination of the two has only been realized relatively recently. Here, we review the current state of terahertz time-domain spectroscopy (THz-TDS) experiments in high magnetic fields. We start with an overview of time-domain terahertz detection schemes with a special focus on how they have been incorporated into optically accessible high-field magnets. advantages and disadvantages of different types of magnets in performing THz-TDS experiments are also discussed. Finally, we highlight some of the new fascinating physical phenomena that have been revealed by THz-TDS in high magnetic fields.