Surface plasmon resonances (SPRs) are often regarded as the collective oscillations of charge carriers localized at the dielectric–metal interface that display an ultrafast response upon light excitation. The recent developments in the fabrication and characterization of plasmonic nanostructures have stimulated continuous effects in the search for their potential applications in the photonic fields. Concentrating on the role of plasmonics in photonics, this review covers recent advances in ultrafast plasmonic materials with a prime focus on all-optical switching. Fundamental phenomena of plasmonic light–matter interaction and plasmon dynamics are discussed by elaborating on the ultrafast processes unraveled by both experimental and theoretical methods, along with a comprehensive illustration of leveraging ultrafast plasmonics for all-optical switching and pulse laser generation with a focus on device design and performance. This review is concluded with a brief highlight of the current progress and the potential future directions in ultrafast plasmonics.