A graphene-based passively mode-locked and passively Q-switched Yb-doped fiber laser is reported. Stable mode-locked pulse train occurres at about 1.2 W incident pump power and the pulse repetition rate is 1.04 MHz in a ring cavity configuration; increasing the incident pump power to 2.3 W, the pulse width is measured to be about 680 ps and the output power is 170 mW, which corresponds to single pulse energy of 163 nJ. Graphene-based passively Q-switched operation is also demonstrated with pulse width of 70 ns and maximum output power of 12 mW in a linear cavity configuration, which corresponds to single pulse energy of 46 nJ. The repetition rate is tunable from 140 to 257 kHz along with the increase of the pump power.