Visible light communication (VLC) is prone to overlap coverage of the adjacent VLC access points (APs), receiver occlusion, and light path interference. This study develops an AP selection algorithm for heterogeneous VLC/wireless fidelity (WiFi) networks. By considering the joint demands of transmitting and receiving, the algorithm employs a quality-of-service (QoS)-based decision-making to satisfy the differentiated resource demands of the receiving end and the transmitting equipment. First, it designs a QoS-oriented weighted information matrix for each receiver and guides the user to select the VLC AP with high received signal-to-interference-and-noise ratio, high received power, low traffic load, and low historical interruption probability. Guided by the same algorithm, the selected VLC AP preferentially serves users with strong anti-interference. Finally, to optimize the AP access, we improve the weights of the APs and users based on gravitation theory. In simulations on a uniform square layout and a mixed circular-square layout, the proposed algorithm delivers 77.3% and 11.4% more throughput respectively than the existing algorithms, and reduces the service fairness index by 53.1% and 41.1%, respectively.