Fig. 1. Illustration of UAVs’ payloads delivery in natural disaster environment.

Fig. 2. Illustration of RL for UAVs. In this illustration, π(s) indicates the action function π(·) with the current state s as input; action i-j, reward i-j, and state i-j refer to the movement, the received rewards, and the new state for UAV i in j-th round, respectively.

Fig. 3. Scenes of input maps that are in two ways: (a) centered or (b) non-centered, where the location of UAV is shown by the star symbol and the point where the dashed lines connect.

Fig. 4. Illustration of the reward rate estimation calculation when UAV moves to a specific grid (like the green grid). The estimation considers the steps moving to the grid, the distance among the grid, each delivery point (DP, orange grid), the optimal landing area (blue grid), the urgent landing area (gray grid), as well as the remaining delivery points. In the left figure, b(i) indicates the battery level of UAV_i and indicates the size of payloads to be delivered at this grid. In the right one, and are the estimated return rates when UAV_i flies to each of these grids.

Fig. 5. Architecture of DQN in the proposed model. The convolutional Q-network will encode various information that exists in the region. Here f_center, f_local, and f_global indicate the functions deriving central, local, and global features, respectively; π(a|s) indicates the probability of each action a given current state s; π(s) refers to strategy functions of action.

Fig. 6. Example trajectories for the Manhattan32 map: Baseline with (a) 2 and (b) 3 agents; our proposed method with (c) 2 and (d) 3 agents.

Fig. 7. Example trajectories for the Urban50 map: Baseline with (a) 2 and (b) 3 agents; our proposed method with (c) 2 and (d) 3 agents.

Fig. 8. Delivery points coverage situations: (a) Manhattan32 and (b) Urban50.

Fig. 9. Impact of the number of UAVs deployed: (a) Manhattan32 and (b) Urban50.

Fig. 10. Impact of the number of delivery points: (a) Manhattan32 and (b) Urban50.

Fig. 11. Influence of maximum battery capacity: (a) Manhattan32 and (b) Urban50.