With the development of unmanned aerial vehicles (UAVs) and computer vision, target detection methods based on UAVs have been increasingly applied in military and civilian fields. Considering the adaptability requirements of low illumination environments such as rain, fog, and night, visible and infrared (IR) sensors are often installed on UAVs to perform in all-weather and all-day conditions. To improve the near-surface detection performance of UAVs in low illumination environments, a pedestrian detection method using image fusion and deep learning is proposed. Visible and IR pedestrian images are collected by the UAV. The corresponding aerial images are registered and annotated. These two different types of aerial images are aligned at the time sequence and matched using the scale invariant feature transform. A U-type generative adversarial network (GAN) is first developed to fuse visible and IR images. A convolutional block attention module is introduced to strengthen the pedestrian target information in the GAN. The spatial domain and channel domain attention mechanisms are proposed to generate color fusion images with rich details and solve the problems of feature extraction as well as fusion rules designed manually in the existing image fusion methods. Then, You Only Look Once Version 3 (YOLOv3)-spatial pyramid pooling combined with transfer learning is adopted using the fused images to train the model on our aerial dataset to verify the pedestrian detection performance. In addition, comparison experiments are carried out. The experimental results demonstrate that the YOLOv3 model is successfully transferred to the target dataset. The performance of the proposed detection model using the fused images for transfer training is the best among the different methods. Finally, the accuracy P, recall R, mean average precision, and F₁ score reach 0.804, 0.923, 0.928, and 0.859, respectively.