Remote sensing image pan-sharpening is the fusion of multispectral (MS) image and panchromatic (PAN) image into fused image with MS and high-resolution information features. In recent years, deep learning technology has been widely used in abounding fields such as text and image recognition. Inspired by this, we proposed a multistream deep ResCNN (RIFRCNN)-based fusion method for remote sensing images. First, we put forward automatic learning matching module (ALMM). The utilization of ALMM can realize dynamic optimization matching of MS and PAN images with different resolutions. Second, we build feature fusion residual blocks suitable for RIFRCN to optimize network parameters. Finally, we input MS image and PAN image with different resolution and complete the fusion task through ALMM, fusion, and reconstruction module. Experiments on WorldView and QuickBird datasets manifest that the proposed RIFRCNN can effectively fuse MS and PAN images, and the fusion images generated in the experiments of true and false color images show better results than the state-of-the-art technology.