A large-scale optical matrix switch is generally composed of a permutation of switching units connected by a network of optical signal paths. In this work, a configuration of optical matrix switches with flexible switching units and Banyan networks is proposed. With Banyan networks, silica-based waveguides, and two types of switching units, the insertion loss of the optical matrix switch configuration is extensively studied for serial scales from to . Typically for an optical matrix switch, the insertion losses are achieved for a rearrangeably nonblocking configuration with two-state switching units, and for a strictly nonblocking configuration with three-state switching units when the intersection angle is designed as . Even at the released intersection angle of , the insertion losses of 4.0 and can be achieved for the rearrangeably nonblocking and strictly nonblocking configurations, respectively. In addition, the blocking property and the scalability of the large-scale matrix optical switches with this regime are analyzed.