Optical switches are critical components for optical communication devices, and currently, they are now bottlenecking the development of optical telecommunications. In this study, we propose an innovative all-optical Kerr switch based on the Brewster angle effect, which exhibits a graphene/SiO₂/graphene/SiO₂ structure and an insertion loss (IL) that can be minimized by tuning the Fermi level of the graphene layers. A novel Brewster’s angle effect is achieved, resulting in a change in the incidence angle under nonlinear conditions as well as on–off switching of the phase change. The proposed switch has perfect absorption at an incident angle of 63 deg, where the phase of the reflected light is shifted from − π to 0. Furthermore, the Brewster angle of the switch was successfully shifted from 63 deg to 80 deg with an incident frequency of 0.8 THz. It was observed that an increase in Fermi level continuously shrank the nonlinear region of the switch. The minimum IL of the proposed optical switch was only 2 dB, which is indicative of excellent energy efficiency. The results of this study can aid in the development of key devices for broadband communications and electromagnetic sensing.