An approach to characterizing the intrinsic phase response of an optical filter free of additional phase shift from the pigtail or the free space is proposed and experimentally demonstrated. The kernel of this approach is based on the fact that the intrinsic phase response of an optical filter with a minimum phase response has a unique relationship with its magnitude response. Thus, the intrinsic phase response can be obtained through the measurement of the magnitude response only. Most importantly, this method avoids the influence of the fiber pigtail or the free space on the phase response measurement, which cannot be easily calibrated in the traditional single-sideband modulation-based optical vector analyzer. A phase retrieval algorithm based on fast Fourier transform is presented, which can be used to accurately retrieve the intrinsic phase response of an optical filter with either a symmetric magnitude response or an asymmetric one. In the experiment, the intrinsic phase response of an active stimulated-Brillouin-scattering-based optical filter with an asymmetric magnitude response and a phase-shift fiber Bragg grating with pigtails is successfully retrieved from the measured magnitude response using the proposed method.