We investigate an optical vector network analysis technique based on optical single-sideband modulation that can provide spectral characterization of optical components in terms of amplitude and phase shift with femtometer wavelength resolution. The fundamentals of the system are analyzed and a mathematical model for the measurement process is derived. Using this model and experimental results, the potential of the technique in terms of frequency resolution is demonstrated. However, it is also found that the measurement accuracy relies on a high suppression of the residual sideband in the optical single-sideband modulation deployed. © 2004 Society of Photo-Optical Instrumentation Engineers.