Three major sources of nonlinear error in the common-path laser heterodyne interferometer (CLHI), which include misorientation of incidence beams relative to the Wollaston prism, elliptical polarization of laser beams, and misorientation of two linear orthogonal polarization beams relative to metal-coated mirrors, are theoretically investigated. The results show that the error from the Wollaston prism is mainly second harmonic, and the error from the elliptical polarization of lasers is first harmonic. We discover that the misorientation of the metal mirror can also generate nonlinear errors, which has rarely been studied before. When the polarization directions of the two linear orthogonal incidence beams are not parallel (or vertical) to the normal plane of the metal mirror, the reflecting beam will become an elliptic polarized beam with orientation of the two ellipses having nonorthogonal and nonequal eccentricity. The frequency mixing may occur when the beam reflecting from the mirror passes through a Wollaston prism. The error caused by the frequency mixing is mainly first harmonic. Additionally, error compensation methods are also discussed to improve the accuracy of the interferometer. © 2002 Society of Photo-Optical Instrumentation Engineers.