A novel phase error compensation method is presented for the reduction of phase error and shape waviness for a three-dimensional shape measurement system based on gray-code and phase-shift light projection. The phase error is mainly caused by the nonsinusoidality of projected phase-shift patterns. According to the system's phase-distribution properties, the proposed method adopts smoothing spline approximation to precisely extract the specific system phase error from the reference phase. This specific system phase error can be used to compensate the object phase during measures. The experimental results show that this compensation can reduce the shape waviness by 70%. In addition the residual object phase error is reduced by a smoothing procedure based on smoothing spline approximation. This procedure shows a 65% reduction in the residual shape waviness.