To address the segmentation errors caused by the difficulty of determining the threshold values in the conventional method, an automatic segmentation method is proposed in this paper that uses both the object image pattern of the measured gear tooth flank and the modulation values obtained from the interference fringe patterns (IFPs). This segmentation method is integrated into the matching step. The matching step establishes the correlation between the image and the actual gear tooth flank by matching the simulated gear tooth flank image to the real one. First, pixel columns are extracted from both the object image pattern and the IFPs at a specified interval. Then, the boundary points of each column are located by the gray values. Second, the boundary points are optimized by the modulation values and the boundary curves of the gear tooth flank region are fitted by these boundary points. Finally, the simulation gear tooth flank image is matched with the boundary curves of the gear tooth flank region, resulting in the final segmented tooth flank region. The proposed method is verified in several IFPs, and the results show its feasibility and accuracy.