The Palomar Radial Velocity Instrument (PARVI) is a J & H band, high resolution (R∼80,000) spectrograph on the Hale 5.08-m telescope at Palomar Observatory. PARVI is a stabilized, single-mode fiber-fed spectrometer designed to search for small rocky planets in the habitable zones of late-type stars. PARVI is one of the first radial velocity (RV) instruments to employ single-mode fibers (SMF) instead of the more common multi-mode fibers (MMF). SMFs provide a number of advantages over MMFs. In particular, they only allow one spatial mode of light to propagate, resulting in a time-invariant, near-Gaussian output beam that eliminates speckle noise present in MMFs. A challenge with SMFs is that the single spatial mode is composed of two polarization states (PS) and the output PS is completely decoupled from the input PS. During the commissioning of PARVI, we discovered a source of polarization noise that manifested as a time-dependent, wavelength-dependent, PSF displacement responsible for RV errors on the order of 10 ms−1 that we call the accordion effect. We set up a warm testbed to test the polarization response of the PARVI optics and determined the cross-dispersing prism and variable PS of injected light were responsible for the polarization noise. We used raytracing software to simulate birefringence in the prism and found it to be consistent with the observed effect. We designed a number of tests with the PARVI spectrometer operating cold and under vacuum to evaluate how modifications to the prism impacted the effect. After mitigating stress-induced birefringence in the prism and installing fiber polarization scramblers before light is injected into the spectrograph, we are able to demonstrate an instrument RV precision of <1 ms−1.