We investigate niobium-silicon mixed films prepared by plasma-assisted reactive magnetron sputtering. Multilayer films composed of niobium-silicon oxide with various Nb fractions were fabricated. The Nb fractions were calculated using the Brüggemann model and measured by x-ray photoelectron spectroscopy. The morphology of the samples shows that the stress in mixed monolayers depends on the Nb fraction and annealing temperature. A high-reflectivity (HR) multilayer film was fabricated from two mixed-oxide materials with Nb fractions of 20% and 95%, which are optimal for stress self-compensation and a maximum difference in refractive index to facilitate film design. The residual stress of this mixed-oxide multilayer HR film is completely self-compensated through annealing. Although annealing this film redshifts the transmission spectra and increases the surface roughness of the film, the results of cavity ring-down tests indicate that the reflectance remains high in the wavelength band of interest (1064 nm) and spatially uniform over the substrate surface. This detailed characterization of these films shows that the mixed-oxide multilayer HR mirror with self-compensated stress would be appropriate in numerous applications.