The observed intergalactic O VI absorbers at z>0 have been regarded as a significant reservoir of the "missing baryons." However, to fully understand how these absorbers contribute to the baryon inventory, it is crucial to determine whether the systems are collisionally ionized or photoionized (or both). Using the identified intergalactic O VI absorbers as tracers, we search for the corresponding X-ray absorption lines, which are useful for finding the missing baryons and for revealing the nature of the O VI absorbers. Stacking the Chandra grating spectra along six AGN sight lines, we obtain three spectra with signal-to-noise ratios of 32, 28, and 10 per 12.5 mÅ spectral bin around the expected O VII Kα wavelength. These spectra correspond to O VI absorbers with various dynamic properties. We find no detectable Ne IX, O VII, O VIII, N VII, or C VI absorption lines in the spectra, but the high counting statistics allows us to obtain firm upper limits on the corresponding ionic column densities (in particular on average at the 95% confidence level). Jointly analyzing these nondetected X-ray lines with the averaged O VI column density, we further limit the average temperature of the O VI-bearing gas to be T 105.7 K in collisional ionization equilibrium. We discuss the implications of these results for physical properties of the putative warm-hot intergalactic medium and its detection in future X-ray observations.