In the near infrared (NIR) band (1-2.3 m), the main source of emission in the night sky is from lines  in the mesosphere. Beyond 2.3 m, and lines situated at lower altitudes dominate. However, at these wavelengths thermal emission from the telescope and optics overwhelms any atmospheric emission . Non-thermal emission from the NIR sky varies on timescales of 10-15 minutes by 10% or more making it difficult to estimate a nominal value for the background.
The water column above a site is another variable in the sky spectrum. Weather data from the San Jose, CA, airport weather station for the first half of 2013 1 is plotted in Figure 1. The mean and median precipitable water vapor for that site is 11 mm. Lick Observatory is situated on Mt. Hamilton at an altitude of 1284 m. This is a higher and drier site than the airport so assuming a water column value of 10 mm is a reasonable estimate.
For the purposes of this exercise we chose sky emission and transmission models available on the Gemini Observatory website 2 for Cerro Pachon (altitude 2700 m) with an airmass of 1.0, water column of 10 mm and with a spectral resolution of . The spectra are constructed with high-resolution sky transmission data generated by ATRAN  to which a 280 K blackbody spectrum for sky emission, OH and O lines, and zodiacal light have been added. The effects of moonlight (which would be strongest in the band) have not been accounted for. Water column and airmass make a difference in the band as seen is Fig 2, where telescope emission dominates so the predicted magnitude limits for ShaneAO should hold for water columns up to 20 mm or an airmass of 2.0 for a 10 mm water column.
Srikar Srinath 2013-10-09