The word as used in english. The story you heard is right. As you say, the word was adopted to describe the trousers, but the specific uniform trousers, not sandy colored ones. It was those specific sand colored uniform trousers made of tight tan cotton (or linen) twill. It wasn't long before trousers with similar material and construction were described with the same word, partially because the British Army screwed around with the shade all the way to an olive, and everyone just used the same word to describe it. Then the Americans got into the game, using surplus British (olive) khakis to outfit our army. Then WWI and WWII happened, with US and British forces wearing khakis in two different colors depending on weather, and well, the game was lost.
As with every other problem on this planet, blame the Brits. I haven't corrupted it, it's just never been as clear and pure as you imagine it to be. I suppose the "pure" position would be that it could be applied to either tan or olive, and no other colors, but really, once you allow more than the color you traditionally think of as "khaki", the game is lost.
This is a big reason why I love the english language. It often makes no fucking sense at all, but there's usually a pretty good story as to why.