All the helmet instructions I have ever read always tells you to not put anything between your helmet and your head. Such as a Beanie.
The idea is that the shape of the helmet is important to the impact absorption and it suppose to match the shape of your head. If during impact your helmet can rotate (because there is another layer between you and the helmet) then it makes it less effective. That is why the normal test for helmet is if you try to move the helmet and your forehead skin doesn't shift with it then your helmet is not fitted right.
Never put that claim to the test though, but I do what the manufacturer suggests. I would imagine the goggle strap doesn't matter since it is so low on the helmet, but I always wear it out.