No it shouldn`t stress your fish at all
I unplug the filter, heater and lights when I do any water changes/cleaning of my tanks, the fish are fine with it. Just ensure the water you`re putting back in is the same (approx) temperature as the water you`re taking out.
A friend of mine once left her heater plugged in whilst she did a large water change, her heater actually exploded!!
I just feel better about mine being unplugged 8)