I would use a very simple model, and assume that the item has only one temperature.
Also i don't think that this will change the result, the real thing is more complicated.
As far as i know, the thermal energy flux is dependent on the thermal energy difference. So the time-dependent solution is something like an exponential function. That means that the temperature of the item converges exponentially to the temperature of the surrounding. This means you have to wait very long, if the difference is very small.
In your example, the item gains quite fast energy if it is taken out of the fridge, as the thermal difference between item and air is large.
If you put it back, the difference is small, and it needs long to equilibrate.
So the energy it will give to the fridge is the same as it got from the room, but it needs longer to give it than to get it.
If you would put it short in the fridge, and than back in the room, it would loose its energy in the fridge faster, than it would get it back in the room.
Additional there are some higher order effects, as the air in the fridge might have, due to an different density and different amount of water in it, another heat capacity, but this shouldn't be to important for this specific question (fridge + room temparture, both air)