There are cases where you can use a sefer for not learning for example you can use another sefer (of equal or less kedusha) to raise the height of the sefer one is learning from. However I couldn't find anything that said specifically that you can't take out a sefer and not learn from it.
In general we should be sure to show great honor and reverence to sefarim, similar to that accorded to Tefillin. [Sefer Chassidim 917 / Beit Yosef O.C. 240:6]
So your litmus test needs to be "Am I treating the Sefer in a respectful manner even though I am not learning from it ?" If yes, e.g cleaning it from dust before Pesach then you don't need to learn from it.
For a list of things that shouldn't be done with seforim you can read this article by Rabbi Ari Elkin He also helps to understand the relationship between a sefer torah and seforim.
Here is a Maaseh Rav from Reb Ahron of Belz ZT"L
The previous Belzer Rebbe - Reb Ahron ZT"L had many personal chumros
including how he would act with Sefrom. If he asked his gabbai to
bring a certain gemorah (or any sefer) to him and by accident another gemorah was
brought then he had the gabbai learn a few lines from the gemorah
before putting it back and getting the correct one.