FWIW I think that from a legal perspective it's a matter of if you will get prosecuted or sued successfully. In this shifting realm there are few guarantees, though you also get the concept of the victim , i.e. who was harmed, how, and how much... Pragmatically it often has little to do with fairness, & much to do with legal resources available & your true cost of litigation.
Totally separate is the moral realm... If we all acted out of the highest moral standards, our world would not have any disparity of income, & in fact perhaps no income as we know it at all. But we're not there yet, obviously, so morals, as most of us practice them anyway, are the closest we can (or are willing to) come towards some ideal. Debating a topic like this is really comparing, exchanging ideas and concepts forming our individual level of compromise. We can and do rationalize our individual stance, but ethics and morals are thought of and taught as absolutes with intentionally very little wiggle room: it *might* be OK to shoot someone in self-defense, but that argument doesn't fly if you broke into their home and took preventative action.
Now the original intent of copyright was to promote new things and art and literature; who would devote time that could be spent earning room and board to new pursuits, if an outcome of starvation was an absolute certainty? Since no one was willing to give potentially everyone a blank check to just go off and create something, s/he had a right to charge customers for their labor without fear of someone selling duplicates & thus reducing their income.
If the owner has no intent of deriving [further] income from their work, then according to it's intent and original purpose any copy right has fulfilled it's purpose, & should logically lapse with the owner releasing any holds or claims. Unfortunately that's a bit rare, so we set an arbitrary date when enough's enough & put it into law. Therein lies the problem...
Something can have absolutely zero worth, as in this case an out of print book, which as long as it is kept out of print, has absolutely no monetary value - - - *until* someone wants to buy it. Morally, or at least in the spirit Copy Right was intended, the owner should have given up any exclusive rights the moment they decided not to print the book any longer. But let's say they didn't, maybe for legitimate reasons that made another run impossible - maybe out of greed the same way some people will hoard food knowing that it exceeds their needs and much of it will spoil.
If the copy right holder is being greedy, as many assuredly are, do they revoke their rights to be treated equally? In my opinion the popular consensus is no -- without advocating for or against, I'd say it's basically the same argument we have often enough, globally today: when do your actions cause you to forfeit rights?
Which all may be moot anyway if the mortgage problems lead to proposed changes in how we legally & commonly calculate value... If it becomes common practice to value any property on what it's worth, not what it might be worth *if* there was a buyer, then grabbing an out of print book may be the same thing as dumpster diving or picking something out of the trash. If you can in good conscience consider an out of print book worthless, taking it is perfectly moral & ethical, depending of course on what you intend to do with it... You could pick up a bat from the trash and beat your neighbor with it & that's just wrong.