What amazon did was the digital equivalent of breaking into your home in the middle of the night and replacing a book on your table with a refund check.
And they didn't just take back what they sold to you. In at least one case, they took more. There was at least one student that had purchased one of the books in question, to complete a summer reading assignment. They took all his work, notes and annotations.
In the physical book world, they could never get away with something like this. They would be treated as criminals if they even tried it.
In the physical book world, the publisher would have been held responsible for their breach of copyright, and if the copyright holder sued them and won, they would be made to pay the copyright holder for that violation, whatever money they made from the sale of those books in addition to some very stiff fines.
Those that had purchased the books would not have to return them. Additionally, I believe that once the publisher is found guilty, the purchaser would be entitled to a refund, if they took that publisher to small claims court or if it was ordered by the court to offer an optional refund to purchasers, making it even more expensive for the violating publisher. But again, nothing would force the end user to return the book for a refund. It would be their option.
Put it this way, in the physical book world, it's the unauthorized publisher that would be royally screwed for the violation, and not the end users. There is nothing in the law that punishes end users that purchase bootleg copies of books, in good faith.
And this whole incident raises another question with regards to Amazon. How carefully are they screening products they are selling to their customers?