... but with many files I can see no reason for the different modified date / size.
This could be due to metadata, adding metadata will increase the file size without affecting the image.
Usually I resort to something like Beyond Compare or enable the extra metadata fields in DOpus.
My personal preference in this case would be to combine the separate file structures onto another drive, any filename clashes would be auto-renamed.
Then I'd run Antidupl, (set at 0%), over that file structure to remove dupes, (ie. leave the original sources alone).
I've had to do this twice for friends whose backup software was doing recursive backups ... it was fun removing a couple of hundred GBs of dupes 🙄
I found it was the fastest method, do the copies then let Antidupl take care of working out what could be removed based on image data, (they did not have any metadata requirements so I could ignore this).