Sorry, I can't really help you on that one, it would require some serious debug...
However, I am using something close to Masonry on one of my website and I support retina as well. I didn't do anything special. You can check the website: http://www.haikyo.org. On each page/post you can look at the images used at the end, this is what you want to achieve it seems...