How to display most popular media files from all posts?

I’m looking a way to display the most popular videos and most popular photos in the sidebar.

Ideally they would be ranked by number of times the play button was clicked, for videos, and the number of times the thumbnail was clicked, from a photo gallery.

Im not sure how possible that is… so an alternative would be basing the media rankings on the number of times the post they reside in has been viewed (even if the video from said post was never actually played)

If anyone has some ideas on how to execute this they would be greatly appreciated.