I’ve had a similar issue with filtering. On WP v3.7.1, and I’ve turned off all plugins.
I’ve noticed an issue with the url. It looks like s& is appended to /upload.php? that is preventing a successful filter.
Click Filter once (fail):
wp-admin/upload.php?s&action=-1&m=0&media_category=11&paged=1&action2=-1&affected&_ajax_nonce=fe0127a0dd&ps
Clicking filter a second time (success):
wp-admin/upload.php?action=-1&m=0&media_category=11&paged=1&action2=-1&affected&_ajax_nonce=fe0127a0dd&ps
This fail/success occurs with each subsequent “Filter” click.
Hope this helps.
