Remove or Restrict Server File Browser

Is is possible to remove the server file browser from the “Add New” page, or at least restrict it to a certain directory? It seems unsafe to have full access to the root of the site directory.

http://wordpress.org/plugins/download-manager/