If you don't want to turn off mod_security altogether, you can turn off the global rule that forbids directory listings. mod_security considers that to be "information give-away". I had the same problem with access upon logging in to the WordPress admin page, and the following was reported in the mod_security audit log:
Message: Access denied with code 403 (phase 4). Pattern match "(?:<(?:TITLE>Index of.*?<H|title>Index of.*?<h)1>Index of|>\[To Parent Directory\]<\/[Aa]>
)" at RESPONSE_BODY. [file "/usr/local/apache2/conf/modsecurity/modsecurity_crs_50_outbound.conf"] [line "54"] [id "970013"] [msg "Directory Listing"] [severity "WARNING"] [tag "LEAKAGE/INFO"]
Sure enough, on line 54 of one of the rulesets, this is preventing access to the directory listing:
52 # Directory Listing
53 #SecRule RESPONSE_BODY "(?:<(?:TITLE>Index of.*?<H|title>Index of.*?<h)1>Index of|>\[To Parent Directory\]<\/[Aa]>
54 # "phase:4,t:none,ctl:auditLogParts=+E,deny,log,auditlog,status:403,msg:'Directory Listing',id:'970013',tag:'LEAKAGE/INFO',severity:'4'"
As you can see, I just commented out that one rule. Then I re-started Apache. No access errors.