I know the file path was correct (I double checked it) because I could download the file and add that to the count, but if I edited/updated the download post it reset it back to zero. Anyhow, what I did was deleted all of the imported download posts and then just added them manually and everything seems fine now. I only have 9 files to download at the moment, so it was not a huge inconvenience. I did not test editing the download count box before adding the files manually. It seems to work fine now though.
I did find one other thing I forgot to mention. When I added
<p> to the "before=" part of the shortcode it did not set the paragraph tag there, instead it added
<li>. The "after" tag did set it to
</p> like I told it. I ended up setting them to an empty string (before/after = ""), which worked fine and did not add any HTML before the links. I then added the tags I wanted to a custom template.