the comments url used by the plugin is subtly different. You need a couple of parameters on the end.
Where blogid is replaced.
Assuming that also returns without error then we can have a look at the count that it's trying to retrieve. If you look in the results of the comments, just below the author section you should see something like this.
<generator version="7.00" uri="http://www.blogger.com">Blogger</generator>
In my example there are just 8 comments to process.
It could be that the values in your feed are causing the plugin a problem?