Confirmation email not sending when people subscribe

I have the jet pack subscribe via email widget on my blog but some people do not receive the confirmation email after putting their email in the widget and signing up. I did a test with 3 email addresses to see if it will work. My gmail address works. My husband and I both have a couple of yahoo emails and neither one of those will get the confirm email. I checked the spam folder also to see if maybe they went in there. I found this topic in your forums but most were old topics I tried installing WP Mail SMTP as suggested and that did not seem to work.
my blog is http://terrisiebert.com/. I have been having a bunch of odd email address of subscribers signing up so I do know it works … sometimes.