Unfortunately, this is a place where the Twitter API sucks. Basically, if you exclude replies it removes those AFTER it grabs the number of tweets you want, and returns whatever's left. So if you want 5 tweets and 3 of your most recent tweets are replies, it will only give you two. You can see the API docs here:
I used to pull a bunch of extra tweets and then throw a bunch out, but it wasn't near as efficient. Now that I'm using a built-in caching system I could probably go back to that, but I don't know when I'll have time to do so.