I went through a deactivated all of my plugins and went back to the wordpress theme.
The only thing that made the lag was the Yoast Plugin.
This may not exactly be a Yoast issue. Have you tried the P3 Profiler plugin to see what is loading the most?
Sometimes strange permalink behavior may be caused by a plugin or theme conflict. The fastest way to rule out conflicts is to deactivate all non-Yoast plugins and switch to a standard theme like Twenty Fourteen. If this resolves the issue, you have a conflict and should activate one plugin at a time until you locate the conflict. Finally, after all plugins are active, switch to your theme. This will narrow down the conflict to a specific plugin or a theme conflict.
If you are unfamiliar with checking for conflicts, we wrote a small guide that will walk you through the steps. You can find it here: How To Check For Plugin Conflicts: http://kb.yoast.com/article/253-how-to-check-for-plugin-conflicts
