3 Level navigation problems

I am still fooling around with this dumb navigation menu. The idea is that when you first go to the page the top level pages are listed, when one of those is clicked the second level menu appears below it with all of the selected pages sub pages and when one of those pages is clicked all of the 3rd level pages from the selected 2nd level page show up below the second level pages.

I have tried to gewt this done various ways, I even tried to make a session store to remember some parts however I just cannot work out a good solution.

Here is what I have done so far:

<?php
wp_nav_menu(array(
    'sort_column' => 'menu_order',
    'theme_location' => 'primary-menu'
));// Prints the primary top level menu
$level = count($post->ancestors);
echo $level;// Just for testing purposes
if ($post->post_parent){
    $children = wp_list_pages("title_li=&child_of=" . $post->post_parent . "&echo=0&depth=1");
     $grandchildren = wp_list_pages('title_li=&child_of='.$post->ID.'&echo=0');
}
else{
    $children = wp_list_pages("title_li=&child_of=" . $post->ID . "&echo=0&depth=1");
}
if ($children) {
    echo $children;
}
if ($grandchildren){
    echo $grandchildren;
}
?>

Here is the problem that I am having. It works one way, so if I click on top level the second level appear, if I click a second level the 3rd level appear but when i click a 3rd level the second level disappears.

This is driving me nuts! I have spent a day on this now reading all about get_post get_children etc etc etc.

Thanks,