Sort by custom field, or lack thereof

Hi, I’ve found lots of posts about sorting by the value of a custom field, most of which miss the fact that the wp_query class now makes this a bit easier using meta_value_num – but I haven’t been able to find anything to sort by a meta_value without excluding all posts lacking the relevant meta_key. What I want is a page which lists posts with a ‘priority’ value set first, but includes those without. All the ways I’ve seen of sorting my the value of a custom field rely on directly or indirectly calling SQL with the equivalent of ‘WHERE meta_key=priority’ so that it knows which key’s value to use.

What I have at the moment is this (adapted from the PageOfPosts template found as an example here on WordPress.org):

$args=array(
    'category__in' => array($cat),
    'meta_key' => 'priority',
    'orderby' => 'meta_value_num, date',
    'order' => 'DESC',
    'paged' => $paged,
    'posts_per_page' => $post_per_page,
    'caller_get_posts' => $do_not_show_stickies
  );
  $temp = $wp_query;  // assign orginal query to temp variable for later use
  $wp_query = null;
  $wp_query = new WP_Query($args);

I guess I could just call this twice, once with the above and once excluding posts with the relevant custom field, but that seems awfully kludgey. Is there a neater way?