if post count = 1 use single-catname.php

I’d like to force a category to actually use the single-categoryname.php template instead of the normal category-name.php template if the post count is 1. Here’s the code I’m currently using without success:

add_action('category_template', 'use_single_for_one_category');
function use_single_for_one_category($template = '') {
        global $wp_query;
        if ( 1 === (int) $wp_query->post_count ) {
                $template = get_single_template();
        }
        return $template;
}