auto set category on custom post type problem

I have setup my custom post type in functions.php and enabled all the categories. All good. I’m creating a plugin which automatically creates a post, assigns it to the custom post type and sets the category as follows:

// assign to custom post type
$new_post["post_type"]       = 'my-custom-post type';
// Insert the post into the database and store the id
$post_id = wp_insert_post($new_post);
// assign to category
wp_set_post_terms( $post_id, intval(3), 'category');

Now the problem is that this all kind of works, BUT say I have 30 posts assigned to this category, when I run the plugin the post gets added to the correct post-type and the correct category is assigned, however when I view categories instead of my-category(31) it shows my-category(3). It stays like this until I go into one of these posts, open a post and update it and the the category shows the correct number of posts.

Interestingly if I remove the function whereby it assigns the post to a custom post type (so it is just a normal post) the category assignment works fine.

Any ideas as I seem to be going around in circles?