Remove Calculate Fee Action Before Order Process

Hello,

I have a custom function which i use to add/remove a custom fee to the Cart Totals. The fee works fine during the Cart Ajax Calculations, but for some reason the fee still gets charged to the order after checkout. How can I remove this before the order is processed? Here is what i currently have to calculate the fee:

function woo_add_cart_fee() {

  global $woocommerce;

		if ( ! $_POST || ( is_admin() && ! is_ajax() ) ) {
        return;
    }

			$checkout = WC()->checkout()->checkout_fields;

			parse_str( $_POST['post_data'], $post_data );

      // Add Fee is no VAT Number is Provided
			if($post_data['vat_number'] == '' OR strlen($post_data['vat_number']) < 1 OR empty($post_data['vat_number'])){

          $vat_total = 25; // $25.00 fee

			  	$woocommerce->cart->add_fee( __('VAT Fee', 'woocommerce'), $vat_total );

			}

}
add_action( 'woocommerce_cart_calculate_fees', 'woo_add_cart_fee' );

The problem is that once the user checks out, the fee is always added, even if they provide a VAT number (my custom field).

So I tried adding this snippet to remove the action completely before the order is processed, but this does not seem to work either:

function action_woocommerce_before_checkout_process( $array ) {
    // make action magic happen here...
      if($_POST['vat_number'] == '' OR strlen($_POST['vat_number']) < 1 OR empty($_POST['vat_number'])){
        remove_action( 'woocommerce_cart_calculate_fees', 'woo_add_cart_fee', 1 );
      }
}

// add the action
add_action( 'woocommerce_before_checkout_process', 'action_woocommerce_before_checkout_process');

Any Idea what could be happening? Thanks!
Regards
-Hector