Validation on not required field

Hi,
I added the billing phone field into myaccount/edit-account page by editing the form-edit-account.php template.

I added those codes to check, validate and save the field but I need also to make this field not required so I added also the code below.

The problem is that the field can’t be left blank and the validation gives always the error message “Please fill the phone number”.

How can I keep the validating codes and make the field not required.

// Check and validate the phone field
add_action( 'woocommerce_save_account_details_errors','billing_phone_field_validation', 20, 1 );
function billing_phone_field_validation( $args ){
    if ( isset($_POST['billing_phone']) && empty($_POST['billing_phone']) )
        $args->add( 'error', __( 'Please fill the phone number', 'woocommerce' ),'');
}

// Save the mobile phone value to user data
add_action( 'woocommerce_save_account_details', 'my_account_saving_billing_phone', 20, 1 );
function my_account_saving_billing_phone( $user_id ) {
    if( isset($_POST['billing_phone']) && ! empty($_POST['billing_phone']) )
        update_user_meta( $user_id, 'billing_phone', sanitize_text_field($_POST['billing_phone']) );
}

//Remove required field requirement
    add_filter('woocommerce_save_account_details_required_fields', 'remove_required_fields');

        function remove_required_fields( $required_fields ) {
            unset($required_fields['billing_phone']);

            return $required_fields;
}

Thank you to anyone can help!