Unable to do WP_User_Query with meta filter

I have a user base of hundreds of users, and trying to create a custom API endpoint to get users in a specific membership group.

The database structure is as follows (using Ultimate Member plugin):

User is stored in wpma_users table
User metadata in table wpma_usermeta contains user meta named mygroups, containing a value a:1:{i:0;s:3:”155″;}
The 155 value is the ID of a term
I already tried this, with $groupid=155:

$args = array(
            'fields'    => 'all_with_meta',
            'tax_query' => array(
                array(
                  'taxonomy' => 'mygroups',
                  'field' => 'term_id',
                  'terms' => $groupid,
                ),
            ),
);

$user_query = new WP_User_Query($args);`
When I var_dump get_terms(array(‘include’ => $groupid)), an actual term is printed:

array(1){
    [
        0
    ]=>object(WP_Term)#27633(10){
        [
            "term_id"
        ]=>int(155)[
            "name"
        ]=>string(8)"Example Group"[
            "slug"
        ]=>string(8)"example_group"[
            "term_group"
        ]=>int(0)[
            "term_taxonomy_id"
        ]=>int(155)[
            "taxonomy"
        ]=>string(11)"um_user_tag"[
            "description"
        ]=>string(0)""[
            "parent"
        ]=>int(149)[
            "count"
        ]=>int(1)[
            "filter"
        ]=>string(3)"raw"
    }
}

Another way I tried, is using the meta_query instead of the tax_query:

'meta_query' => array(
            array(
                'key' => 'mygroups',
                'value' => $groupid,
                'compare' => 'LIKE',
            )
),

This actually filters the query, but when I use $groupid=1, it filters where groupid contains 1, so also 10, 11, 100 etc.

How do I find users with a specific value that’s listed in the mygroups user meta field using WP_User_Query?