Hi, those are real links that Directorist creates, so the bots are just following them.
On each Author Profile page there is a “Filter by category” option, and the ?category= value filters that author’s listings by one listing category. The dropdown lists every listing category on the site, not only the categories the author actually uses. So every author page links to every category, most of these filtered pages show no results, and bots crawl all of them. That’s why you see the same pattern on other Directorist sites.
They are harmless, but if you want to reduce the crawling you can:
- Block them in robots.txt, for example:
User-agent: *
Disallow: /author-profile/*?category=
(adjust the path if your author profile pages use a different slug)
- Or set these filtered author pages to noindex in your SEO plugin.
Showing only the author’s own categories in that dropdown would be a good feature request for the Directorist team.
Which SEO plugin do you use? I can point you to the right setting for noindex.
Thanks. The SEO Framework does not have a setting to noindex only these ?category= links. Its noindex option on the Author Profile page would hide all author profiles, because Directorist shows every author through that one page, so I would not use it here.
The robots.txt rule is the better way. The SEO Framework makes a virtual robots.txt and has no box to add your own rules, so create a robots.txt file in your site’s main folder (the same folder as wp-config.php).
Once a real file exists, the virtual one is no longer used. So first open yoursite.com/robots.txt in your browser, copy what it shows into the new file, then add:
User-agent: *
Disallow: /author-profile/*?category=
If the file already has a User-agent: * group, just add the Disallow line under it. Google, Bing and other well-behaved bots will stop crawling those links. Bad bots ignore robots.txt anyway.
Thread Starter
dimal
(@dimalifragis)
thanks for the reply. Actually Claude AI made me a small snippet to display a 404 to everything after the “user name”.
So all those urls show a 404 now.
Hi @dimalifragis,
We checked the code and reproduced the issue. The ?category= parameter is a valid author listing filter, but the dropdown includes categories with no listings for that author. This creates unnecessary links to empty results, so our earlier description of them as harmless was incomplete.
We’ve prepared and locally tested a fix that shows only relevant categories for the author and selected directory, and returns HTTP 404 for category filters with no matching listings. The fix is awaiting review and hasn’t been released yet.
Could you share the snippet you’re using, with any private details removed? Blocking everything after the username could also affect valid category filters, directory links, and pagination.
Best Regards,
Sohan Chowdhury
Thread Starter
dimal
(@dimalifragis)
Here it is.
/**
* Return a proper 404 for nonexistent Directorist author profiles.
*/
add_filter( 'template_include', function ( $template ) {
if ( ! function_exists( 'get_directorist_option' ) ) {
return $template;
}
$profile_page_id = (int) get_directorist_option( 'author_profile_page' );
// Only apply to the designated Directorist Author Profile page.
if ( ! $profile_page_id || ! is_page( $profile_page_id ) ) {
return $template;
}
$profile_slug = get_query_var( 'author_id' );
// No username or ID supplied.
if ( empty( $profile_slug ) ) {
return directorist_profile_404_template();
}
// Look up the user by ID or username.
$user = is_numeric( $profile_slug )
? get_user_by( 'id', absint( $profile_slug ) )
: get_user_by( 'login', rawurldecode( $profile_slug ) );
// Nonexistent user: return the 404 template.
if ( ! $user ) {
return directorist_profile_404_template();
}
// Existing user: preserve the original template.
return $template;
}, 99 );
/**
* Set the HTTP 404 status and return the theme's 404 template.
*/
function directorist_profile_404_template() {
global $wp_query;
$wp_query->set_404();
status_header( 404 );
nocache_headers();
return get_404_template();
}
Feel free to remove it if not suitable
Hi @dimalifragis
Thanks for sharing the snippet. This clarifies its behaviour: it checks whether the author exists and returns a 404 when the username/ID is missing or invalid.
For an existing author, it preserves the page, including category filters, directory links, and pagination. That means a URL such as /author-profile/existing-user/?category=unrelated-category would still pass through this snippet, even when that author has no listings in the selected category.
The fix we prepared addresses that separate issue by limiting the dropdown to relevant categories and returning 404 for category filters with no matching listings.
One detail to keep in mind: your snippet also makes the Author Profile page without a username return 404, including for logged-in users who would otherwise see their own profile.
Best Regards.
Thread Starter
dimal
(@dimalifragis)
Thanks for Clarifications
I will check it further.
Thread Starter
dimal
(@dimalifragis)
Just did a test
1. Disabled the snippet
URL: https:/author-profile/starassociatespmc/?category=electronics-appliances
Displays the Author and ZERO listings (since this is an unralated caegory). The User has a listing in an other category.
2. Enabled the snippet
Goes to a 404
For me it seems fine.
Hi @dimalifragis,
Thanks for testing and sharing the result. That confirms the snippet changes the response on your site, although the code you shared does not directly check the category.
Could you also test the same author’s profile without ?category=, and then with a category that contains their listings, while the snippet is enabled? Both should remain accessible.
If those valid pages work and only the unrelated category returns 404, you can keep the snippet temporarily. Our category-filter fix is still awaiting review and has not been released yet.
Best Regards.
Sohan
