Exclude internal search results URLs from crawlers | WordPress.org

Resolved idearius
(@idearius)

3 years, 2 months ago

Hello. Sorry if this is a bit off-topic.

Is there a way to exclude internal search results URLs from being crawled by Google and other search engines?

We’ve seen a rise of internal search results URLs listed in google.com, that contain spam text and URLs.

These indexed URLs from clean (not hacked) websites are not actual pages, but somehow spammers are abusing the internal WordPress search system. They look like http://www.ourdomain.com/?s=ExternalSpamWebsiteAddress+SpamText.

Viewing 2 replies - 1 through 2 (of 2 total)

Plugin Author Micah Wood
(@wpscholar)

3 years, 1 month ago

@idearius This plugin won’t do it, but you can do it with an SEO plugin like Yoast: https://yoast.com/blocking-your-sites-search-results/
Thread Starter idearius
(@idearius)

3 years, 1 month ago
Thanks for your answer, Micah.

I saw Yoast can do that, but I really don’t need the rest of the plugin, so it would have been overkill.

After failing to find specific info for WordPress Search URLs, in the end I used the robots.txt file to disallow URLs with certain parameters:
```
User-agent: * 
Disallow: /?s=*
```
Google’s Search Console robots.txt tester didn’t complain 😉
https://www.google.com/webmasters/tools/robots-testing-tool

Viewing 2 replies - 1 through 2 (of 2 total)

The topic ‘Exclude internal search results URLs from crawlers’ is closed to new replies.

In: Plugins
2 replies
2 participants
Last reply from: idearius
Last activity: 3 years, 1 month ago
Status: resolved