Google has suddenly decided this robots.txt file blocks normal indexing

  • Resolved Tyler Tork

    (@tylertork)


    6 months ago

    It’s unclear what value of User-agent they’re testing with to decide that this site is blocking them from indexing, but this seems to be a recent change that I only got notice from them about today. I tried adding lines to your Custom field to specifically allow Google indexing, but that doesn’t seem to have unblocked them.

    I’m kind of suspecting that they’re being little shits and saying, well if you won’t let us scrape your site for AI training, we’re just not going to show it in search results at all. But I’m not sure how to tell whether that’s really what’s happening.

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author lastsplash (a11n)

    (@lastsplash)

    6 months ago

    Hi @tylertork –

    I looked at your site’s robots.txt file and unfortunately I’m not sure what the specific issue is here. Are you seeing the error in Google Search Console? If so, does it perhaps provide more information?

    It also looks like you have 3 different plugins adding to robots.txt. Does the issue go away if you remove the other plugins/entries and test with just this plugin?

    FWIW, I have multiple sites using this plugin and they are all being indexed by Google, so I’m unsure of the specific issue here.

    Plugin Author lastsplash (a11n)

    (@lastsplash)

    5 months, 1 week ago

    Since there was no response here, I’m going to mark this as resolved for now. Please don’t hesitate to reach out if you continue having issues.

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.