Page Not Found

  • Resolved delta5

    (@delta5)


    5 years, 11 months ago

    Hi all,

    I have a whos online plugin installed on my blog, and it shows what bots are lurking and what they are doing. Im seeing a lot of this:

    3 Bots Online Now
    #1 – Bing 157.55.39.127 on July 14, 2020 @ 1:49 pm
    FedUpWithLiberals.comPage not found – FedUpWithLiberals.com [url]

    #2 – Facebook 173.252.95.16 on July 14, 2020 @ 1:46 pm
    FedUpWithLiberals.comPage not found – FedUpWithLiberals.com [url]

    #3 – Archive.org 207.241.231.163 on July 14, 2020 @ 1:45 pm
    FedUpWithLiberals.comPage not found – FedUpWithLiberals.com [url] [referral]

    I do see indexing bots looking at other things, like archives and even topic tags.
    Im sure something is not configured properly. Anyone have any ideas?

    Thanks,
    Richard

Viewing 4 replies - 1 through 4 (of 4 total)
  • devnihil

    (@devnihil)

    5 years, 11 months ago

    @delta5 Are any of the pages that the bots are visiting set to ‘noindex‘ or have you disallowed them in some other way in which you think the Yoast SEO plugin is not operating as expected?

    As for what these bots are doing, they are likely indexing pages or getting metadata from the URL in question. Bing is a well known search engine. Facebook bots will typically scrape your page for metadata anytime someone shares a link (either in a post, via Messenger, etc). This is to provide link preview data (preview image, title, description). As for archive.org, that is a site which provides a utility called Wayback Machine, which essentially archives the appearance of websites over time. For example, it allows someone to view what a particular URL or site looked like 10 years ago.

    That being said, none of these bots are ones you would typically disallow from indexing of your site. But, if you want to disallow particular bots, you can do so by editing your site’s robots.txt settings. We also have more information on editing the robots.txt file using Yoast SEO here: https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/

    Thread Starter delta5

    (@delta5)

    5 years, 11 months ago

    I will have to check that. the site is still being built and hasnt been optimized yet. im sure yoast hasnt been set correctly. I will check the robots.txt file. I will update you when i have more info.

    Jerlyn

    (@jerparx)

    5 years, 11 months ago

    @delta5
    We’ll be leaving this thread open for a few days and will wait for your update.

    devnihil

    (@devnihil)

    5 years, 10 months ago

    We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Page Not Found’ is closed to new replies.

Tags