robots.txt not modified by the plugin

  • postcd

    (@postcd)


    1 day, 14 hours ago

    Hello, I am on a WP 7.0, PHP 8.4 and plugin 1.1.0.

    Settings page contains a checkbox “Add a Disallow rule for the trap base path to keep compliant crawlers out of trap URLs.” which was checked. I do not remember saving this page yet though robots.txt does not contain any disallow rule for the “Trap Base Path” mentioned on same settings page. Clicking “Save changes” button, does not update it (file permissions are -rw-r–r–), file access time is yesterday.
    Maybe this checkbox should ask user to verify change (or do the verification and report if it failed), providing line which needs to be insert by user (i assume “Disallow: /mentioned-trap-base-path-here”)

    Checking debug log, i do not see any related line.

    If plugin is doing any repeated cron task, maybe it should also check existence of the trap path in robots.txt file to avoid blocking bots which were not disallowed to access the maze.

    —-

    Btw. loading the trap path of my website via Tor browser (to avoid being scored/blocked), results in 404 page with text “Sorry, but nothing matched your search terms.”… this is HivePress site. Also the page source code does not contain “Trap Base Path” and analytics page shows count only near active trap pages, no visits.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author yoren

    (@yoren)

    19 hours, 53 minutes ago

    Hello –

    We’ve released 1.2.0 to resolve the robots.txt you reported. Previously we only handle virtual version of it but not the physical file as you described. We also added the detection so it’ll be more clear to the users.

    WP.org now holds off the release until 24 hours later. But you can download a copy and instal it to validate the result.

    As to the 404 issue, it sounds like the pages are not generated properly, a quick test is to flush Permalinks in Settings (just by resaving the Settings – Permalinks) page to see if that helps. Also, as a logged in admin, you can view those trap pages without being blocked so no worries. If after flushing permalinks you still get a 404, please let us know again.

    Thanks.

    Thread Starter postcd

    (@postcd)

    15 hours, 53 minutes ago

    I have downloaded and having active version 1.2.0. For some reason, the behavior described does not change. File robots.txt modification time not changed, file not have new disallow line. Debug log contains no related line. Let me know how to proceed please if you think it should work.

    Regarding 404, only /trap-path-here is not accessible, but subpages in it are accessible and also visible at /wp-admin/admin.php?page=jtzl-bot-maze-trap-pages

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.