WordPress.org

Support

Support » Plugins and Hacks » [Resolved] How to allow a specific bot (wget) access

[Resolved] How to allow a specific bot (wget) access

Viewing 10 replies - 1 through 10 (of 10 total)
  • Plugin Author AITpro
    Participant

    @aitpro

    wget is explicitly blocked in your root .htaccess file in these BPS security filters below. you can remove/delete wget from these filters, but the better/safer approach would be to use lynx -source instead of wget -O for CRON commands. If the plugin does not allow you to change the command or use different commands then you will just have to remove wget from the security filers in your root .htaccess file.

    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
    
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    Plugin Author AITpro
    Participant

    @aitpro

    Please post a status update or if the issue/problem is resolved then resolve this Thread. Thanks.

    Plugin Author AITpro
    Participant

    @aitpro

    Please post a status update or if the issue/problem is resolved then resolve this Thread. Thanks.

    Plugin Author AITpro
    Participant

    @aitpro

    Please post a status update or if the issue/problem is resolved then resolve this Thread. Thanks.

    Plugin Author AITpro
    Participant

    @aitpro

    Resolving thread due to lack of response. If the problem is still occuring please reopen/unresolve this thread and post a status update. Thank you.

    NinthRealm
    Member

    @ninthrealm

    Sorry for the delay. Finally got the client to remove wget from the .htaccess filters but the page still returns a 404 error. I also noticed that when using a web browser the header also returns a 404 error, but a page does render.

    lynx is not a viable solution because of large post payloads that are being sent with wget.

    Status Complete
    Response Code 404 Not Found
    Protocol HTTP/1.1
    Method GET
    Content-Type text/html; charset=UTF-8

    Plugin Author AITpro
    Participant

    @aitpro

    A 404 error usually indicates that the site has a permalink structure problem or some other URL/linking problem.

    Which web host is this? Does this host use cPanel?
    What is the custom permalink structure? Example: /%postname%/
    Have you put BPS in Default Mode to eliminate/verify that BPS is causing this issue/problem?

    1. Make a backup of your .htaccess files using BulletProof Security built-in Backup.
    2. Activate Default Mode on the Security Modes page.
    3. Use the Delete wp-admin .htaccess feature on the Security Modes page.
    4. Test your plugin or theme.
    5. Restore your .htaccess files using BulletProof Security built-in Restore.

    To completely uninstall BulletProof Security you would do steps 2 and 3 above and then just delete the BulletProof Security plugin on the WP Plugins page.

    Plugin Author AITpro
    Participant

    @aitpro

    Also since you said you added a skip rule for this plugin. Please post that skip rule.

    NinthRealm
    Member

    @ninthrealm

    Host does not use cPanel, it’s a Debian Distro.

    Everything worked before client installed BPS (at least had no issues with wget script uploading into file.

    Here’s the skip rule

    #Risk Survey Spreadsheet Skip
    RewriteCond %{REQUEST_URI} ^/members/wp-content/plugins/Biz-risk-survey/ [NC]
    RewriteRule . - [S=13]

    There is no permalink for this particular url. It is a custom script designed to executed directly with minimal tie-in to wordpress. I can provide more information via a less public venue. If you need more information, please email me at matt@ninthrealmstudios.com

    Plugin Author AITpro
    Participant

    @aitpro

    So what happened after you put BPS in Default Mode? Did the custom plugin/script work?

    When you say “There is no permalink for this particular url.” I do not really understand what you are saying. A plugin folder or plugin file would not have a permalink. Permalinks are for Posts and Pages. Maybe you mean URI?

    If you have a custom script and lets say it is named example.php then you can just do something like this.

    RewriteCond %{REQUEST_URI} (example\.php) [NC]
    RewriteRule . - [S=13]

    But since you are not getting a 403 Forbidden error then a skip/bypass rule is probably not going to be what you need. You need to find out why the URL is not being found – 404 Not Found error.

Viewing 10 replies - 1 through 10 (of 10 total)
  • The topic ‘[Resolved] How to allow a specific bot (wget) access’ is closed to new replies.