• Resolved ibiza69

    (@ibiza69)


    Hi! I’m writting because we are facing a weird problem. We have job sites and there is an international company that wants to index our feeds as many other companies do since years ago, but this company tell us that when they try to index (invoke) our feeds, they get bloqued. I spoke with our server admins and is not from that side, so I though that maybe asking you if BPS could possibly block this access to the feeds? and if could, then is there is a way for including at htaccess file a command for whitelisting their IP and where to place it (root or admin) and in which section of your plugin.

    Thanks in advance for any help, we really need to work with this company πŸ™‚

Viewing 15 replies - 1 through 15 (of 24 total)
  • Plugin Author AITpro

    (@aitpro)

    Go to the BPS Security Log page and look for any Security Log entries that show what is being blocked by BPS. You would look for any Security Log entries that contain some identifying info in the log entry. ie an IP address or host or domain of the international company that is trying to index your Feeds. Post one of those Security Log entries so I can take a look at it to see what is being blocked. In general, BPS would not block Feeds, but maybe the method used to index your Feeds is being blocked by BPS.

    Plugin Author AITpro

    (@aitpro)

    Forgot to mention what I think is most likely being blocked and that would be the User Agent in whatever method the international company is using to index your Feeds.

    Plugin Author AITpro

    (@aitpro)

    Yikes you copied your entire Security Log in your forum reply. I’ll have the WordPress Moderators delete that. I’ve copied your Security Log file and am looking through it now. I need something to check for. Please ask the international company that is indexing your Feeds for something I can check for. ie hostname, domain, User Agent, etc.

    Plugin Author AITpro

    (@aitpro)

    Is the company Criteo by any chance?

    Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    Moderator note: I removed the reply.

    Thread Starter ibiza69

    (@ibiza69)

    Hi @aitpro and thanks @sterndata for deleting that for us.

    @aitpro I’ve asked the company if they can provide this information for you.

    Do you think that maybe is because we have to have mod_secure disabled? Our server burned in OVH fire at Estrasbourg and I saw mod_secure was enabled in this new server and in the past If I remmember well (not sure) you told me to disable it.

    When I have this company answer I shall get back to you, thanks once more!

    Thread Starter ibiza69

    (@ibiza69)

    Hi again, this company seems not to provide all this information… So we canΒ΄t do much else on this πŸ™

    One more thing, we are facing problems with ultima member plugin, that profiles are not shown, etc… Do you know if BPS affects in any point in this case and if we have to include anything for this not to happen?

    Thanks once more for your time πŸ™‚

    Thread Starter ibiza69

    (@ibiza69)

    We fixed, the plugin author mistaked the update and so we went back to a lower version and all worked well. Thanks for your time.

    Thread Starter ibiza69

    (@ibiza69)

    Hi @aitpro, we got an answer from the company getting our job feeds error:

    17:54:31 ERROR [app] HTTP/1.1 403 Forbidden returned for "https://www.mydomain.com/indeed-job-feed/". ["message" => "HTTP/1.1 403 Forbidden returned for "https://www.mydomain.com/indeed-job-feed/".","called" => ["file" => "/var/www/vhosts/insertia.net/httpdocs_new/vendor/symfony/http-client/Response/TransportResponseTrait.php","line" => 69],"occurred" => ["file" => "/var/www/vhosts/insertia.net/httpdocs_new/vendor/symfony/http-client/Response/CommonResponseTrait.php","line" => 178]]
    
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n
    
    <html><head>\n
    <title>403 Forbidden</title>\n
    </head><body>\n
    <h1>Forbidden</h1>\n
    <p>You don't have permission to access this resource.</p>\n
    <p>Additionally, a 403 Forbidden\n
    error was encountered while trying to use an ErrorDocument to handle the request.</p>\n
    </body></html>\n

    Thanks hopefully there is a solution πŸ™‚

    • This reply was modified 3 years ago by ibiza69.
    Plugin Author AITpro

    (@aitpro)

    Do you have a website URL/link for the international company that is indexing your Feed? If so, please post the URL/link and take it from there.

    Also to confirm or eliminate that BPS is blocking your Feeds from being indexed do the BPS these BPS troubleshooting steps below and let me know what happens.

    1. On the Security Modes page, click the Root Folder BulletProof Mode Deactivate button.
    2. On the Security Modes page, click the wp-admin Folder BulletProof Mode Deactivate button.

    The most likely thing that could be blocking your Feeds from being indexed would be the BPS nuisance User Agent security rules. So if doing the troubleshooting steps above does work to allow your Feeds to be indexed then try this fix below.

    1. Copy the modified BPS Query String code below into this BPS Root Custom Code text box: 12. CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS
    2. Click the Save Root Custom Code button.
    3. Go to the BPS Setup Wizard page and run the Setup Wizard again.

    # BEGIN BPSQSE BPS QUERY STRING EXPLOITS
    # The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
    # Good sites such as W3C use it for their W3C-LinkChecker. 
    # Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
    # User Agent filters directly below or to modify/edit/change any of the other security code rules below.
    #RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    #RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
    RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
    RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
    RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
    RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
    RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
    RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
    RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
    RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
    RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
    RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
    RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR] 
    RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
    RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
    RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
    RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
    RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
    RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
    RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
    RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
    RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
    RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
    RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
    RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
    RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
    RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
    RewriteRule ^(.*)$ - [F]
    # END BPSQSE BPS QUERY STRING EXPLOITS
    Thread Starter ibiza69

    (@ibiza69)

    Hi @aitpro, I’ve just updated the code as you said and wrote the company for them to test the job feeds.

    Did you see my message? as I think we both wrote at the same time πŸ™‚ Just for confirming.

    Plugin Author AITpro

    (@aitpro)

    Oh and if doing the BPS troubleshooting steps above does not fix the issue/problem then BPS is not blocking your Feeds from being indexed. You would then need to look at likely plugins that could be doing that such as caching plugins, other security plugins or things of that nature.

    Plugin Author AITpro

    (@aitpro)

    I just saw your other posts after posting mine. It looks like you are using a 3rd party app > Symfony. Give me a few minutes to figure this out.

    Plugin Author AITpro

    (@aitpro)

    Ok so the simplest solution would be to create a new RewriteEngine Off .htaccess file and copy it into the /symfony/ folder.

    1. Download the BPS Root htaccess file.
    2. Delete all htaccess code in the BPS Root htaccess file.
    3. Copy this 1 line of htaccess code into the BPS Root htaccess file: RewriteEngine Off
    4. Save your edits.
    5. Upload the .htaccess file into the /symfony/ folder.

    Thread Starter ibiza69

    (@ibiza69)

    I never heard before about Symfony, maybe is contained in some plugin… I saw inded many many kind of apps in the log I sended you but I never used them in our sites. What do you think, as I donΒ΄t know where to find them, at FTP didnΒ΄t πŸ™‚

    Thanks once more!

Viewing 15 replies - 1 through 15 (of 24 total)
  • The topic ‘IPs and whitelist’ is closed to new replies.