LiteSpeedCache blocks Screaming Frog crawling

  • vickmckey

    (@vickmckey)


    2 years, 9 months ago

    We want to crawl with Scraming Frog SEO Spider a preprod website that is opened to my IP.

    • we cannot and get a noindexn nofollow status on the homepage disallowing the crawl to go further
    • we desactivate RankMath and a few plugins, assuming they are blocking the crawl
    • nothing happened, I still just get the homepage crawl and check out for the noindex, nofollow : still there. Yet all the SEO plugins and general settings in WordPress allow the crawl.
    • we start a/b testing and figure out LiteSpeed is blocking the crawling
    • There is an tricky option in LiteSpeed localized in the Crawler options : Crawler:Inactive using the htaccess to put a noindex, nofollow tag in the file.

    The page I need help with: [log in to see the link]

The topic ‘LiteSpeedCache blocks Screaming Frog crawling’ is closed to new replies.