Robots.txt file is not updating correctly in multisite

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support Steve M

    (@wpsmort)

    8 years ago

    It all depends on whether you set the multisite up as sub domain or sub directory. If it’s sub directory then each site will have it’s own directory and you can place the robots.txt file in that directory.

    Thread Starter Mort

    (@rugbert)

    8 years ago

    AH I see, ok we had to add a custom robots.txt file to stop bots from hitting the site as much (using crawl-delay) and that is being readin instead of the virtual robots.txt file

    Is there anyway you could add that option to the plugin so we could use site specific robot.txt files?

    Plugin Support Steve M

    (@wpsmort)

    8 years ago

    Do you mean adding crawl-delay as an option in our Robots.txt Generator module?

    Thread Starter Mort

    (@rugbert)

    8 years ago

    Yes indeed, that would be great. I’ve had sites taken down by bots and setting the crawl delay helped alot.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Robots.txt file is not updating correctly in multisite’ is closed to new replies.