W3 Total Cache
Can't Find robots.txt (7 posts)

  1. ibmethatswhoib
    Posted 4 years ago #

    I use W3 Total Cache on a couple websites and can't find the robots.txt, it's showing up on http://www.webstie.com/robots.txt but it's not in the file manager or ftp. I contacted support and they said it's in the database. Is there a way to change the robots.txt through the plugin settings I don't see it anywhere? All I want to do is add some things but can't find the actual file or where to change it.



  2. Pothi Kalimuthu
    Posted 4 years ago #

    There is no setting to change this file and I'm unsure how it is related to this plugin. Anyway, to modify this file, please do the following...

    - Have a copy of what you see in your site's robots.txt file
    - Create a new file named robots.txt file via FTP or 'File Manager' in the correct folder (probably in the folder where WordPress files reside)
    - Paste the previously copied text to robots.txt file
    - Do whatever you wish with the newly created file

  3. ibmethatswhoib
    Posted 4 years ago #

    Huh, the plugin created the robot.txt file so it's pretty related. I did create a new robot.txt file and added it to the main folder but it's still showing the one W3 Total created, it's not over-riding it.

    So does anyone know how to change the file within W3 Total, or how to access it?

  4. napok
    Posted 4 years ago #

    The reason why you are unable to access the actual file is because:

    - the real file is named something else but can be accessed as robots.txt through a rewrite rule
    - there is a file permission problem with your hosting provider

    You would have to check your .htaccess or nginx.conf for possible rewrite rules. However I am quite certain it is the latter which can only be resolved by your hosting provider.

  5. ibmethatswhoib
    Posted 4 years ago #

    ahh ok, thanks I'll look into that.

  6. Pothi Kalimuthu
    Posted 4 years ago #

    Huh, the plugin created the robot.txt file so it's pretty related.

    Please deactivate this plugin, create your own robots.txt file, try to access it from another browser. Please let us know what you see. If you still see the old robots.txt file, then there is no relation between robots.txt and this plugin.

    Also, it may help us, if you link your website's URL.

  7. Frederick Townes
    Plugin Author

    Posted 4 years ago #

    Are you sure you don't have another plugin that dynamically creates robots.txt?

Topic Closed

This topic has been closed to new replies.

About this Plugin

  • W3 Total Cache
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic