• Hi Guys! Need your help.
    I want my robots.txt file to look like this:

    1. Robots.txt
    I fix it using Yoast plugin as:

    User-agent: *
    Disallow: /comment-page-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /license.txt
    Disallow: /readme.html
    Disallow: /xmlrpc.php
    Disallow: /*.php$
    Disallow: /?s=*

    Sitemap: http:// hannasles dot com/sitemap_index.xml

    But within a certain period of time it changes automatically to:

    User-agent: *
    Crawl-delay: 1
    Disallow: /wp-content/plugins/
    Disallow: /wp-admin/

    I have found out that WordPress can generates a virtual robots.txt file if the site root does not contain a physical file. But I checked and I have got the robots.txt in the root as a physical file. And I am using Yoast plugin.

    Can you give me any suggestions?

    thanks!
    Hanna

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)

The topic ‘wordpress generates its own virtual robots.txt file’ is closed to new replies.