Problem with robots.txt

  • Resolved krystian1

    (@krystian1)


    3 years, 9 months ago

    Hello there is some problem with my site. I’ve got Yoast SEO plugin and I don’t have ‘file editor’ where I can edit or create robots.txt file. Also I can not crawl my site in ahrefs, what might be the problem? (I already have robots.txt file on my server, but its not showing in Yoast SEO)

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • Thread Starter krystian1

    (@krystian1)

    3 years, 9 months ago

    Also I was contacting with my host, they can’t help me neither

    Plugin Support Jerlyn

    (@jerparx)

    3 years, 9 months ago

    Hi @krystian1 ,

    We are sorry to hear about your issue. If you do not see the ‘File Editor’ option in WordPress > SEO > Tools, this may mean that file editing is disabled on your WordPress installation. If file editing is disabled, please enable file editing or alternatively you can edit the file using FTP. When file editing is disabled this is due to the following directive being contained in your wp-config.php file:
    define( 'DISALLOW_FILE_EDIT', true );
    You can read more about enabling file editing here: https://wordpress.org/support/article/editing-wp-config-php/

    If you are not confident in checking/changing your file permissions via FTP, then it’s best to check with your hosting provider. You may provide them the article to them as a guide.

    Hope this helps!

    Thread Starter krystian1

    (@krystian1)

    3 years, 9 months ago

    It helps – now I can edit the robots.txt file, but still I can’t crawl this site in ahrefs. Why?

    Pcosta88

    (@pcosta88)

    3 years, 9 months ago

    Hi,

    We are not familiar with ahrefs or how it works. However, we looked at the robots.txt file here https://www.xn--dittinterir-pgb.no/robots.txt and you have two disallow rules which prevent crawlers from crawling the site.

    W suggest having the default rules, as if crawlers need to access resources in those files those disallow rules prevent them from doing so. This means they do not have all the information to understand and then index the page. This guide explains more: https://yoast.com/wordpress-robots-txt-example/ and this guide explains how to edit the robots.txt: https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/

    You can use the defaults below.

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Plugin Support devnihil

    (@devnihil)

    3 years, 8 months ago

    We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Problem with robots.txt’ is closed to new replies.

Tags