Support » Fixing WordPress » Robots.text by AIOSEO

  • jmfvecu

    (@jmfvecu)


    Can someone please tell me if this robots.text file is not blocking my site?

    The file is generated by AIOSEO and store in the database so I cannot edit it myself, I disabled the file in AIOSEO but it is still to be found if I type robots.text to my url. The google console tells me almost all my front end pages are blocked. I lookecke in my server and i discover a seconde file which I have deleted, i post it here too I think one of this two file are blocking my pages

    Robots.text

    User-agent: *
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-admin/
    
    Sitemap: https://www.mysite/sitemap.xml
    
    

    User-agent: *
    disallow:

    User-agent: *
    Disallow: /search
    Disallow: /category/
    Disallow: /tag/

    • This topic was modified 5 months ago by jmfvecu.
Viewing 10 replies - 1 through 10 (of 10 total)
  • threadi

    (@threadi)

    I suspect you posted 2 different robots.txt contents in your post. I’m just not sure where one starts and the other ends.

    Correct would be:

    User-agent: *
    Allow: /wp-admin/admin-ajax.php
    Disallow: /wp-admin/
    
    Sitemap: https://www.mysite/sitemap.xml

    The following excludes your page from indexing:

    User-agent: *
    disallow:
    
    User-agent: *
    Disallow: /search
    Disallow: /category/
    Disallow: /tag/
    Thread Starter jmfvecu

    (@jmfvecu)

    Thanks, I have deleted the robots.text file in the server and I am using the robots geberate by AIOSEO whicjh is stored in the databse, I have modified it and it looks like this now. Anyway google is bloking all my site, Post are not indexed and if I render them manually to google in the search console either, videos are hosted on my site but not indexed by google either, so I got a big problem to solve, I am on it but I need some help. Google says that they could not determine the prominet video on the page. They are actually videos that I publish as posts in my site but without text content.

    `User-agent: *

    Allow: /wp-admin/admin-ajax.php

    Allow: /allow: /wp-content/uploads/

    Disallow: /wp-admin/

    Disallow: /disallow: /wp-content/plugins/

    Disallow: /disallow: /readme.html

    Disallow: /disallow: /refer/

    Disallow: /disallow: /tag/ /

    Disallow: /disallow: /author/

    Disallow: /disallow: /404-error/

    Sitemap: https://www.mysite/sitemap.xml

    • This reply was modified 5 months ago by jmfvecu.
    threadi

    (@threadi)

    Show a link to the website in question, then it can certainly be evaluated here. Or you can simply ask directly at https://wordpress.org/support/plugin/all-in-one-seo-pack/ – there are certainly more people there who can assess this correctly.

    Thread Starter jmfvecu

    (@jmfvecu)

    Thanks, in the link is the url to my site, i post here some screen shots of the error message in Google search console.

    Mysite

    search console 1

    search console 2

    • This reply was modified 4 months, 4 weeks ago by jmfvecu.

    Your robots.txt is formatted a bit awkwardly:

    User-agent: *
    Allow: /wp-admin/admin-ajax.php
    Allow: /allow: /wp-content/uploads/
    Disallow: /wp-admin/
    Disallow: /disallow: /wp-content/plugins/
    Disallow: /disallow: /readme.html
    Disallow: /disallow: /refer/
    Disallow: /disallow: /tag/ /
    Disallow: /disallow: /author/
    Disallow: /disallow: /404-error/
    
    Sitemap: https://www.julioflores.site/sitemap.xml

    This would be the right way:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-admin/
    Disallow: /wp-content/plugins/
    Disallow: /readme.html
    Disallow: /refer/
    Disallow: /tag/ /
    Disallow: /author/
    Disallow: /404-error/
    Allow: /wp-admin/admin-ajax.php
    
    Sitemap: https://www.julioflores.site/sitemap.xml

    If you cannot change this yourself, then I would again recommend that you contact the support of the plugin you are using.

    The GSC forum might also be able to help, since the actual problem is a conspicuousness on Google: https://support.google.com/webmasters/community?hl=de

    Wordpress support here cannot help you in this regard.

    Thread Starter jmfvecu

    (@jmfvecu)

    I can change it myself but why you took off the next line in your example?, and why disallow wp admin is two times in the code?

    allow: /wp-content/uploads/

    • This reply was modified 4 months, 4 weeks ago by jmfvecu.
    • This reply was modified 4 months, 4 weeks ago by jmfvecu.

    I overlooked it, but it doesn’t matter and doesn’t create a syntactical error. The upload directory was already removed in your code. I don’t see any reason for that either. You can also omit the line if your media files are all allowed to be indexed (some exclude this in their project because of PDF files).

    Thread Starter jmfvecu

    (@jmfvecu)

    Thanks for your help, I have disabled the AIOSEO robots and i have uploaded mine to the root. I want to improve my robots.text cause google is not indexing my posts nor my videos which are hosted in my own site but posted as posts with no text content, the reason in google search console is that

    “Google could not determine the prominent video on the page”

    and

    “No thumbnail URL provided”.

    I am not sure how to solve this.

    MY ROBOTS LOOKS LIKE THIS NOW

    `User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /readme.html
    Disallow: /refer/
    Disallow: /tag/ /
    Disallow: /author/
    Disallow: /404-error/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://www.mysite/sitemap.xml

    • This reply was modified 4 months, 4 weeks ago by jmfvecu.
    • This reply was modified 4 months, 4 weeks ago by jmfvecu.
    • This reply was modified 4 months, 4 weeks ago by jmfvecu.
    • This reply was modified 4 months, 4 weeks ago by jmfvecu.

    This has nothing to do with the robots.txt but with the way videos are embedded in your page. In order not to blow up the topic here, I would recommend you (also this time) to ask a question about this in the AIOSEO support forum. They are more familiar with these topics than the general WordPress forum.

    Thread Starter jmfvecu

    (@jmfvecu)

    I know, I have published in the AIOSEO forum too but nobody answer so far.

    AIOSEO Forum

Viewing 10 replies - 1 through 10 (of 10 total)
  • You must be logged in to reply to this topic.