WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] Please help robots.txt problems (13 posts)

  1. flisshayes
    Member
    Posted 1 year ago #

    Please help, I have no clue whats happened but rankings have dropped and out of all 1200 supposed indexed pages 934 are blocked by robots, help me please

    elissa :)

  2. tomontoast
    Member
    Posted 1 year ago #

    Could you post a link to your site? How did you work out which pages were blocked to robots?

  3. flisshayes
    Member
    Posted 1 year ago #

    hiya, google webmaster tools, says blocked by robots on 900+ pages, http://www.finalduties.co.uk thanks :)

  4. flisshayes
    Member
    Posted 1 year ago #

    Warnings

    Url blocked by robots.txt.

    Sitemap contains urls which are blocked by robots.txt.

    135 http://www.finalduties.co.uk/probate/tag/average-cost-of-probate/ what are these tag pages~?never seen these before

  5. tomontoast
    Member
    Posted 1 year ago #

    Have a look at http://www.finalduties.co.uk/robots.txt it seems to be fine to me but you might disagree with the rules.

    Tag pages are default wordpress pages which collate posts with the same tags, these should be blocked by robots.txt

  6. petermeadit
    Member
    Posted 1 year ago #

    Hi guys, should be pretty easy to fix. Just figure out what you want to Disallow and then remove the rest of the entries.

    I see you have some pdf files being blocked but I guess you might not want that?

    Hope that helps.

  7. flisshayes
    Member
    Posted 1 year ago #

    I just cant understand why i had 73 page one rankings and the word probate was ranking page 2, It says 976 pages are bloked by robots?

    I am so lost guys, hw do i block the tag pages they were always blocked seem something so strange has happened, the onlt thing i changed was to remove seo all in one and add yoast, which i have now taken off because as soon as i did that the page rankings went down.

    Total indexed
    Not selected
    Blocked by robots
    1,038

    1,399

    827 this is what i dont get, and then to get

    PLEASE HELP,

    Date Total indexed Ever crawled Not selected Blocked by robots Removed
    12/25/11 1010 2030 173 0 0
    01/01/2012 981 2031 173 0 0
    01/08/2012 1010 2031 173 0 0
    1/15/12 1025 2034 173 0 0
    1/22/12 1056 2036 173 0 0
    1/29/12 1043 2036 173 0 0
    02/05/2012 1055 2037 173 0 0
    02/12/2012 1053 2037 173 0 0
    2/19/12 1035 2039 173 0 0
    2/26/12 1030 2040 173 0 0
    03/04/2012 1102 2044 173 0 0
    03/11/2012 1055 2045 171 0 0
    3/18/12 1053 2045 172 0 0
    3/25/12 1068 2049 172 0 0
    04/01/2012 1059 2049 172 940 0
    04/08/2012 1058 2050 171 943 0
    4/15/12 1079 2051 171 945 0
    4/22/12 1045 2064 179 951 0
    4/29/12 1045 2070 259 938 0
    05/06/2012 1081 2079 270 918 0
    5/13/12 1088 2088 277 896 0
    5/20/12 1084 2090 270 894 0
    5/27/12 1152 2094 271 877 0
    06/03/2012 1192 2096 274 888 0
    06/10/2012 1175 2096 272 862 0
    6/17/12 1181 2097 308 857 0
    6/24/12 1186 2098 309 793 0
    07/01/2012 1178 2102 307 721 0
    07/08/2012 1210 2106 304 627 0
    7/15/12 1216 2110 303 556 0
    7/22/12 1223 2112 304 466 0
    7/29/12 1266 2113 305 499 0
    08/05/2012 1288 2116 309 525 0
    08/12/2012 1304 2119 309 599 0
    8/19/12 1322 2124 310 632 0
    8/26/12 1322 2134 312 687 0
    09/02/2012 1408 2135 315 706 0
    09/09/2012 1434 2139 305 766 0
    9/16/12 1344 2144 680 1108 0
    9/23/12 1344 3095 1051 862 0
    9/30/12 1344 3132 1137 904 0
    10/07/2012 1207 3168 1167 964 0
    10/14/12 1179 3187 1182 979 0
    10/21/12 1179 3199 1205 1010 0
    10/28/12 1128 3214 1216 1017 0
    11/04/2012 1128 3219 1223 1006 0
    11/11/2012 1138 3222 1231 975 0
    11/18/12 1138 3237 1242 969 0
    11/25/12 1030 3253 1248 910 0
    12/02/2012 1038 3500 1403 892 0
    12/09/2012 1038 3500 1407 835 0
    12/16/12 1038 3500 1399 827 0

  8. flisshayes
    Member
    Posted 1 year ago #

    sorry that didnt come out right 3 weeks ago i had non blocked by robots and now 900+pages?

  9. tomontoast
    Member
    Posted 1 year ago #

    You installed an SEO or sitemap plugin probably?

  10. flisshayes
    Member
    Posted 1 year ago #

    yeah xml sitemap plugin ? someone in the office updated the site map in the root file on friday, what can this do?

    Can i turn the robots.txt off? just too see what happens and how would i do that, ? thanks Tom I am stressing so much

    Elissa

  11. flisshayes
    Member
    Posted 1 year ago #

    well that was a month ago whats the best way to have the site map? i have plugin http://www.arnebrachhold.de/projects/wordpress-plugins/google-xml-sitemaps-generator/

  12. tomontoast
    Member
    Posted 1 year ago #

    I wouldn't stress about this too much, your robots.txt file is ok. I would read http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

  13. flisshayes
    Member
    Posted 1 year ago #

    just seen this aswell :( what does this mean

    Server error
    1
    errors

    Not followed
    6
    errors

Topic Closed

This topic has been closed to new replies.

About this Topic