Title: Please help robots.txt problems Last modified: August 20, 2016 --- # Please help robots.txt problems * Resolved [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 5 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/) * Please help, I have no clue whats happened but rankings have dropped and out of all 1200 supposed indexed pages 934 are blocked by robots, help me please * elissa 🙂 Viewing 12 replies - 1 through 12 (of 12 total) * [tomontoast](https://wordpress.org/support/users/tomontoast/) * (@tomontoast) * [13 years, 5 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283214) * Could you post a link to your site? How did you work out which pages were blocked to robots? * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283220) * hiya, google webmaster tools, says blocked by robots on 900+ pages, [http://www.finalduties.co.uk](http://www.finalduties.co.uk) thanks 🙂 * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283227) * Warnings * Url blocked by robots.txt. * Sitemap contains urls which are blocked by robots.txt. * 135 [http://www.finalduties.co.uk/probate/tag/average-cost-of-probate/](http://www.finalduties.co.uk/probate/tag/average-cost-of-probate/) what are these tag pages~?never seen these before * [tomontoast](https://wordpress.org/support/users/tomontoast/) * (@tomontoast) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283232) * Have a look at [http://www.finalduties.co.uk/robots.txt](http://www.finalduties.co.uk/robots.txt) it seems to be fine to me but you might disagree with the rules. * Tag pages are default wordpress pages which collate posts with the same tags, these should be blocked by robots.txt * [petermeadseo](https://wordpress.org/support/users/petermeadit/) * (@petermeadit) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283242) * Hi guys, should be pretty easy to fix. Just figure out what you want to Disallow and then remove the rest of the entries. * I see you have some pdf files being blocked but I guess you might not want that? * Hope that helps. * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283246) * I just cant understand why i had 73 page one rankings and the word probate was ranking page 2, It says 976 pages are bloked by robots? * I am so lost guys, hw do i block the tag pages they were always blocked seem something so strange has happened, the onlt thing i changed was to remove seo all in one and add yoast, which i have now taken off because as soon as i did that the page rankings went down. * Total indexed Not selected Blocked by robots 1,038 * 1,399 * 827 this is what i dont get, and then to get * PLEASE HELP, * Date Total indexed Ever crawled Not selected Blocked by robots Removed 12/25/ 11 1010 2030 173 0 0 01/01/2012 981 2031 173 0 0 01/08/2012 1010 2031 173 0 0 1/15/12 1025 2034 173 0 0 1/22/12 1056 2036 173 0 0 1/29/12 1043 2036 173 0 0 02/05/2012 1055 2037 173 0 0 02/12/2012 1053 2037 173 0 0 2/19/12 1035 2039 173 0 0 2/26/12 1030 2040 173 0 0 03/04/2012 1102 2044 173 0 0 03/11/2012 1055 2045 171 0 0 3/18/12 1053 2045 172 0 0 3/25/12 1068 2049 172 0 0 04/01/2012 1059 2049 172 940 0 04/08/2012 1058 2050 171 943 0 4/15/12 1079 2051 171 945 0 4/22/12 1045 2064 179 951 0 4/29/12 1045 2070 259 938 0 05/06/2012 1081 2079 270 918 0 5/13/12 1088 2088 277 896 0 5/20/12 1084 2090 270 894 0 5/27/12 1152 2094 271 877 0 06/03/2012 1192 2096 274 888 0 06/10/2012 1175 2096 272 862 0 6/17/12 1181 2097 308 857 0 6/24/12 1186 2098 309 793 0 07/01/2012 1178 2102 307 721 0 07/ 08/2012 1210 2106 304 627 0 7/15/12 1216 2110 303 556 0 7/22/12 1223 2112 304 466 0 7/29/12 1266 2113 305 499 0 08/05/2012 1288 2116 309 525 0 08/12/2012 1304 2119 309 599 0 8/19/12 1322 2124 310 632 0 8/26/12 1322 2134 312 687 0 09/02/ 2012 1408 2135 315 706 0 09/09/2012 1434 2139 305 766 0 9/16/12 1344 2144 680 1108 0 9/23/12 1344 3095 1051 862 0 9/30/12 1344 3132 1137 904 0 10/07/2012 1207 3168 1167 964 0 10/14/12 1179 3187 1182 979 0 10/21/12 1179 3199 1205 1010 0 10/28/12 1128 3214 1216 1017 0 11/04/2012 1128 3219 1223 1006 0 11/11/2012 1138 3222 1231 975 0 11/18/12 1138 3237 1242 969 0 11/25/12 1030 3253 1248 910 0 12/ 02/2012 1038 3500 1403 892 0 12/09/2012 1038 3500 1407 835 0 12/16/12 1038 3500 1399 827 0 * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283247) * sorry that didnt come out right 3 weeks ago i had non blocked by robots and now 900+pages? * [tomontoast](https://wordpress.org/support/users/tomontoast/) * (@tomontoast) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283248) * You installed an SEO or sitemap plugin probably? * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283249) * yeah xml sitemap plugin ? someone in the office updated the site map in the root file on friday, what can this do? * Can i turn the robots.txt off? just too see what happens and how would i do that,? thanks Tom I am stressing so much * Elissa * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283250) * well that was a month ago whats the best way to have the site map? i have plugin [http://www.arnebrachhold.de/projects/wordpress-plugins/google-xml-sitemaps-generator/](http://www.arnebrachhold.de/projects/wordpress-plugins/google-xml-sitemaps-generator/) * [tomontoast](https://wordpress.org/support/users/tomontoast/) * (@tomontoast) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283251) * I wouldn’t stress about this too much, your robots.txt file is ok. I would read [http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress](http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress) * Thread Starter [flisshayes](https://wordpress.org/support/users/flisshayes/) * (@flisshayes) * [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283253) * just seen this aswell 🙁 what does this mean * Server error 1 errors * Not followed 6 errors Viewing 12 replies - 1 through 12 (of 12 total) The topic ‘Please help robots.txt problems’ is closed to new replies. ## Tags * [robots](https://wordpress.org/support/topic-tag/robots/) * In: [Fixing WordPress](https://wordpress.org/support/forum/how-to-and-troubleshooting/) * 12 replies * 3 participants * Last reply from: [flisshayes](https://wordpress.org/support/users/flisshayes/) * Last activity: [13 years, 4 months ago](https://wordpress.org/support/topic/please-help-robotstxt-problems/#post-3283253) * Status: resolved ## Topics ### Topics with no replies ### Non-support topics ### Resolved topics ### Unresolved topics ### All topics