Hi mima12. If you wanted to prevent search engines from indexing your pages, you would add this line to your robots.txt:
User-agent: *
Disallow: /
While you don’t have that specific code, you do have:
User-agent: *
Disallow:
That last line I believe is what is preventing you from getting indexed in search engines. I recommend removing that.
Thread Starter
mima12
(@mima12)
Thank you. I think one of the plugins has created this robots.txt by it self… Anyway i created a new file that allows all by using webmasters tool and uploaded it to my server. Hope it works..
Hello
# Added by Link Alias Generator (LAG) module
User-agent: *
Disallow: /go/
# End LAG
the above script is added by seo ultimate in wordpress i believe..which helps in link masking and the one below
User-agent: *
Disallow:
Sitemap: http://www.xxxxxxx.co.il/sitemap.xml.gz
should help search engines to crawl site..
that is what i learnt searching varios blogs
now i see there is a progress in my crawl stats on webmaster tools but still on crawl errors page there is a message “URL restricted by robots.txt”
I need to use linking masking which is why the script from top to “End Lag” is needed, I believe and at the same time i want search engines crawl.. please help anyone…so confused
Thanks in advance.
Hi Musthafiz, from my experience, “link masking” in the traditional sense is not really used that much. You can use .htaccess files to perform a lot of the same things.
Can you give us a few more details about why you are using link masking and what exactly you are trying to do with it ( a few examples would be helpful )
Hi Brad, I have learned so far from my research that link masking are used instead of nofollow attribute, link masking actually prevents link juice to flow from urs site to another (for which you have put a link on your site)…now if you are reproducing an article that u came across another site u should put a link of that site on your content…in such case i never use these tags….for example: u have reproduced a content from X site, now on the X site they have put a link of Y site which u didnt follow at all…but want to put this link in your content for ur readers…u can use link masking on Y site…so that users if go to Y site from yours, ur link juice doesnt flow to Y…..Hope I have made it clear.
