Support » Fixing WordPress » restrict database tables to robots

  • Is it possible to direct robots to only crawl certain WordPress sql database tables, for example, only wp9posts, wp9comments? And if so, would it be a line inserted in the robots.txt file, and what would that code be? And is it possible to only allow certain robots, for example Googlebot to crawl my website, and what would that code be?

Viewing 7 replies - 1 through 7 (of 7 total)
  • Robots don’t crawl the database tables. They can only crawl the pages on your site.

    You can restrict the robots from various parts of your site. There’s a lot of info on so do some reasearch and I’m sure you’ll find what you’re after.

    Thanks Michael, but aren’t wordpress pages stored in the database in the “Post” table? So you can’t edit WordPress pages, like ASP pages, with Dreamweaver, but have to use the WordPress Dashpanel Pages editor while online, to create and edit WordPress pages and content, which are called “Posts”?

    Yes they are stored like that, but that doesn’t have anything to do with robots crawling the site. The database only holds the content and details of the pages. There’s no direct access to the database from anything outsdie of your wordpress site. Robots only crawl the URL’s, not page data, not category data, not anything else that’s stored in your database. All that they see is the pages that are publicly dipslayed on your website.

    Hmmm … Ok, so if I had some tables that were full of spam and malware and bad URLs, but they were not displayed on the WordPress website to the public, then the robots could not see them. After watching my website drop from 900 visitors a day to 200 per day as of March 22, 2012, I have been concentrating on cleaning out my database, which was half full of spam I discovered (I did not partake in any backlink schemes either). Perhaps I should be concentrating my efforts elswhere, like Google Webmaster Guidelines? Or perhaps Google just chose to put other websites ahead of mine with their new algorythms and I’m wasting my time with organic search engine marketing? All I got from Google Webmaster Tools was notifications of a severe drop in traffic to my website, with no explanation as to WHY.

    Unless the content is on your pages, then no, it won’t be indexed at all. What sort of spam was/is being collected in your database?

    After the latest changes by Google there’s a lot of sites that went down like that. Lost of those were ones that didn’t offer much decent human-focused content and didn’t have respecable backlinks. I haven’t looked at your site, but if that sounds like something that could be an issue I’d look at that as more of a cause.

    If you are not adhering to the webmasters guidelines that Google has, that stands out a sa very big reason that your site will get dropped because there’s a lot of sites that are doing things the right way.

    If you do real organis search engine marketing, it will always help to a degree, but the latest changes are made to put an end to people buying rankings. Of course they won’t ever be able to do that completely, but it’s a start. For a site to do well now it needs to grow properly with content, baclklinks and just general user interest.

    You won’t ever get a reason from Google as to why your traffic has jumped/dropped. If they tell anyone what to do, they’ll be revealing the secrets of their search ranking algorythms, and they are not about to do that.

    Thanks so much for your help Michael. I would really appreciate it if you could take a look at my website.

    1. Most of the spam was for womens clothing sales, which has nothing to do with insurance claims advice. My database was around 30 meg and dropped to 15 meg after deleting all the spam.

    2. 99 percent of the website content is insurance claim questions and answers by experts. Im not sure how you define “human focused content”. I have been in insurance claims for 32 years and have learned much. There are also many insurance professionals who use RSS feeds to monitor the website.

    3. re Webmaster Guidelines, I dont have a sitemap because an SEO guy a few years back advised against it. Dont rememner why. And one expert recently said I had too many embedded links on my “Claims Resources” page (right sidebar), which totally baffeled me since they were links to legitimate claims resources to the benefit of users. It actually hurt my website by referring users away from my site. So I have removed ALL embeded links from the website.

    I would again appreciate it if you could take a look at and give me your opinion as to why the big traffic drop.


Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘restrict database tables to robots’ is closed to new replies.