sitemaps valid but showing 404 in google

  • Resolved Miz.Michele

    (@mizmichele)


    10 years, 1 month ago

    I had a hard time getting your plugin to work because of the FD Feedburner feedsmith plugin which I removed.

    I’ve generated sitemap and news sitemap and they are valid.

    Google and Bing Webmaster tools show 404 and I can’t figure out why. I was using WordPress Seo with sitemaps. The sitemaps weren’t working so I still use the SEO portion but not their sitemap.

    I also use BPS Security.

    Any help on why Google and Bing can’t access the site maps?

    My robot.txt file looks like this

    # XML Sitemap & Google News Feeds version 4.3.2 - http://status301.net/wordpress-plugins/xml-sitemap-feed/
Sitemap: http://www.lasvegasworldnews.com/sitemap.xml
Sitemap: http://www.lasvegasworldnews.com/sitemap-news.xml
Sitemap: http://www.lasvegasworldnews.com/?feed=sitemap

User-agent: *
Disallow: /wp-admin/
Disallow: */xmlrpc.php
Disallow: */wp-*.php
Disallow: */trackback/
Disallow: *?wptheme=
Disallow: *?comments=
Disallow: *?replytocom
Disallow: */comment-page-
Disallow: *?s=
Disallow: */wp-content/plugins/
Allow: */wp-content/uploads/

    https://wordpress.org/plugins/xml-sitemap-feed/

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Author Rolf Allard van Hagen

    (@ravanh)

    10 years, 1 month ago

    Hi Michele, it looks like you are using W3 Total Cache and Cloudflare CDN… Can you configure W3TC to NOT cache your sitemap and can you also make Cloudflare NOT upload the sitemap to their CDN?

    I suspect the issue is related to these services, or at least it will make debugging easier because I’ve already seen a 502 (bad gateway) response and a 404 response.

    Plugin Author Rolf Allard van Hagen

    (@ravanh)

    10 years, 1 month ago

    Also, URLs like http://www.lasvegasworldnews.com/sitemap-news.xml give a 302 response redirecting to http://www.lasvegasworldnews.com/sitemap-news.xml/ (with a trailing slash)… I’m not sure why this would happen on your site. Might have something to do with the forementioned services, or with your Permalink structure.

    Can you tell me what is exactly in the permalinks field on Settings > Permalinks ?

    Thread Starter Miz.Michele

    (@mizmichele)

    10 years, 1 month ago

    Thanks for the help it’s driving me crazy.

    For W3 I had this information from when I was using the WP Seo sitemap. I have the option checked to not process 404 errors for static objects with WordPress and then you are able to add exceptions. My exception list is,

    robots\.txt
sitemap(_index)?\.xml(\.gz)?
[a-z0-9_\-]+-sitemap([0-9]+)?\.xml(\.gz)?

    does that look good to you?

    I also added in all of the sitemaps to never be cached.
    And then not sure if I should have done this or not but I removed this info `sitemap(_index)?\.xml(\.gz)?
    [a-z0-9_\-]+-sitemap([0-9]+)?\.xml(\.gz)?` from the no trailing slash pages box.

    Cloudflare. They allow you to make custom rules so I added http://www.lasvegasworldnews.com/sitemap* the * is supposed to have the rule follow anything with the same pattern and I set it to bypass cloudflare caching and turned anything I could off on their end for the rule.

    My permalink structure is http://www.lasvegasworldnews.com/%postname%/%post_id%/
    And I cleared all of my caches.
    What do you think? Thanks again for the help.

    Thread Starter Miz.Michele

    (@mizmichele)

    10 years, 1 month ago

    After making changes, I tried to submit sitemaps again. It tested in fine in google and as soon as I submitted there were errors and it would not even accept the news sitemap.

    What do you recommend?

    Thread Starter Miz.Michele

    (@mizmichele)

    10 years, 1 month ago

    Nothing word and my site is completely down. This is so disheartening

    Plugin Author Rolf Allard van Hagen

    (@ravanh)

    10 years, 1 month ago

    Try these rules for the exceptions in W3TC:

    robots\.txt
sitemap(.*)?\.xml

    Thread Starter Miz.Michele

    (@mizmichele)

    10 years, 1 month ago

    Thanks I did that. I removed your plugin because I’ve been pulling my hair out. I don’t know if it affects your plugin but here’s what I did. I think it was an issue with W3 Cache.

    I went with he BWG sitemap and I got it working by changing the format in the 404 exception list from

    sitemap(_index|[0-9]+)?\.xml(\.gz)?
    sitemap\.xml(\.gz)?

    to

    (sitemapindex|[a-z0-9_-]+)\.xml

    I don’t know if it would have worked with your plugin but I figured I’d let you know. Thanks a lot for taking the time to help me.

    Plugin Author Rolf Allard van Hagen

    (@ravanh)

    10 years, 1 month ago

    No problem 🙂

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘sitemaps valid but showing 404 in google’ is closed to new replies.