Title: Indexing issues
Last modified: March 22, 2024

---

# Indexing issues

 *  Resolved [inceptiontech](https://wordpress.org/support/users/inceptiontech/)
 * (@inceptiontech)
 * [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/)
 * Hello,
 * It seems that initial indexing works. However, when trying to URL inspect afterwards,
   it tells us “Failed: Blocked due to access forbidden (403)” but the crawl is 
   allowed.
 * We use Yoast SEO (free) and iThemes Security
   -I’ve tried disabling iThemes and
   the issue still remained.Google bot is allowed in robots.txtSitemap added to 
   robots.txtNothing on our firewall suggests that it blocks Google Bots – Otherwise
   the original indexing wouldn’t have worked at the first time around.
 * Thoughts?
 * The page I need help with: _[[log in](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fwordpress.org%2Fsupport%2Ftopic%2Findexing-issues-11%2F%3Foutput_format%3Dmd&locale=en_US)
   to see the link]_

Viewing 4 replies - 1 through 4 (of 4 total)

 *  [Mushrit Shabnam](https://wordpress.org/support/users/611shabnam/)
 * (@611shabnam)
 * [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/#post-17525834)
 * Hi [@inceptiontech](https://wordpress.org/support/users/inceptiontech/),
 * Can you kindly remove the _User-agent:_ directives for Googlebot and other search
   engine bots that you have added manually in your [robots.txt file](https://www.dynamicspine.com.au/robots.txt)?
 * To correct this, kindly [follow the suggested directives](https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/).
 *  Thread Starter [inceptiontech](https://wordpress.org/support/users/inceptiontech/)
 * (@inceptiontech)
 * [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/#post-17530098)
 * Thank you for the reply,
   I’ve removed those directives from the robots.txt
 * Let me know if you come across anything
 *  Plugin Support [Maybellyne](https://wordpress.org/support/users/maybellyne/)
 * (@maybellyne)
 * [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/#post-17533785)
 * Please navigate to _WordPress > Yoast SEO > Tools > File Editor > Robots.txt_
   and ensure you have these directives:
 * `# START YOAST BLOCK
   # --------------------------- User-agent: * Disallow: Sitemap:
   https://www.dynamicspine.com.au/sitemap_index.xml # ---------------------------#
   END YOAST BLOCK
    -  This reply was modified 2 years, 3 months ago by [Maybellyne](https://wordpress.org/support/users/maybellyne/).
 *  [Mushrit Shabnam](https://wordpress.org/support/users/611shabnam/)
 * (@611shabnam)
 * [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/#post-17541467)
 * This thread was marked resolved due to a lack of activity, but you’re always 
   welcome to re-open the topic. Please read [this post](https://wordpress.org/support/topic/read-this-before-your-post/)
   before [opening a new request](https://wordpress.org/support/plugin/wordpress-seo#new-post).

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Indexing issues’ is closed to new replies.

 * ![](https://ps.w.org/wordpress-seo/assets/icon-256x256.gif?rev=3419908)
 * [Yoast SEO - Advanced SEO with real-time guidance and built-in AI](https://wordpress.org/plugins/wordpress-seo/)
 * [Frequently Asked Questions](https://wordpress.org/plugins/wordpress-seo/#faq)
 * [Support Threads](https://wordpress.org/support/plugin/wordpress-seo/)
 * [Active Topics](https://wordpress.org/support/plugin/wordpress-seo/active/)
 * [Unresolved Topics](https://wordpress.org/support/plugin/wordpress-seo/unresolved/)
 * [Reviews](https://wordpress.org/support/plugin/wordpress-seo/reviews/)

## Tags

 * [indexing](https://wordpress.org/support/topic-tag/indexing/)

 * 4 replies
 * 3 participants
 * Last reply from: [Mushrit Shabnam](https://wordpress.org/support/users/611shabnam/)
 * Last activity: [2 years, 3 months ago](https://wordpress.org/support/topic/indexing-issues-11/#post-17541467)
 * Status: resolved