Jump to content
thirty bees forum
  • 0

Default robots.txt issues



I have a few problems with the default robots.txt and I would appreciate some help if anybody knows how to fix these issues.

  1. I recently remade my robots.txt within TB and since then it is indexing every image as a single page, so 1,000+ blank (image only) indexed pages hit my GWT. I cannot find these pages within google but they hit google webmaster tools and crawling programs. Isn't this weird? Any ideas? They hit GWT literally overnight.

  2. The default robots.txt blocks search parameters but this block also blocks pages that have 'search' in the name for example. So, if I had a book that says 'A search for freedom', for example, it would not be indexed. I removed a line, and pages index fine, but would you guys like me to provide data about this?

  3. It blocks module access by default, which seems to block footer image indexation on warehouse theme. (I had to remove this line) This one doesn't need any fix I suppose, but I am wondering why it blocks /module by default?

Link to comment
Share on other sites

1 answer to this question

Recommended Posts

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...