Pedalman Posted June 6, 2020 Share Posted June 6, 2020 I just generated the robots.txt via Thirtybees backoffice and noticed that my old robots.txt had these lines at the top that are missing in the new one: User-agent: * Disallow: User-agent: Googlebot Disallow: User-agent: Googlebot-image Disallow: # Allow Directives The new one looks like this: User-agent: * # Allow Directives Should I bother? Do they mean the same or does the new one generated via Thirtybees block the google image bot? Link to comment Share on other sites More sharing options...
0 Traumflug Posted June 6, 2020 Share Posted June 6, 2020 This third 'Disallow:' in the old version looks like broken code, as there's nothing noted on what to disallow. The other two directives disallow Google crawlers. Usually not what merchants want. Link to comment Share on other sites More sharing options...
0 Pedalman Posted June 9, 2020 Author Share Posted June 9, 2020 Thanks, I will have a look at Google Webmaster. Two years ago a SEO expert told me to add these lines ?! 😞 Link to comment Share on other sites More sharing options...
Question
Pedalman
I just generated the robots.txt via Thirtybees backoffice and noticed that my old robots.txt had these lines at the top that are missing in the new one:
The new one looks like this:
Should I bother? Do they mean the same or does the new one generated via Thirtybees block the google image bot?
Link to comment
Share on other sites
2 answers to this question
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now