robots.txt

[Pedalman]

I just generated the robots.txt via Thirtybees backoffice and noticed that my old robots.txt had these lines at the top that are missing in the new one:

User-agent: *
Disallow:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow: 
# Allow Directives

The new one looks like this:

User-agent: *
# Allow Directives

 

Should I bother? Do they mean the same or does the new one generated via Thirtybees block the google image bot?

[Traumflug]

This third 'Disallow:' in the old version looks like broken code, as there's nothing noted on what to disallow.

The other two directives disallow Google crawlers. Usually not what merchants want.

[Pedalman]

Thanks, I will have a look at Google Webmaster. Two years ago a SEO expert told me to add these lines ?! 😞