Comments on: How to Write WordPress robots.txt for SEO http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/ Free Online Resource Wed, 08 Feb 2012 07:52:00 +0000 hourly 1 http://wordpress.org/?v= By: Aaron http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-4028 Aaron Tue, 23 Nov 2010 12:31:12 +0000 http://zacklive.com/?p=18#comment-4028 Hi, One thing I should point out (which has only come to my attention recently) is that robots.txt is a file where you explicitly <i>disallow</i> bots from searching pages. It is exclusive rather than inclusive, and as such does not require (nor support) 'Allow' commands! Hi,

One thing I should point out (which has only come to my attention recently) is that robots.txt is a file where you explicitly disallow bots from searching pages. It is exclusive rather than inclusive, and as such does not require (nor support) ‘Allow’ commands!

]]> By: John http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3569 John Wed, 22 Sep 2010 02:33:02 +0000 http://zacklive.com/?p=18#comment-3569 WP has a virtual robots.txt file located around 1700. Go to wp-includes then the functions.php. WP has a virtual robots.txt file located around 1700. Go to wp-includes then the functions.php.

]]> By: Gee http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3539 Gee Wed, 01 Sep 2010 10:41:23 +0000 http://zacklive.com/?p=18#comment-3539 Hi, I'm using thesis theme for my self-hosted wordpress blog. Where can I find the robot.txt file? Sorry, it might seem like a silly question but I'm figuring out the answer to this for some time now because my Google adsense says I'm blocking their robot. Thanks! Hi, I’m using thesis theme for my self-hosted wordpress blog. Where can I find the robot.txt file? Sorry, it might seem like a silly question but I’m figuring out the answer to this for some time now because my Google adsense says I’m blocking their robot. Thanks!

]]> By: Chote Ustad http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3534 Chote Ustad Wed, 25 Aug 2010 20:27:19 +0000 http://zacklive.com/?p=18#comment-3534 Here is an standard Robots.txt file. I think it is perfect for all blogs, User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /category Disallow: /tag Disallow: /author Disallow: /trackback Disallow: /*trackback Disallow: /*trackback* Disallow: /*/trackback Disallow: /*?* Disallow: /*.html/$ Disallow: /*feed* # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* If you install WordPress in a separate directory for ex: WordPress or blog or any name I am considering WordPress here if you have any other name replace it with WordPress User-agent: * Disallow: /cgi-bin Disallow: /wordpress/wp-admin Disallow: /wordpress/wp-includes Disallow: /wordpress/wp-content/plugins Disallow: /wordpress/wp-content/cache Disallow: /wordpress/wp-content/themes Disallow: /category Disallow: /tag Disallow: /author Disallow: /trackback Disallow: /*trackback Disallow: /*trackback* Disallow: /*/trackback Disallow: /*?* Disallow: /*.html/$ Disallow: /*feed* # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* This form is not allowing me to write the last string Sitemap: copy it from above post. Here is an standard Robots.txt file. I think it is perfect for all blogs,

User-agent: *

Disallow: /cgi-bin

Disallow: /wp-admin

Disallow: /wp-includes

Disallow: /wp-content/plugins

Disallow: /wp-content/cache

Disallow: /wp-content/themes

Disallow: /category

Disallow: /tag

Disallow: /author

Disallow: /trackback

Disallow: /*trackback

Disallow: /*trackback*

Disallow: /*/trackback

Disallow: /*?*

Disallow: /*.html/$

Disallow: /*feed*

# Google Image

User-agent: Googlebot-Image

Disallow:

Allow: /*

# Google AdSense

User-agent: Mediapartners-Google*

Disallow:

Allow: /*

If you install WordPress in a separate directory for ex: WordPress or blog or any name

I am considering WordPress here if you have any other name replace it with WordPress

User-agent: *

Disallow: /cgi-bin

Disallow: /wordpress/wp-admin

Disallow: /wordpress/wp-includes

Disallow: /wordpress/wp-content/plugins

Disallow: /wordpress/wp-content/cache

Disallow: /wordpress/wp-content/themes

Disallow: /category

Disallow: /tag

Disallow: /author

Disallow: /trackback

Disallow: /*trackback

Disallow: /*trackback*

Disallow: /*/trackback

Disallow: /*?*

Disallow: /*.html/$

Disallow: /*feed*

# Google Image

User-agent: Googlebot-Image

Disallow:

Allow: /*

# Google AdSense

User-agent: Mediapartners-Google*

Disallow:

Allow: /*

This form is not allowing me to write the last string

Sitemap:

copy it from above post.

]]> By: sudeep http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3516 sudeep Mon, 09 Aug 2010 09:41:19 +0000 http://zacklive.com/?p=18#comment-3516 Hello can you suggest me robot for my blog,please help me regarding this because i am confused about using which one or am i doing it correct.You can check my website and reply me. <a href="http://www.techdunes.com" rel="nofollow">http://www.techdunes.com</a> (presently I am using your robottxt example above.But please reply me is it okay for my blog. I would be grateful to you. Your comment is awaiting moderation. Hello can you suggest me robot for my blog,please help me regarding this because i am confused about using which one or am i doing it correct.You can check my website and reply me.
http://www.techdunes.com
(presently I am using your robottxt example above.But please reply me is it okay for my blog.

I would be grateful to you.

Your comment is awaiting moderation.

]]> By: ZackLive http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3492 ZackLive Sat, 31 Jul 2010 03:26:44 +0000 http://zacklive.com/?p=18#comment-3492 That's a good move. That's a good move.

]]> By: David the Web Designer http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-3464 David the Web Designer Mon, 19 Jul 2010 21:06:12 +0000 http://zacklive.com/?p=18#comment-3464 Thanks for posting, just what i needed for my site. I used to just disallow the whole wp-admin file. Thanks for posting, just what i needed for my site. I used to just disallow the whole wp-admin file.

]]> By: Vegas SEO http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-791 Vegas SEO Sat, 09 May 2009 09:25:32 +0000 http://zacklive.com/?p=18#comment-791 It take time for Google to get your website indexed. keep patience it will be working soon. It take time for Google to get your website indexed.

keep patience it will be working soon.

]]> By: Nacho Plaza http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-766 Nacho Plaza Thu, 23 Apr 2009 19:02:20 +0000 http://zacklive.com/?p=18#comment-766 That robots.txt has errors. Not all search engines robots allow wildcards and the "Allow" term. Google, Yahoo & Live.com accept them, but not others. Another advice: place "Allows" just after "User-agent", because if there is a "Disallow" before that matches the url, the spider will not index it. Check the Google recomendations for robots.txt syntax & Wikipedia information: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360 http://en.wikipedia.org/wiki/Robots_exclusion_standard Use a robots.txt syntax validator to check your file (this identifies "Allow" lines like errors, but they are correct for Google, Yahoo and Live, but not for * User-agent): http://tool.motoricerca.info/robots-checker.phtml You can check the robots.txt on my blog ( http://www.weterede.com/robots.txt ). I am new with Wordpress, so it is under construction, because I do not jnow the WordPress internal structure. That robots.txt has errors. Not all search engines robots allow wildcards and the “Allow” term. Google, Yahoo & Live.com accept them, but not others. Another advice: place “Allows” just after “User-agent”, because if there is a “Disallow” before that matches the url, the spider will not index it.

Check the Google recomendations for robots.txt syntax & Wikipedia information:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
http://en.wikipedia.org/wiki/Robots_exclusion_standard

Use a robots.txt syntax validator to check your file (this identifies “Allow” lines like errors, but they are correct for Google, Yahoo and Live, but not for * User-agent):

http://tool.motoricerca.info/robots-checker.phtml

You can check the robots.txt on my blog ( http://www.weterede.com/robots.txt ). I am new with WordPress, so it is under construction, because I do not jnow the WordPress internal structure.

]]> By: Hermann http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-730 Hermann Wed, 25 Mar 2009 10:39:05 +0000 http://zacklive.com/?p=18#comment-730 Why adsensebot=allow-whole-site doesn't poses ranking degradation? Why adsensebot=allow-whole-site doesn’t poses ranking degradation?

]]> By: SEO-Explained-By-Google http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-583 SEO-Explained-By-Google Mon, 02 Feb 2009 14:31:07 +0000 http://zacklive.com/?p=18#comment-583 [...] Use the robots.txt file. Tell the search engines where and what to index and not index. For example, you dont want them to index your download page, right? Google also provides a useful too for that as well: Google robots.txt Tool Tips: disallow your internal SERPs to be indexed, near duplicate pages and URLs created by a proxy service. How to write a robot.txt file for Wordpress [...] [...] Use the robots.txt file. Tell the search engines where and what to index and not index. For example, you dont want them to index your download page, right? Google also provides a useful too for that as well: Google robots.txt Tool Tips: disallow your internal SERPs to be indexed, near duplicate pages and URLs created by a proxy service. How to write a robot.txt file for WordPress [...]

]]> By: Zack http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-376 Zack Mon, 29 Dec 2008 16:24:14 +0000 http://zacklive.com/?p=18#comment-376 @Chuck Hi, Chuck, you need to modify the robots.txt file yourself, or else you got nothing inside it. You can try my one. @Chuck
Hi, Chuck, you need to modify the robots.txt file yourself, or else you got nothing inside it. You can try my one.

]]> By: Chuck http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-373 Chuck Sun, 28 Dec 2008 06:52:23 +0000 http://zacklive.com/?p=18#comment-373 After discovering this article about the robots file I went and checked mine. The only thing listed in my robots.txt file is the location for the sitemap, nothing else. I use the 'all in one seo' plugin on my site.. I have some options checked in the setting for 'nofollow'. Is that the same thing? Or is my robots file seriously messed up? Thank you After discovering this article about the robots file I went and checked mine. The only thing listed in my robots.txt file is the location for the sitemap, nothing else.

I use the ‘all in one seo’ plugin on my site.. I have some options checked in the setting for ‘nofollow’. Is that the same thing?

Or is my robots file seriously messed up?

Thank you

]]> By: Zack http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-290 Zack Sun, 30 Nov 2008 01:03:33 +0000 http://zacklive.com/?p=18#comment-290 I guess it may take some time for google to index your blog. I guess it may take some time for google to index your blog.

]]> By: Roy Nottage http://zacklive.com/how-to-write-wordpress-robotstxt-seo/18/#comment-287 Roy Nottage Sat, 29 Nov 2008 21:49:32 +0000 http://zacklive.com/?p=18#comment-287 Hi there, I was wondering if you could verify something for me? I have had my blog running for a few months now, but only a week or 2 ago realised my robots.txt is probably the reason my wordpress blog images don't appear in Google. I had a gander to see what I should change it to, but I don't know if they (who I got advice from) were right or not after reading your post. The link to what I have is here http://royzy.co.uk/robots.txt If it is right and does allow Google Images to search it, then it might be because I haven't given Google enough time yet to index them. If it is wrong, I'll change it as you have described in your post. Cheers, Roy Hi there, I was wondering if you could verify something for me? I have had my blog running for a few months now, but only a week or 2 ago realised my robots.txt is probably the reason my wordpress blog images don’t appear in Google. I had a gander to see what I should change it to, but I don’t know if they (who I got advice from) were right or not after reading your post.

The link to what I have is here http://royzy.co.uk/robots.txt

If it is right and does allow Google Images to search it, then it might be because I haven’t given Google enough time yet to index them. If it is wrong, I’ll change it as you have described in your post.

Cheers,

Roy

]]>