My WordPress Ping List
You may have noticed there is an Update Services under Settings -> Writing Setting in your WordPress. When you read into the description, you will find it very interesting:
What is a robots.txt? Here is the answer from Google:
A robots.txt file provides restrictions to search engine robots (known as “bots”) that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.
Do we need a robots.txt for your WordPress blog? Yes, we do. Because every WordPress blog has content that is not necessary for search engine to index, for example, the wp-admin folder which includes admin files. Another reason for creating robots.txt is to prevent the search engine from reaching the same content from more than one location, you know, we can find the same post in the categary folders, monthly archives, or particular tag. Google never likes content duplication. A great robots.txt file will increase efficiency of google clawler, that’s a good job done for SEO.
Now the questiion is how to write a proper robots.txt. What should be include in the robots.txt file? Let’s make a list. Some obvious folders and url keyword should be write down at first, here is the first part of my robots.txt file:
The first line “User-agent: *” means allowing all robots(Googlebot, Adsbot-google, etc) to index my blog. I disallowed all folders excluding the /wp-content/uploads folder, because this folder contains files uploaded for post content.