Google includes Rss and Atom Feeds for Crawling and Indexing Webpages

Google has included Rss and atom feeds in their Indexing inventory. That means, now Googlebot will crawl the rss or atom feed in your site to discover new url’s.

According to the Official announcement of Google :

Using feeds for discovery allows us to get these new pages into our index more quickly than traditional crawling methods. We may use many potential sources to access updates from feeds including Reader, notification services, or direct crawls of feeds

The biggest advantage of Rss and atom feed crawling would be fighting the post duplication issue. There are a lot of lazy people out there who just copy and republish content using your feeds. With this new feature, Google will crawl your feed sooner and any post duplication will add to your benefit as the Googlebot can identify the original source.

Let Google crawl your Blog’s feed

To allow the Googlebot to crawl your feed check your Robots.txt file. Generally the robots.txt is placed in the root directory of your blog and it should not block the Google-bot to index and crawl your feed.

e.g If your robots.txt file contains any of the following lines then it is blocking your Rss feed to be crawled :

User-agent: *
Disallow: /feed

User-agent: Googlebot
Disallow: /feed

You can check the status of your robots.txt file from Google webmaster central reports or by entering http://yourdomain.com/robots.txt in the browser address bar.

Till now, Google generally used incoming links, sitemaps to index and crawl websites. With the addition of Rss and atom feeds newer websites won’t find any problem getting their blog indexed.

Reading: Google includes Rss and Atom Feeds for Crawling and Indexing Webpages