A straightforward method to detecting RSS and Atom feeds in websites with HtmlAgilityPack

Let's create a tool that will be able to query any website and detect if they provide feeds in one of the common syndication formats. By using HtmlAgilityPack we can apply screen scraping techniques and examine the meta tags to see what alternative formats are available.

Easily extracting links from a snippet of html with HtmlAgilityPack

The HtmlAgilityPack is a powerful library that makes screen scraping in asp.net a breeze. This is the second in a continuing series where I demonstrate a way for you to extract all the links from a snippet of html.

Introduction To The HtmlAgilityPack Library

If you haven't heard of the HtmlAgilityPack before then you could be in for a treat. It's an open source library on codeplex which greatly eases html screen scraping.

What is screen scraping?

Screen scraping is the term for downloading a web page and then parsing its html to pull out information from it. In the old days before RSS feeds this was the only way to get at a websites information. With the advent of syndicated content and web services it has become a lot easier to get information from other sites but there are still situations where you need to go back to screen scraping to get the job done.

One example of a reason that you would still want to use this technique in a modern day environment is a price comparison site. When the complete stock information is not made available you have to read the data out of the website. Other reasons include weather data monitoring, website change detection (such as the release of a new version of software) and getting meta information from a page.

Doesn't .net support this natively?

The .net library comes with built in classes to manipulate XML documents with ease however in most cases you wont get won't get very far trying to screen scrape with these libraries. This is because they require standards compliant mark-up which is not very common for web sites. The XML standard has a strict policy of failing at the first error so unless you give it perfectly formatted code you wont be able to use these classes.

The HtmlAgilityPack provides a set of classes that makes it easy for you to download html pages into memory and then query them using XPath syntax. It doesn't matter if the page isn't standards compliant, the library will just do the best with what it has.

Where can I download it?

The codeplex project is located at the following location:

• http://www.codeplex.com/htmlagilitypack

To download it just click the Download Now button located down the right hand side of the page and then press I Agree to accept the codeplex license agreement.

The pack comes with a couple of examples to get you started and there are most posts in this series here on this site.

More In This Series

This article is part of a series. You can find more posts in this series here:

• http://runtingsproper.blogspot.com/2009/09/htmlagilitypack-article-series.html

Further reading

• http://en.wikipedia.org/wiki/Web_scraping

  

HtmlAgilityPack Article Series

This is an index post which will pull together the various HtmlAgilityPack posts on here. If you don't know what the HtmlAgilityPack or the idea of Screen Scraping is then you should read the first introductory article, otherwise you can dive right in!

Article Index

• Introduction
• Extract all the links from a snippet of html
• Discover the syndication feeds in a page