Introduction To The HtmlAgilityPack Library

Posted on: Sunday, 27 September 2009 13:29

If you haven't heard of the HtmlAgilityPack before then you could be in for a treat. It's an open source library on codeplex which greatly eases html screen scraping.

What is screen scraping?

Screen scraping is the term for downloading a web page and then parsing its html to pull out information from it. In the old days before RSS feeds this was the only way to get at a websites information. With the advent of syndicated content and web services it has become a lot easier to get information from other sites but there are still situations where you need to go back to screen scraping to get the job done.

One example of a reason that you would still want to use this technique in a modern day environment is a price comparison site. When the complete stock information is not made available you have to read the data out of the website. Other reasons include weather data monitoring, website change detection (such as the release of a new version of software) and getting meta information from a page.

Doesn't .net support this natively?

The .net library comes with built in classes to manipulate XML documents with ease however in most cases you wont get won't get very far trying to screen scrape with these libraries. This is because they require standards compliant mark-up which is not very common for web sites. The XML standard has a strict policy of failing at the first error so unless you give it perfectly formatted code you wont be able to use these classes.

The HtmlAgilityPack provides a set of classes that makes it easy for you to download html pages into memory and then query them using XPath syntax. It doesn't matter if the page isn't standards compliant, the library will just do the best with what it has.

Where can I download it?

The codeplex project is located at the following location:

To download it just click the Download Now button located down the right hand side of the page and then press I Agree to accept the codeplex license agreement.

The pack comes with a couple of examples to get you started and there are most posts in this series here on this site.

More In This Series

This article is part of a series. You can find more posts in this series here:

Further reading

kick it Shout it vote it on WebDevVote.com

HtmlAgilityPack Article Series

Posted on: 13:25

This is an index post which will pull together the various HtmlAgilityPack posts on here. If you don't know what the HtmlAgilityPack or the idea of Screen Scraping is then you should read the first introductory article, otherwise you can dive right in!

Article Index

Where to get an asp.net 2.0 compatible AjaxControlToolkit

Posted on: Thursday, 17 September 2009 23:01

(Updated: To include the latest controls asp.net 2.0 users are missing out on)

This is just a simple link so that I don't have to keep finding it and explaining the situation each time the topic comes up.

Asp.net 2.0 support

The AjaxControlToolkit doesn't support asp.net 2.0 any more. This means that if you are still using asp.net 2.0 then you will have to download an older release.

The last version that was released with 2.0 support was toolkit version 1.0.20229. Download it here:

Asp.net 3.5 support

If you're looking for the latest 3.5 version then I suggest heading directly to the front page of the site so that you can get the latest version. Actually this url seems to automatically redirect you to the latest version available:

But if it breaks for you in the future then use this link and click the download button in the top right:

What's missing for asp.net 2.0 users?

The following controls (at the time of writing) are missing from the older AjaxControlToolkit:

  • HTMLEditor
  • ComboBox
  • ColorPicker
  • MultiHandleSlider
  • Seadragon Image Viewer
  • AsyncFileUpload

Round off time to the nearest minute

Posted on: 07:59

Say you have a DateTime object like this:

DateTime someTime = DateTime.Parse("00:00:38");

Rounding Up

How would you round this up to the nearest minute? There isn't a built in function to do this so you have to use a little bit of maths to get there. There are 60 seconds in a minute. We already have 38 seconds on the clock. So we need to add on 60 - 38 = 22 more seconds.

In code this looks like:

DateTime RoundUp = DateTime.Parse("00:00:38");
RoundUp = RoundUp.AddSeconds(60 - RoundUp.Second);

Now our RoundUp contains "00:01:00".

Rounding Down

To round down we use the same idea:

DateTime RoundDown = DateTime.Parse("00:01:38");
RoundDown = RoundDown.AddSeconds(-RoundDown.Second);

Note

The AddSeconds() method doesn't actually alter the DateTime its working on - it just returns a new one. This is why I assigned the DateTime to itself in the examples above.

Farewell VSJ print edition

Posted on: Tuesday, 15 September 2009 19:27

In the first page of VSJ there is a small message from the editor which discreetly announces that this month will be the final month of VSJ in print. I will miss reading this publication which has become a part of my life over the past year.

Being a massive geek it was a nice fix when I found myself being forced to leave my computer for a few minutes such as when I have to wait for somebody.

The magazine is still going to continue as an online publication but if I am honest I wont be reading it. There is already enough content to read when I am at my computer and I dont have time to get through all of it!

So thats that then. Does anybody know of any other great .net focused print magazines?

A better way to reference your wizard steps using named steps

Posted on: Saturday, 12 September 2009 10:52

Note: this article uses the plain vanilla <asp:Wizard> but the concepts apply equally well to its popular counterpart <asp:CreateUserWizard>.

By far the most common way that I see wizard steps reference in code snippets is by their index.

Something like this is common:

void Wizard1_NextButtonClick(object sender, System.Web.UI.WebControls.WizardNavigationEventArgs e)
{
     if (Wizard1.ActiveStepIndex == 1)
     {
          // jump to step three
          Wizard1.ActiveStepIndex = 3;
     }
}

This was actually taken from some MSDN documentation. But what's the problem with this? It works doesn't it?

Yes it works today but the problem is when you come back in a few weeks and want to re-order your steps, or add a new one in somewhere. Suddenly all those numbers seem to have a lot less meaning. Its a chore to change the steps over and its not something that will get caught by the compiler of you miss one or get it wrong.

There is a better way though because you don't have to tie yourself down to the index. The trick in a nutshell is to give each of your <asp:WizardStep>'s an ID. By default the Visual Studio GUI doesn't do this but you can add them in yourself. This gives you a meaningful name that you can use and its not tied to the order of the steps.

Example 1 - Finding a control in a WizardStep

The first challenge that is enhanced by this technique is finding controls that are inside wizard steps. Instead of having to go through the Wizard control you can now use FindControl() directly on your WizardStep because you have a way to access it.

TextBox firstName = (TextBox)StepNameHere.FindControl("FirstName");

Example 2 - Comparing against CurrentStepIndex

The second example is my favourite trick because I didn't find out about it for a while after I had started using named WizardSteps.

If you have a WizardStep called StepBillingDetails for example, you can use the Wizard1.WizardSteps.IndexOf() to find the index of that named step. This turns your named step into an integer which can be used to compare against properties such as ActiveStepIndex property of the Wizard and CurrentStepIndex in your NextButtonClick event.

protected void Wizard1_NextButtonClick(object sender, WizardNavigationEventArgs e)
{
    Wizard w = (Wizard)sender;

    // check if billing details step has been completed
    if (e.CurrentStepIndex == w.WizardSteps.IndexOf(StepBillingDetails))
    {
        // user has just completed the wizard step "StepBillingDetails"
    }
}

Example 3 - Predictable Hot Jumping

From looking back at the first example that I gleaned from MSDN you can probably see how we can improve it using these new techniques. While it is a slightly contrived example (I doubt I would always want to unconditionally jump from step one to step three) it does illustrate my point.

Lets take a look at what it looks like now that it uses named steps:

void Wizard1_NextButtonClick(object sender, System.Web.UI.WebControls.WizardNavigationEventArgs e)
{
     Wizard wizard1 = (Wizard)sender;

     if (wizard1.ActiveStepIndex == wizard1.WizardSteps.IndexOf(StepPersonalDetails))
     {
          // jump to step three
          wizard1.ActiveStepIndex = wizard1.WizardSteps.IndexOf(StepOrderComplete);
     }
}

Complete Sample

Here is a complete sample demonstrating how to take advantage of named WizardSteps with both of the techniques described above.

Markup:

        <asp:Wizard ID="Wizard1" runat="server" 
            onnextbuttonclick="Wizard1_NextButtonClick">
            <WizardSteps>
                <asp:WizardStep ID="StepPersonalDetails" runat="server" Title="Personal Details">
                    <asp:TextBox ID="FirstName" runat="server"></asp:TextBox>
                </asp:WizardStep>
                <asp:WizardStep ID="StepBillingDetails" runat="server" Title="Billing Details">
                </asp:WizardStep>
                <asp:WizardStep ID="StepOrderComplete" runat="server" Title="Order Complete">
                    First Name: <asp:Label ID="FirstNameLabel" runat="server" Text="Unknown"></asp:Label>
                </asp:WizardStep>
            </WizardSteps>
        </asp:Wizard>

Code behind:

    protected void Wizard1_NextButtonClick(object sender, WizardNavigationEventArgs e)
    {
        Wizard w = (Wizard)sender;

        // check if we just completed the first step
        if (e.CurrentStepIndex == w.WizardSteps.IndexOf(StepPersonalDetails))
        {
            // find the first name control
            TextBox firstName = (TextBox)StepPersonalDetails.FindControl("FirstName");

            // check it has a value
            if (!String.IsNullOrEmpty(firstName.Text))
            {
                // find the label on the last page
                Label firstNameLabel = (Label)StepOrderComplete.FindControl("FirstNameLabel");

                // assign the value
                firstNameLabel.Text = firstName.Text;
            }

            // jump to order complete step
            w.ActiveStepIndex = w.WizardSteps.IndexOf(StepOrderComplete);
        }
    }

Downloadable version:

Further Reading

Not a whole lot of further reading for you to today as I have really said everything I wanted to say. Here is one just for completeness:

You will find a few gotcha's on there that aren't related to this article but things that might trip you up in the future.

Using aspnet_regsql via the command line to setup membership roles and profiles only

Posted on: Monday, 24 August 2009 19:29

When you setup the asp.net membership, profile, etc using the aspnet_regsql wizard its pretty much an all or nothing affair - it adds all of the possible features in one go with no option to select which features are added.

When setting up the login systems you only need to add the following feature support: Membership, Role Manager and Profiles. The two remaining features (Personalization and SQL Web event provider) are not needed.

So to only install the features you require you should use the command line version of the tool.

Note: The version listed below (v2.0.50727) is valid for both .net 2.0 and .net 3.5. Without going into the technical details .net 3.5 is a layer on top of 2.0 and the same tool is used for both versions.

If you are working with a version that is newer than 3.5 then check that versions folder for an updated version of this tool as it may be updated in future versions.

Getting the command window open (Windows Vista)

  1. Click the start button then copy and paste this into the search box C:\Windows\Microsoft.NET\Framework\
  2. While holding the shift key down, right click on the folder called "v2.0.50727".
  3. You will see the normal context menu but with an extra option called "Command Window Here"

Getting the command window open (Older versions of Windows)

  1. Click the start button, select Run.
  2. Type "cmd" without the speech marks and press enter
  3. Copy this line of text: cd C:\Windows\Microsoft.NET\Framework\v2.0.50727\
  4. Right click in the command prompt window and choose paste.
  5. Press enter

Setting up the database

Now that you are in the correct location you should type the text below into the command line.

aspnet_regsql.exe -E -S .\SQLEXPRESS -A mrp -d databasename

The -E, -S, -A and -d are all case-sensitive so make sure you type the line exactly!

Replace SQLEXPRESS with the instance name of your database server - if you're using the free version of Sql Server then you will almost certainly be called .\SQLEXPRESS. The .\ bit at the start means "this computer" so if your sql server is on a different server then put the computer name there such as COMPUTER\SQLEXPRESS.

Replace databasename with the name of the database you want to add these features to.

What do the command line parameters mean?

You can get a full explanation of all the options available by typing:

aspnet_regsql -?

Here is a run down of the parameters we used:

-E Auth with windows credentials -S server instance -A services to add (membership, roles, profiles) -d database name

REMEMBER - Parameters are case sensitive

Did it work?

Press enter and you should see this:

Start adding the following features:
Membership
Profile
RoleManager

.......

Finished.

C:\Windows\Microsoft.NET\Framework\v2.0.50727>

A final tip

One thing that I did not realise the first time I used these features is that you can use an existing database when you set these tables up. In my first site that I used these features I ended up with two databases like ClientDatabase and ClientDatabaseAspNetDB because I thought that it would overwrite my data! This is NOT the case! You can specify the same database you are using for all your other development.

Further Reading

kick it Shout it vote it on WebDevVote.com

Setting a default button so your users can press enter to submit your form

Posted on: Thursday, 20 August 2009 20:03

You might have noticed that when you press enter to submit a form in your asp.net page it just refreshes the page instead of submitting as you would expect.

The reason behind this is that asp.net web forms uses the <form> html tag to power the whole postback mechanism. This means that the default behaviour of submitting the form is still occurring but as far as asp.net is concerned you have just submitted its postback form with no commands so it just refreshes.

DefaultButton property

Anyway enough of the back story. The way around this is to set the DefaultButton property. The DefaultButton property was added to the HtmlForm and Panel classes in asp.net 2.0.

You can set this programmatically:

Panel1.DefaultButton = Button1.UniqueID;

or via the code markup:

<asp:Panel ID="Panel1" runat="server" DefaultButton="Button1">
</asp:Panel>

Per panel

The best use of the DefaultButton property is at the Panel level. The asp.net webforms postback model defines a single html form that all elements of the page are contained in.

If you have for example a login button and a search box in your masterpage, a feedback form in the content area then you have several logical forms contained behind the scenes in a single html form tag. Your users will expect to press enter in the search box and trigger a search. They dont want to see your feedback form validators kicking up a fuss.

By putting each logical form inside an you can set the default button to improve the user experience.

At the form level

As I have already hinted you can also do this at the page level by setting the DefaultButton on the behind-the-scenes page level form tag. As you will already see this is not a good idea for serious projects but you may want to set it in certain circumstances.

If you are in a content page of a masterpage then you will need to do it in the code behind:

this.Form.DefaultButton = this.Button1.UniqueID;

Accessibility

As an added bonus you are also improving the experience of visually impaired or otherwise disabled visitors. By defining the default button you are helping the screen reader software to have a fighting chance at understanding your asp.net page.

Gotcha - Buttons and ImageButtons only (not LinkButtons)

You can't set a <asp:LinkButton> as your DefaultButton. Technically you are supposed to be able use any control which implements IButtonControl but you cannot use the LinkButton. I think this is probably something to do with the fact that LinkButton is a javascript postback while the other buttons render out to proper form elements.

Further Reading