VB.net link extraction with HtmlAgilityPack
I was able to extract URLs with simple href tags like this:
<a href="http://www.samplesite.com">
but my problem is how do i extract a link from an href tag that looks like
this?
<a
href="http://www.wherecreativitygoestoschool.com/vancouver/left_right/rb_test.htm"
onmousedown="return
rwt(this,'','','','1','AFQjCNHvlwTxfBVEYcqGUnilAZN0uY2IXw','','0CCsQFjAA','','',event)">
Right Brain vs Left Brain Creativity <em>Test</em> at The Art Institute of
<b>...</b></a>
Here is my complete code:
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles
Button1.Click
Dim webClient As New System.Net.WebClient
Dim WebSource As String =
webClient.DownloadString("http://www.google.com.ph/search?hl=en&as_q=test&as_epq=&as_oq=&as_eq=&as_nlo=&as_nhi=&lr=&cr=countryCA&as_qdr=all&as_sitesearch=&as_occt=any&safe=images&tbs=ctr%3AcountryCA&as_filetype=&as_rights=#as_qdr=all&cr=countryCA&fp=1&hl=en&lr=&q=test&start=20&tbs=ctr:countryCA")
Dim doc = New HtmlAgilityPack.HtmlDocument()
doc.LoadHtml(WebSource)
Dim links = GetLinks(doc, "test")
For Each Link In links
ListBox1.Items.Add(Link.ToString())
Next
End Sub
Public Class Link
Public Sub New(Uri As Uri, Text As String)
Me.Uri = Uri
Me.Text = Text
End Sub
Public Property Text As String
Public Property Uri As Uri
Public Overrides Function ToString() As String
Return String.Format(If(Uri Is Nothing, "", Uri.ToString()))
End Function
End Class
Public Function GetLinks(doc As HtmlAgilityPack.HtmlDocument,
linkContains As String) As List(Of Link)
Dim uri As Uri = Nothing
Dim linksOnPage = From link In doc.DocumentNode.Descendants()
Where link.Name = "a" _
AndAlso link.Attributes("href") IsNot Nothing _
Let text = link.InnerText.Trim()
Let url = link.Attributes("href").Value
Where url.IndexOf(linkContains,
StringComparison.OrdinalIgnoreCase) >= 0 _
AndAlso uri.TryCreate(url, UriKind.Absolute,
uri)
Dim Uris As New List(Of Link)()
For Each link In linksOnPage
Uris.Add(New Link(New Uri(link.url, UriKind.Absolute),
link.text))
Next
Return Uris
End Function
I have noticed that my code does not extract links that ends with </a>. Is
there anything i can do to modify my code that it would extract links
ending with </a>?
No comments:
Post a Comment