Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


String url = "http://stackoverflow.com/questions/3152138";
Document document = Jsoup.connect(url).get();

Element question = document.select("#question .post-text p").first();
System.out.println("Question: " + question.text());

Elements answerers = document.select("#answers .user-details a");
for (Element answerer : answerers) {
    System.out.println("Answerer: " + answerer.text());
}

Almost all known HTML parsers implements the W3C DOM API (part of the JAXP API, Java API for XML processing) and gives you a org.w3c.dom.Document back which is ready for direct use by JAXP API. The major differences are usually to be found in the features of the parser in question. Most parsers are to a certain degree forgiving and lenient with non-wellformed HTML ("tagsoup"), like JTidy, NekoHTML, TagSoup and HtmlCleaner. You usually use this kind of HTML parsers to "tidy" the HTML source (e.g. replacing the HTML-valid <br> by a XML-valid <br />), so that you can traverse it "the usual way" using the W3C DOM and JAXP API.

And here's an example how to do exactly the same with Jsoup:

Do you see the difference? It's not only less code, but Jsoup is also relatively easy to grasp if you already have moderate experience with CSS selectors (by e.g. developing websites and/or using jQuery).

Here's an example which uses a "plain" W3C DOM parser like JTidy in combination with XPath to extract the first paragraph of your question and the names of all answerers (I am using XPath since without it, the code needed to gather the information of interest would otherwise grow up 10 times as big, without writing utility/helper methods).

HtmlUnit provides a completely own API which gives you the possibility to act like a webbrowser programmatically. I.e. enter form values, click elements, invoke JavaScript, etcetera. It's much more than alone a HTML parser. It's a real "GUI-less webbrowser" and HTML unit testing tool.

Jsoup also provides a completely own API. It gives you the possibility to select elements using jQuery-like CSS selectors and provides a slick API to traverse the HTML DOM tree to get the elements of interest.

Jsoup is good. I tried to interface it with another module that works with org.w3c.dom.* API. Found that Jsoup doesn't obey the org.w3c.dom.* contract

Particularly the traversing of the HTML DOM tree is the major strength of Jsoup. Ones who have worked with org.w3c.dom.Document know what a hell of pain it is to traverse the DOM using the verbose NodeList and Node APIs. True, XPath makes the life easier, but still, it's another learning curve and it can end up to be still verbose.

The only ones which jumps out are HtmlUnit and Jsoup.

The pros and cons of each should be clear enough now. If you just want to use the standard JAXP API to traverse it, then go for the first mentioned group of parsers. There are pretty a lot of them. Which one to choose depends on the features it provides (how is HTML cleaning made easy for you? are there some listeners/interceptors and tag-specific cleaners?) and the robustness of the library (how often is it updated/maintained/fixed?). If you like to unit test the HTML, then HtmlUnit is the way to go. If you like to extract specific data from the HTML (which is more than often the real world requirement), then Jsoup is the way to go.

There is a huge pro/con that is omitted here: Jericho is the only parser I know that allows you to manipulate nasty HTML while preserving whitespace formatting and the incorrectness of the HTML (if there is any).

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


I found Jericho HTML Parser to be very well written, kept up to date (which many of the parsers are not), no dependencies, and easy to use.

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


I also use HTMLParser and Jericho for a table extraction task, which replaced some code written using Perl's libhtml-tableextract-perl. I use HTMLParser to filter the HTML for the table, then use Jericho to parse it. I agree with MJB's and Adam's comments that Jericho is good in some cases because it preserves the underlying HTML. It has a kind of non-standard SAX interface, so for XPath processing HTMLCleaner is better.

I have tried most of the parsers here for a crawler / data extraction framework I have been developing. I use HTMLCleaner for the bulk of the data extraction work. This is because it supports a reasonably modern dialect of HTML, XHTML, HTML 5, with namespaces, and it supports DOM, so it is possible to use it with Java's built in XPath implementation.

It's a lot easier to do this with HTMLCleaner than some of the other parsers: JSoup for example supports a DOM like interface, rather than DOM, so some assembly required. Jericho has a SAX-line interface so again it is requires some work although Sujit Pal has a good description of how to do this but in the end HTMLCleaner just worked better.

Parsing HTML in Java is a surprisingly hard problem as all the parsers seem to struggle on certain types of malformed HTML content.

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


Add The validator.nu HTML Parser, an implementation of the HTML5 parsing algorithm in Java, to your list.

In practice, such problems only affect obscure corner cases, and is for all practical purposes, an excellent parser.

On the minus side, no browsers' legacy parsing works exactly like this, and as HTML5 is still in draft, subject to change.

On the plus side, it's specifically designed to match HTML5, and at the heart of the HTML5 validator, so highly likely to match future browser's parsing behaviour to a very high degree of accuracy.

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


String url = "http://stackoverflow.com/questions/3152138";
Document document = Jsoup.connect(url).get();

Element question = document.select("#question .post-text p").first();
System.out.println("Question: " + question.text());

Elements answerers = document.select("#answers .user-details a");
for (Element answerer : answerers) {
    System.out.println("Answerer: " + answerer.text());
}

Almost all known HTML parsers implements the W3C DOM API (part of the JAXP API, Java API for XML processing) and gives you a org.w3c.dom.Document back which is ready for direct use by JAXP API. The major differences are usually to be found in the features of the parser in question. Most parsers are to a certain degree forgiving and lenient with non-wellformed HTML ("tagsoup"), like JTidy, NekoHTML, TagSoup and HtmlCleaner. You usually use this kind of HTML parsers to "tidy" the HTML source (e.g. replacing the HTML-valid <br> by a XML-valid <br />), so that you can traverse it "the usual way" using the W3C DOM and JAXP API.

And here's an example how to do exactly the same with Jsoup:

Do you see the difference? It's not only less code, but Jsoup is also relatively easy to grasp if you already have moderate experience with CSS selectors (by e.g. developing websites and/or using jQuery).

Here's an example which uses a "plain" W3C DOM parser like JTidy in combination with XPath to extract the first paragraph of your question and the names of all answerers (I am using XPath since without it, the code needed to gather the information of interest would otherwise grow up 10 times as big, without writing utility/helper methods).

HtmlUnit provides a completely own API which gives you the possibility to act like a webbrowser programmatically. I.e. enter form values, click elements, invoke JavaScript, etcetera. It's much more than alone a HTML parser. It's a real "GUI-less webbrowser" and HTML unit testing tool.

Jsoup also provides a completely own API. It gives you the possibility to select elements using jQuery-like CSS selectors and provides a slick API to traverse the HTML DOM tree to get the elements of interest.

Jsoup is good. I tried to interface it with another module that works with org.w3c.dom.* API. Found that Jsoup doesn't obey the org.w3c.dom.* contract

Particularly the traversing of the HTML DOM tree is the major strength of Jsoup. Ones who have worked with org.w3c.dom.Document know what a hell of pain it is to traverse the DOM using the verbose NodeList and Node APIs. True, XPath makes the life easier, but still, it's another learning curve and it can end up to be still verbose.

The only ones which jumps out are HtmlUnit and Jsoup.

The pros and cons of each should be clear enough now. If you just want to use the standard JAXP API to traverse it, then go for the first mentioned group of parsers. There are pretty a lot of them. Which one to choose depends on the features it provides (how is HTML cleaning made easy for you? are there some listeners/interceptors and tag-specific cleaners?) and the robustness of the library (how often is it updated/maintained/fixed?). If you like to unit test the HTML, then HtmlUnit is the way to go. If you like to extract specific data from the HTML (which is more than often the real world requirement), then Jsoup is the way to go.

There is a huge pro/con that is omitted here: Jericho is the only parser I know that allows you to manipulate nasty HTML while preserving whitespace formatting and the incorrectness of the HTML (if there is any).

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


I also use HTMLParser and Jericho for a table extraction task, which replaced some code written using Perl's libhtml-tableextract-perl. I use HTMLParser to filter the HTML for the table, then use Jericho to parse it. I agree with MJB's and Adam's comments that Jericho is good in some cases because it preserves the underlying HTML. It has a kind of non-standard SAX interface, so for XPath processing HTMLCleaner is better.

I have tried most of the parsers here for a crawler / data extraction framework I have been developing. I use HTMLCleaner for the bulk of the data extraction work. This is because it supports a reasonably modern dialect of HTML, XHTML, HTML 5, with namespaces, and it supports DOM, so it is possible to use it with Java's built in XPath implementation.

It's a lot easier to do this with HTMLCleaner than some of the other parsers: JSoup for example supports a DOM like interface, rather than DOM, so some assembly required. Jericho has a SAX-line interface so again it is requires some work although Sujit Pal has a good description of how to do this but in the end HTMLCleaner just worked better.

Parsing HTML in Java is a surprisingly hard problem as all the parsers seem to struggle on certain types of malformed HTML content.

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


Add The validator.nu HTML Parser, an implementation of the HTML5 parsing algorithm in Java, to your list.

In practice, such problems only affect obscure corner cases, and is for all practical purposes, an excellent parser.

On the minus side, no browsers' legacy parsing works exactly like this, and as HTML5 is still in draft, subject to change.

On the plus side, it's specifically designed to match HTML5, and at the heart of the HTML5 validator, so highly likely to match future browser's parsing behaviour to a very high degree of accuracy.

Note
Rectangle 27 0

What are the pros and cons of the leading Java HTML parsers?


I'll just add to @MJB answer after working with most of the HTML parsing libraries in Java, there is a huge pro/con that is omitted: parsers that preserve the formatting and incorrectness of the HTML on input and output.

Jericho is the only parser I know that allows you to manipulate nasty HTML while preserving whitespace formatting and the incorrectness of the HTML (if there is any).

That is most parsers when you change the document will blow away the whitespace, comments, and incorrectness of the DOM particularly if they are an XML like library.

Note