Rectangle 27 0

By default, Nutch no longer comes with a Hadoop distribution, however when run in local mode e.g. running Nutch in a single process on one machine, then we use Hadoop as a dependency. This may suit you fine if you have a small site to crawl and index, but most people choose Nutch because of its capability to run on in deploy mode, within a Hadoop cluster.

Need an open source crawler like Apache Nutch without Hadoop - Stack O...

hadoop web-crawler nutch
Rectangle 27 0

Before fetching job, the generating job is performed in Nutch. In the generating job, Nutch will select topN URLs, which have the highest scores among all URLs in CrawlDB, for fetching. Therefore the reason of your crawler taking too long time before fetching would be you set topN is too high compared to your system capacity, and the number of URLs in crawlDB is large (selecting process will take time).

If I select topN a small number, is there any drawback?

if you set topN, the number of URLs you get in each crawling round is also small.

is there a way to do limitless topN as I do not know exactly how many urls are there in database. Or What should be the configuration for a local language search engine to crawl but do not take too mutch time in starting( generation)

I believe if you dont specify the topN there is no limit set.

web crawler - apache nutch taking too long time in generate phase - St...

apache web-crawler nutch
Rectangle 27 0

I had the same problem but with more characters so I changed! New URLs add to Queue in "feeding" section! you have to find this line:


and replace it with this:


How to crawl urls having space using Apache Nutch? - Stack Overflow

Rectangle 27 0

At first, we should respect the robots.txt file if you are crawling any external sites. Otherwise you are at risk - your IP banned or worse can be any legal case.

If your site is internal and not expose to external world, then you should change the robots.txt file to allow your crawler.

If your site is exposed to the Internet and if data is confidential, then you can try out the following option. Because here you cannot take a risk of modifying the robots.txt file since external crawler can use your crawler name and crawl the site.

if (!rules.isAllowed(fit.u.toString())) { }

This is the block that is responsible for blocking the URLs. You can play around this code block to resolve your issue.

java - how to bypass robots.txt with apache nutch 2.2.1 - Stack Overfl...

java nutch robots.txt web-crawler
Rectangle 27 0

As I was also facing similiar problem. Actual problem was with regionserver (Hbase deamon ). So try to restart it as it is shutdown when used with default seeting and data is too mutch in hbase. For more information, see log files of regionserver.

java - Apache nutch is not crawling any more - Stack Overflow

java hadoop hbase web-crawler nutch
Rectangle 27 0

Although Nutch was built to be a web scale search engine, this is not the case any more. Currently, the main purpose of Nutch is to do a large scale crawling. What you do with that crawled data is then up to your requirements. By default, Nutch allows to send data into Solr. That is why you can run

crawl url crawl solraddress depth level

You can emit the solr url parameter also. In that case, nutch will not send the crawled data into Solr. Without sending the crawled data to solr, you will not be able to search data. Crawling data and searching data are two different things but very related.

Generally, you will find the crawled data in the crawl/segments not crawl/crawdb. The crawl db folder stores information about the crawled urls, their fetching status and next time for fetching plus some other useful information for crawling. Nutch stores actual crawled data in crawl/segments.

If you want to have an easy way to view crawled data, you might try nutch 2.x as it can store its crawled data into several back ends like MySQL, Hbase, Cassandra and etc through the Gora component.


Otherwise, you can always push your data into different stores via adding indexer plugins. Currently, Nutch supports sending data to Solr and Elasticsearch. These indexer plugins send structured data like title, text, metadata, author and other metadata.

The following summarizes what happens in Nutch:

seed list -> crawldb -> fetching raw data (download site contents) 
-> parsing the raw data -> structuring the parse data into fields (title, text, anchor text, metadata and so on)-> 
sending the structured data to storage for usage (like ElasticSearch and Solr).

thanks for the reply. iam working on nutch1.9 using cygwin in windows. crawl/segments is not getting created for me. Only crawl/crawdb is getting created.And when i run the command ` ./crawl urls -crawl -depth 3 -topN 5` am getting following error. how to resolve!!! Fetcher: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/C:/cygwin64/home/apache-nutch-1.9-bin/apache-nutch-1.9/bin/-crawl/segments/crawl_generate

can you recommend a valid tutorial for setting up nutch in windows. am frustrated on this

i honestly will not recommend running Nutch in windows. It is just unpredictable. Try to run in virtual machine.

am using cygwin..isnt it same as unix the steps i have to follow must be same as unix env.. right..

Nutch uses hadoop and hadoop does not work that well in cygwin. I have not used Nutch in cygwin, so I can not help much here. Btw, can you accept the answer if it solves your original question.

Apache nutch and solr : queries - Stack Overflow

apache solr nutch
Rectangle 27 0

As far as I know you cannot crawl Facbook Using nutch. specifies that content inside facebook is not available for crawling.

apache - Getting the number of facebook likes from a certain page with...

facebook apache indexing web-crawler nutch
Rectangle 27 0

NutchTutorial covers step by step instructions for configuring Nutch and Solr Integration

glassfish - How to integrate apache nutch with apache solr on linux? -...

solr glassfish nutch
Rectangle 27 0

You have to Schedule ta Job for Firing the Job However, Nutch AdaptiveFetchSchedule should enable you to crawl and index pages and detect whether the page is new or updated and you don't have to do it manually.

Article describes the same in detail.

Ok, I read the article and I have another question.Do I have to use any job sheduler for run my command for crawl the given url or I need Adaptive Fetch scheduler to do this? And if the Adaptive Fetch is the right one how can I use it?

you can configure adaptice schedule wihtin in config. And you would need a scheduler to fire the job e.g. Autosys, Quartz etc.

I will have to disagree with you here. The class you mention works according to the crawled site's "if-modified-since" and "last-modified" http headers. And I must tell, none of the sites around (except for google, youtube, stackoverflow etc.) mustn't be trusted on the truthfulness of these headers.

If you are building the site, its upon you to take care of this so that crawling works fine for you.

apache - Recrawl URL with Nutch just for updated sites - Stack Overflo...

apache solr lucene nutch web-crawler
Rectangle 27 0

You can use the plugin below to extract content based on XPath queries. If your content is in a specific div, you can use this plugin to extract the content you want from that specific section.

Thanks for the link. But its useful for a website of known structure "Using this plugin we are now able to extract the desired data from web site with known structure." I am looking for a way to extract from a website with unknown structure

the "Description <p>" is just one example, not all the sites I crawled would have a similar structure

apache - How to control the way Nutch parses and Solr indexes a URL wh...

apache solr web-crawler nutch solr4
Rectangle 27 0

Well, nutch will write the crawled data in binary form so if if you want that to be saved in html format, you will have to modify the code. (this will be painful if you are new to nutch).

If you want quick and easy solution for getting html pages:

  • If the list of pages/urls that you intend to have is quite low, then better get it done with a script which invokes wget for each url.

Writing a your own nutch plugin will be great. Your problem will get solved plus you can contribute to nutch by submitting your work !!! If you are new to nutch (in terms of code & design), then you will have to invest lot of time building a new plugin ... else its easy to do.

Here is a page which talks about writing own nutch plugin.

Start with See lines 647-648. That is the place where you can get the fetched content on per url basis (for those pages which got fetched successfully).

pstatus = output(fit.url, fit.datum, content, status, CrawlDatum.STATUS_FETCH_SUCCESS);

You should add code right after this to invoke your plugin. Pass content object to it. By now, you would have guessed that content.getContent() is the content for url you want. Inside the plugin code, write it to some file. Filename should be based on the url name else it will be difficult to work with that. Url can be obtained by fit.url.

Thank you, TejasP. I just heard that Nutch has a plugin mechanism to extend its functionality. I'm wondering if I can write some plugin to make it possible?

@Freedom : see my edit above. hope that helps you.

Thanks for the details and It's very helpful to me. It's a guide for me to plunge into Nutch. Very appreciate it!

search engine - How do I save the origin html file with Apache Nutch -...

search-engine web-crawler nutch
Rectangle 27 0

Alternately, here is what you can try

bin/nutch mergesegs crawl/merged crawl/segments/*
bin/nutch readseg -dump crawl/merged/* dumpedContent

apache - Dump all segments from nutch - Stack Overflow

apache nutch
Rectangle 27 0

If you want Nutch to crawl and index your pdf documents, you have to enable document crawling and the Tika plugin:

  • Document crawling 1.1 Edit regex-urlfilter.txt and remove any occurence of "pdf" # skip image and other suffixes we can't yet parse # for a more extensive coverage use the urlfilter-suffix plugin -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$ 1.2 Edit suffix-urlfilter.txt and remove any occurence of "pdf" 1.3 Edit nutch-site.xml, add "parse-tika" and "parse-html" in the plugin.includes section <property> <name>plugin.includes</name> <value>protocol-http|urlfilter-regex|parse-(html|tika|text)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value> <description>Regular expression naming plugin directory names to include. Any plugin not matching this expression is excluded. In any case you need at least include the nutch-extensionpoints plugin. By default Nutch includes crawling just HTML and plain text via HTTP, and basic indexing and search plugins. In order to use HTTPS please enable protocol-httpclient, but be aware of possible intermittent problems with the underlying commons-httpclient library. </description> </property>

If what you really want is to download all pdf files from a page, you can use something like Teleport in Windows or Wget in *nix.

hadoop - How to Crawl .pdf links using Apache Nutch - Stack Overflow

apache hadoop nutch
Rectangle 27 0

I'm using OSX, but I had that same error about Could not find or load main class ...InjectorJob, and I believe that it is the result of a dirty source directory, in my case, I had checked it out via Git, and changed branches a few times, as I was trying out various features. So, you've been running ant or ant runtime to rebuild the runtime/deploy directory, but to solve this, I had to run:

ant clean

Which deletes this compiled output, and recompiles it properly. After this point, the crawl command runs properly.

apache nutch 2.2.1 Error in execution - Stack Overflow

Rectangle 27 0

The simplest way to validate your data sounds like what you are trying to do: query the data and make sure it returns the expected results. Some help there:

When you say you tried a basic query string, do you mean from the solr admin, or through the rest API? If you are using the solr admin, you don't need to escape that first *. So q would be : directly. In the Rest API, the * needs to be properly encoded. Something like this:


Another thing you can do is validate some of nutch's intermediary data is to dump the crawl or link dbs using the bin/nutch commands readdb, readlinkdb, mergedb.

Thank you so much for you reply! After some more analysis I found that the default search field (i.e., <defaultSearchField>content</defaultSearchField>) present in schema.xml (schema.xml copied from conf directory of nutch) was not matching with the one from solrconfig.xml (i.e., <str name="df">text</str> ). Notice that 'content' against 'text'. The problem got resolved after changing 'text' to 'content' in <str name="df">text</str>. After that I could configure 'nutch' (2.2.1) with HBase (0.90.4). I could crawl the data but now I don't know how to verify the same.

OK, so then the problem is writing your verification test? You need to compare the output from your query to the output you expected to get for the crawled site. So if you were crawling a file system where the doc id is the filesystem path, you might compare the results of an ls -R to a query where q is ., and fl:id. Web sites could do something similar if you have a site index. What kind of data are you trying to browse?

My main aim is to crawl a websites for product details and store this crawled information in HBase. I could crawl the data and it got stored in HBase. However, when I scan the particular table from HBase, I don't see the data from the crawled site. For this particular issue, I have created separate thread at- Could you please take a look at this thread and let me know your thought. I would be extremely thankful to you for any help provided in this regards.

Sorry, have only used it with the 1.x-series Hadoop wrt your new question. If you are sure your data isn't getting to Solr, useing the readdb commands to get more information as to whether your crawls are succeeding or now.

apache - How to see data crawled by nutch using solr? - Stack Overflow

apache solr lucene nutch
Rectangle 27 0

The error message clearly indicates the problem (and where to look for a solution):

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/hung/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/hung/.m2/repository/org/slf4j/slf4j-jdk14/1.6.1/slf4j-jdk14-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/hung/.m2/repository/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.

How could i run the apache nutch 2.2.1 src

@jackyesind you don't run apache nutch 2.2.1 source but rather you compile it using ant, go the directory that contains the source and run the command "ant runtime" when compilation finish you have a directory structure that looks like "nutch-src/runtime/local/bin" inside this path you will see the binaries that you need to execute.

Step by step running apache Nutch 2.2.1 - Stack Overflow

Rectangle 27 0

I think, the message is not problem. batch_id not assigned to all of url. So, if batch_id is null , skip url. Generate url when batch_id assined for url.

There is a problem that I want to crawl a lot of websites... When a lot of pages are skipped, I have very few collected metadates and it is bad for me. And I also dont know why the page is fetched and parsed and for example the page is skipped... and most of the other pages of the domain is skipped... it is the same domain name so why?

You think that,if you have too many same domain, urls is crawled continuously for the domain and may be the another domains are crawled after a long time.

It was added to track by maintainer of mailing-list.

java - Apache Nutch 2.1 - Skipping; ...

java apache nutch
Rectangle 27 0

You can use solr for indexing purpose. Solr is an open-source search server based on the Lucene Java search library and easily configurable with Nutch.

It will crawl seed urls list up to specified depth and index them to specified solr server. Solr internally creates lucene indexes..

I don't have the possibility to use SOLR for the moment, this is a constraint that I have and I have to use the existing Lucene Analysers for indexing purpose. The indexes generated by Nutch seem to be different than those of Lucene, I'm not sure if there is a way to use pure Lucene Analysers with the Nutch Segments.

Apache Nutch with Lucene - Stack Overflow

apache lucene indexing nutch
Rectangle 27 0

I finally did it.I was easy to do. i am sharing my experience here. May be it can help someone.

1- change the configuration file of hbase-site.xml for pseudo distributed mode.

2- MOST IMPORTANT THING: on hbase machine, replace localhost ip in /etc/hosts with your real network ip like this

hbase machine's ip = (note: if you won't change your hbase machine's localhost ip, remote nutch crawler won't be able to connect to it)

hadoop - how can i connect apache nutch 2.x to a remote hbase cluster ...

hadoop hbase nutch zookeeper