edu.uci.ics.crawler4j.crawler

Class WebCrawler

java.lang.Object

edu.uci.ics.crawler4j.crawler.WebCrawler

All Implemented Interfaces:

Runnable

public class WebCrawler
extends Object
implements Runnable

WebCrawler class in the Runnable class that is executed by each crawler thread.

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.log4j.Logger	logger
protected CrawlController	myController The controller instance that has created this crawler thread.
protected int	myId The id associated to the crawler thread running this instance

Constructor Summary

Constructors

Constructor and Description
WebCrawler()

Method Summary

Methods

Modifier and Type	Method and Description
CrawlController	getMyController()
int	getMyId() Get the id of the current crawler instance
Object	getMyLocalData() The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread.
Thread	getThread()
protected void	handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) This function is called once the header of a page is fetched.
void	init(int id, CrawlController crawlController) Initializes the current instance of the crawler
boolean	isNotWaitingForNewURLs()
void	onBeforeExit() This function is called just before the termination of the current crawler instance.
protected void	onContentFetchError(WebURL webUrl) This function is called if the content of a url could not be fetched.
protected void	onParseError(WebURL webUrl) This function is called if there has been an error in parsing the content.
void	onStart() This function is called just before starting the crawl by this crawler instance.
void	run()
void	setThread(Thread myThread)
boolean	shouldVisit(WebURL url) Classes that extends WebCrawler can overwrite this function to tell the crawler whether the given url should be crawled or not.
void	visit(Page page) Classes that extends WebCrawler can overwrite this function to process the content of the fetched and parsed page.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.apache.log4j.Logger logger

myId

protected int myId

The id associated to the crawler thread running this instance

myController

protected CrawlController myController

The controller instance that has created this crawler thread. This reference to the controller can be used for getting configurations of the current crawl or adding new seeds during runtime.

Constructor Detail

WebCrawler

public WebCrawler()

Method Detail

init

public void init(int id,
        CrawlController crawlController)

Initializes the current instance of the crawler

Parameters:

id - the id of this crawler instance

crawlController - the controller that manages this crawling session

getMyId

public int getMyId()

Get the id of the current crawler instance

Returns:

the id of the current crawler instance

getMyController

public CrawlController getMyController()

onStart

public void onStart()

This function is called just before starting the crawl by this crawler instance. It can be used for setting up the data structures or initializations needed by this crawler instance.

onBeforeExit

public void onBeforeExit()

This function is called just before the termination of the current crawler instance. It can be used for persisting in-memory data or other finalization tasks.

handlePageStatusCode

protected void handlePageStatusCode(WebURL webUrl,
                        int statusCode,
                        String statusDescription)

This function is called once the header of a page is fetched. It can be overwritten by sub-classes to perform custom logic for different status codes. For example, 404 pages can be logged, etc.

Parameters:

webUrl -

statusCode -

statusDescription -

onContentFetchError

protected void onContentFetchError(WebURL webUrl)

This function is called if the content of a url could not be fetched.

Parameters:

webUrl -

onParseError

protected void onParseError(WebURL webUrl)

This function is called if there has been an error in parsing the content.

Parameters:

webUrl -

getMyLocalData

public Object getMyLocalData()

The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread. Classes that extend WebCrawler can override this function to pass their local data to their controller. The controller then puts these local data in a List that can then be used for processing the local data of crawlers (if needed).

run

public void run()

Specified by:

run in interface Runnable

shouldVisit

public boolean shouldVisit(WebURL url)

Classes that extends WebCrawler can overwrite this function to tell the crawler whether the given url should be crawled or not. The following implementation indicates that all urls should be included in the crawl.

Parameters:

url - the url which we are interested to know whether it should be included in the crawl or not.

Returns:

if the url should be included in the crawl it returns true, otherwise false is returned.

visit

public void visit(Page page)

Classes that extends WebCrawler can overwrite this function to process the content of the fetched and parsed page.

Parameters:

page - the page object that is just fetched and parsed.

getThread

public Thread getThread()

setThread

public void setThread(Thread myThread)

isNotWaitingForNewURLs

public boolean isNotWaitingForNewURLs()