org.dice_research.squirrel.frontier

Interface Frontier

All Superinterfaces:

AutoCloseable, Closeable

All Known Subinterfaces:

ExtendedFrontier

public interface Frontier
extends Closeable

A Frontier is a central class of the crawler managing a queue of URIs that should be crawled in the future. This includes to give access to the queue in terms of 1) getting the next URIs to crawl and 2) adding new URIs to the queue. Note that the Frontier has the ability to check whether a URI should be crawled and, thus, should be added to the queue or not. For example, a Frontier might not add a URI that has already been crawled.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Method Summary

Modifier and Type	Method and Description
void	addNewUri(CrawleableUri uri) Add this URIs to the Frontiers internal queue if the internal rules of the Frontier allow it.
void	addNewUris(List<CrawleableUri> newUris) Adds the given list of URIs to the Frontier.
void	crawlingDone(List<CrawleableUri> uris) This method should be called after a list of URIs have been requested using the getNextUris() method and the crawling has been finished.
boolean	doesRecrawling() Indicates whether this frontier does recrawling.
List<CrawleableUri>	getNextUris() Returns the next chunk of URIs that should be crawled or null.
int	getNumberOfPendingUris() (optional) Returns the number of URIs that have been requested from the Frontier using getNextUris() and have not been marked as crawled using Frontier#crawlingDone(Map).

Methods inherited from interface java.io.Closeable

close

Method Detail

getNextUris

List<CrawleableUri> getNextUris()

Returns the next chunk of URIs that should be crawled or null. Note that if URIs are received from the Frontier using this method, the Frontier should be notified if the crawling of these URIs is done using the #crawlingDone(Dictionary) method.

Returns:

the next chunk of URIs that should be crawled or null if no URIs are available

addNewUri

void addNewUri(CrawleableUri uri)

Add this URIs to the Frontiers internal queue if the internal rules of the Frontier allow it.

Parameters:

uri - the URI that should be added to the Frontier

addNewUris

void addNewUris(List<CrawleableUri> newUris)

Adds the given list of URIs to the Frontier. It is like calling addNewUri(CrawleableUri) with every single URI.

Parameters:

newUris - the URIs that should be added to the Frontier

crawlingDone

void crawlingDone(List<CrawleableUri> uris)

This method should be called after a list of URIs have been requested using the getNextUris() method and the crawling has been finished. Internally, the Frontier marks the URIs as crawled and adds the new URIs using the addNewUris(List) method.

Parameters:

uris - crawled URIs

getNumberOfPendingUris

int getNumberOfPendingUris()

(optional) Returns the number of URIs that have been requested from the Frontier using getNextUris() and have not been marked as crawled using Frontier#crawlingDone(Map).

Returns:

the number of pending URIs.

doesRecrawling

boolean doesRecrawling()

Indicates whether this frontier does recrawling.

Returns:

True iff recrawling is active.