org.dice_research.squirrel.data.uri.filter

Interface KnownUriFilter

All Superinterfaces:

UriFilter

All Known Subinterfaces:

KnownUriFilterDecorator

All Known Implementing Classes:

AbstractKnownUriFilterDecorator, InMemoryKnownUriFilter

public interface KnownUriFilter
extends UriFilter

A UriFilter that works like a blacklist filter and contains only those URIs on its blacklist that the crawler already has seen before.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Method Summary

Modifier and Type	Method and Description
void	add(CrawleableUri uri, long nextCrawlTimestamp) Adds the given URI to the list of already known URIs.
void	add(CrawleableUri uri, long lastCrawlTimestamp, long nextCrawlTimestamp) Adds the given URI to the list of already known URIs together with the the time at which it has been crawled.
long	count() count the numbers of known URIs
List<CrawleableUri>	getOutdatedUris() Returns all CrawleableUris which have to be recrawled.

Methods inherited from interface org.dice_research.squirrel.data.uri.filter.UriFilter

isUriGood

Method Detail

add

void add(CrawleableUri uri,
         long nextCrawlTimestamp)

Adds the given URI to the list of already known URIs. Works like calling add(CrawleableUri, long) with the current system time.

Parameters:

uri - the URI that should be added to the list.

nextCrawlTimestamp - The time at which the given URI should be crawled next.

add

void add(CrawleableUri uri,
         long lastCrawlTimestamp,
         long nextCrawlTimestamp)

Adds the given URI to the list of already known URIs together with the the time at which it has been crawled.

Parameters:

uri - the URI that should be added to the list.

lastCrawlTimestamp - the time at which the given URI has eben crawled.

nextCrawlTimestamp - The time at which the given URI should be crawled next.

getOutdatedUris

List<CrawleableUri> getOutdatedUris()

Returns all CrawleableUris which have to be recrawled. This means their time to next crawl has passed.

Returns:

The outdated CrawleableUris.

count

long count()

count the numbers of known URIs

Returns:

the number of lines in that database