org.dice_research.squirrel.data.uri.filter

Class InMemoryKnownUriFilter

java.lang.Object

org.dice_research.squirrel.data.uri.filter.InMemoryKnownUriFilter

All Implemented Interfaces:

KnownUriFilter, UriFilter

public class InMemoryKnownUriFilter
extends Object
implements KnownUriFilter

A simple in-memory implementation of the KnownUriFilter interface.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	InMemoryKnownUriFilter.UriInfo

Field Summary

Fields

Modifier and Type	Field and Description
protected long	defaultRecrawlTime
private boolean	frontierDoesRecrawling Indicates whether the Frontier using this filter does recrawling.
protected Hashtable<CrawleableUri,InMemoryKnownUriFilter.UriInfo>	uris - key: the crawled (known) uri - value: the info about the URI (see InMemoryKnownUriFilter.UriInfo), including the reference list

Constructor Summary

Constructors

Constructor and Description
InMemoryKnownUriFilter() Constructor.
InMemoryKnownUriFilter(boolean frontierDoesRecrawling, long defaultRecrawlTime) Constructor.
InMemoryKnownUriFilter(Hashtable<CrawleableUri,InMemoryKnownUriFilter.UriInfo> uris, boolean frontierDoesRecrawling) Constructor.

Method Summary

Modifier and Type	Method and Description
void	add(CrawleableUri uri, long nextCrawlTimestamp) Adds the given URI to the list of already known URIs.
void	add(CrawleableUri uri, long lastCrawlTimestamp, long nextCrawlTimestamp) Adds the given URI to the list of already known URIs together with the the time at which it has been crawled.
long	count() count the numbers of known URIs
List<CrawleableUri>	getOutdatedUris() Returns all CrawleableUris which have to be recrawled.
boolean	isUriGood(CrawleableUri uri) Returns true if the given CrawleableUri object fulfills the requirements imposed by this filter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

uris

protected Hashtable<CrawleableUri,InMemoryKnownUriFilter.UriInfo> uris

- key: the crawled (known) uri - value: the info about the URI (see InMemoryKnownUriFilter.UriInfo), including the reference list

frontierDoesRecrawling

private boolean frontierDoesRecrawling

Indicates whether the Frontier using this filter does recrawling.

defaultRecrawlTime

protected long defaultRecrawlTime

Constructor Detail

InMemoryKnownUriFilter

public InMemoryKnownUriFilter(boolean frontierDoesRecrawling,
                              long defaultRecrawlTime)

Constructor.

Parameters:

frontierDoesRecrawling - Value for frontierDoesRecrawling.

InMemoryKnownUriFilter

public InMemoryKnownUriFilter()

Constructor.

InMemoryKnownUriFilter

public InMemoryKnownUriFilter(Hashtable<CrawleableUri,InMemoryKnownUriFilter.UriInfo> uris,
                              boolean frontierDoesRecrawling)

Constructor.

Parameters:

uris - Value for uris.

frontierDoesRecrawling - Value for frontierDoesRecrawling.

Method Detail

add

public void add(CrawleableUri uri,
                long nextCrawlTimestamp)

Description copied from interface: KnownUriFilter

Adds the given URI to the list of already known URIs. Works like calling KnownUriFilter.add(CrawleableUri, long) with the current system time.

Specified by:

add in interface KnownUriFilter

Parameters:

uri - the URI that should be added to the list.

nextCrawlTimestamp - The time at which the given URI should be crawled next.

add

public void add(CrawleableUri uri,
                long lastCrawlTimestamp,
                long nextCrawlTimestamp)

Description copied from interface: KnownUriFilter

Adds the given URI to the list of already known URIs together with the the time at which it has been crawled.

Specified by:

add in interface KnownUriFilter

Parameters:

uri - the URI that should be added to the list.

lastCrawlTimestamp - the time at which the given URI has eben crawled.

nextCrawlTimestamp - The time at which the given URI should be crawled next.

isUriGood

public boolean isUriGood(CrawleableUri uri)

Description copied from interface: UriFilter

Returns true if the given CrawleableUri object fulfills the requirements imposed by this filter.

Specified by:

isUriGood in interface UriFilter

Parameters:

uri - the CrawleableUri object that is checked

Returns:

true if the given CrawleableUri object fulfills the requirements imposed by this filter. Otherwise false is returned.

getOutdatedUris

public List<CrawleableUri> getOutdatedUris()

Description copied from interface: KnownUriFilter

Returns all CrawleableUris which have to be recrawled. This means their time to next crawl has passed.

Specified by:

getOutdatedUris in interface KnownUriFilter

Returns:

The outdated CrawleableUris.

count

public long count()

Description copied from interface: KnownUriFilter

count the numbers of known URIs

Specified by:

count in interface KnownUriFilter

Returns:

the number of lines in that database