org.dice_research.squirrel.queue

Class AbstractIpAddressBasedQueue

java.lang.Object

org.dice_research.squirrel.queue.AbstractIpAddressBasedQueue

All Implemented Interfaces:

IpAddressBasedQueue, UriQueue

Direct Known Subclasses:

InMemoryQueue

public abstract class AbstractIpAddressBasedQueue
extends Object
implements IpAddressBasedQueue

This abstract class manages two important aspects of an IpAddressBasedQueue. It uses a mutex to access the queue and it manages a set containing IPs that are currently blocked by one of the workers.

Author:

Michael Röder (roeder@informatik.uni-leipzig.de)

Field Summary

Fields

Modifier and Type	Field and Description
private Set<InetAddress>	blockedIps
private static org.slf4j.Logger	LOGGER
private Semaphore	queueMutex

Constructor Summary

Constructors

Constructor and Description
AbstractIpAddressBasedQueue()

Method Summary

Modifier and Type	Method and Description
protected abstract void	addToQueue(CrawleableUri uri)
void	addUri(CrawleableUri uri) Adds the given CrawleableUri instance to the queue.
protected abstract Iterator<IpUriTypePair>	getIterator()
List<CrawleableUri>	getNextUris() Returns the next chunk of URIs that should be crawled or null.
int	getNumberOfBlockedIps() Returns the number of IP addresses that are currently blocked.
protected abstract List<CrawleableUri>	getUris(IpUriTypePair pair)
void	markIpAddressAsAccessible(InetAddress ip) Marks the given IP address as accessible.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.queue.IpAddressBasedQueue

getIPURIIterator

Methods inherited from interface org.dice_research.squirrel.queue.UriQueue

close, isEmpty, open

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

queueMutex

private Semaphore queueMutex

blockedIps

private Set<InetAddress> blockedIps

Constructor Detail

AbstractIpAddressBasedQueue

public AbstractIpAddressBasedQueue()

Method Detail

addUri

public void addUri(CrawleableUri uri)

Description copied from interface: UriQueue

Adds the given CrawleableUri instance to the queue.

Specified by:

addUri in interface UriQueue

Parameters:

uri - the CrawleableUri instance that should be added to the queue.

addToQueue

protected abstract void addToQueue(CrawleableUri uri)

getNextUris

public List<CrawleableUri> getNextUris()

Description copied from interface: UriQueue

Returns the next chunk of URIs that should be crawled or null. Note that this method removes the URIs from the queue.

Specified by:

getNextUris in interface UriQueue

Returns:

the next chunk of URIs that should be crawled or null if no URIs are available

getIterator

protected abstract Iterator<IpUriTypePair> getIterator()

getUris

protected abstract List<CrawleableUri> getUris(IpUriTypePair pair)

markIpAddressAsAccessible

public void markIpAddressAsAccessible(InetAddress ip)

Description copied from interface: IpAddressBasedQueue

Marks the given IP address as accessible.

Specified by:

markIpAddressAsAccessible in interface IpAddressBasedQueue

Parameters:

ip - the IP address that should be marked as accessible.

getNumberOfBlockedIps

public int getNumberOfBlockedIps()

Description copied from interface: IpAddressBasedQueue

Returns the number of IP addresses that are currently blocked.

Specified by:

getNumberOfBlockedIps in interface IpAddressBasedQueue

Returns:

the number of IP addresses that are currently blocked.