org.dice_research.squirrel.collect

Class SimpleUriCollector

java.lang.Object

org.dice_research.squirrel.collect.SimpleUriCollector

All Implemented Interfaces:

UriCollector, SinkBase

public class SimpleUriCollector
extends Object
implements UriCollector

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOGGER
protected Serializer	serializer
private long	total_uris
protected Map<String,Map<String,byte[]>>	urisOfUris

Constructor Summary

Constructors

Constructor and Description
SimpleUriCollector(Serializer serializer)

Method Summary

Modifier and Type	Method and Description
void	addNewUri(CrawleableUri uri, CrawleableUri newUri) Adds the given new URI to the list of URIs collected for the given URI.
void	closeSinkForUri(CrawleableUri uri) Closes the resources necessary for storing the data of the given URI.
long	getSize()
long	getSize(CrawleableUri uri) Returns the total of uris that have been collected
Iterator<byte[]>	getUris(CrawleableUri uri) Returns a list of serialized CrawleableUri instances that have been collected for the given URI.
void	openSinkForUri(CrawleableUri uri) Opens the sink to process data for the given URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dice_research.squirrel.collect.UriCollector

addNewUri, addNewUri, addTriple

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

total_uris

private long total_uris

urisOfUris

protected Map<String,Map<String,byte[]>> urisOfUris

serializer

protected Serializer serializer

Constructor Detail

SimpleUriCollector

public SimpleUriCollector(Serializer serializer)

Method Detail

openSinkForUri

public void openSinkForUri(CrawleableUri uri)

Description copied from interface: SinkBase

Opens the sink to process data for the given URI.

Specified by:

openSinkForUri in interface SinkBase

Parameters:

uri - the URI for which data should be stored.

getUris

public Iterator<byte[]> getUris(CrawleableUri uri)

Description copied from interface: UriCollector

Returns a list of serialized CrawleableUri instances that have been collected for the given URI.

Specified by:

getUris in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

An Iterator that iterates over the already serialized URIs that have been collected for the given URI.

addNewUri

public void addNewUri(CrawleableUri uri,
                      CrawleableUri newUri)

Description copied from interface: UriCollector

Adds the given new URI to the list of URIs collected for the given URI.

Specified by:

addNewUri in interface UriCollector

Parameters:

uri - The URI from which the given new URI has been collected.

newUri - The new URI that has been collected.

getSize

public long getSize()

closeSinkForUri

public void closeSinkForUri(CrawleableUri uri)

Description copied from interface: SinkBase

Closes the resources necessary for storing the data of the given URI.

Specified by:

closeSinkForUri in interface SinkBase

Parameters:

uri - the URI for which data has been stored and for which the resources should be freed.

getSize

public long getSize(CrawleableUri uri)

Description copied from interface: UriCollector

Returns the total of uris that have been collected

Specified by:

getSize in interface UriCollector

Parameters:

uri - The URI from which the returned serialized URIs have been collected.

Returns:

A long number of total uris collected